The responses of orientation-selective neurons in primate visual cortex can be profoundly affected by the presence and orientation of stimuli falling outside the classical receptive field. Our perception of the orientation of a line or grating also depends upon the context in which it is presented. For example, the perceived orientation of a grating embedded in a surround tends to be repelled from the predominant orientation of the surround. Here, we used fMRI to investigate the basis of orientation-specific surround effects in five functionally-defined regions of visual cortex: V1, V2, V3, V3A/LO1 and hV4. Test stimuli were luminance-modulated and isoluminant gratings that produced responses similar in magnitude. Less BOLD activation was evident in response to gratings with parallel versus orthogonal surrounds across all the regions of visual cortex investigated. When an isoluminant test grating was surrounded by a luminance-modulated inducer, the degree of orientation-specific contextual modulation was no larger for extrastriate areas than for V1, suggesting that the observed effects might originate entirely in V1. However, more orientation-specific modulation was evident in extrastriate cortex when both test and inducer were luminance-modulated gratings than when the test was isoluminant; this difference was significant in area V3. We suggest that the pattern of results in extrastriate cortex may reflect a refinement of the orientation-selectivity of surround suppression specific to the colour of the surround or, alternatively, processes underlying the segmentation of test and inducer by spatial phase or orientation when no colour cue is available.