Contextual modulation due to feature contrast between the receptive field and surrounding region has been reported for numerous stimuli in primary visual cortex. One type of this modulation, iso-orientation surround suppression, has been studied extensively. The degree to which surround suppression is related to other forms of contextual modulation remains unknown. We used shape-from-shading stimuli in a field of distractors to test the latency and magnitude of contextual modulation to a stimulus that cannot be distinguished with an orientation-selective mechanism. This stimulus configuration readily elicits perceptual pop-out in human observers and induces a long-latency contextual modulation response in neurons in macaque early visual cortex. We found that animals trained to detect the location of a pop-out stimulus were better at finding a sphere that appeared to be lit from below in the presence of distractors that were lit from above. Furthermore, neuronal responses were stronger and had shorter latency in the condition where behavioral performance was best. This asymmetry is compatible with earlier psychophysical findings in human observers. In the population of V1 neurons, the latency of the contextual modulation response is 145 ms on average (ranging from 70 to 230 ms). This is much longer than the latency for iso-orientation surround suppression, indicating that the underlying circuitry is distinct. Our results support the idea that a feature-specific feedback signal generates the pop-out responses we observe and suggest that V1 neurons actively participate in the computation of perceptual salience.