*Introgression of cultivar alleles into wild plant populations via crop-wild hybridization is primarily governed by their fitness effects as well as those of linked loci. The fitness of crop-wild hybrids is often dependent on environmental factors, but less is understood about how aspects of the environment affect individual cultivar alleles. *This study investigated the effects of naturally occurring herbivory on patterns of phenotypic selection and the genetic architecture of plant-herbivore interactions in an experimental sunflower crop-wild hybrid population in two locales. *Phenotypic selection analyses suggested that cultivar alleles conferring increased size were generally favored, but at one site cultivar-like flowering time was favored only if three types of herbivory were included in the selection model. Quantitative trait locus (QTL) mapping identified three regions in which the cultivar allele conferred a selective advantage for a number of co-localized traits. Quantitative trait loci for several measures of insect herbivory were detected and, although the cultivar allele increased herbivory damage at the majority of these QTLs, they rarely colocalized with advantageous cultivar alleles for morphological traits. *These results suggest that a subset of cultivar traits/alleles are advantageous in natural environments but that herbivory may mitigate the selective advantage of some cultivar alleles.