Birdsong is a classic example of a learned trait with cultural inheritance, with selection acting on trait expression. To understand how song responds to selection, it is vital to determine the extent to which variation in song learning and neuroanatomy is attributable to genetic variation, environmental conditions, or their interactions. Using a partial cross fostering design with an experimental stressor, we quantified the heritability of song structure and key brain nuclei in the song control system of the zebra finch and the genotype-by-environment (G × E) interactions. Neuroanatomy and song structure both showed low levels of heritability and are unlikely to be under selection as indicators of genetic quality. HVC, in particular, was almost entirely under environmental control. G × E interaction was important for brain development and may provide a mechanism by which additive genetic variation is maintained, which in turn may promote sexual selection through female choice. Our study suggests that selection may act on the genes determining vocal learning, rather than directly on the underlying neuroanatomy, and emphasizes the fundamental importance of environmental conditions for vocal learning and neural development in songbirds.