To determine the impact of genetic variants on the brain, we used genetically informed brain atlases in genome-wide association studies of regional cortical surface area and thickness in 39,898 adults and 9136 children. We uncovered 440 genome-wide significant loci in the discovery cohort and 800 from a post hoc combined meta-analysis. Loci in adulthood were largely captured in childhood, showing signatures of negative selection, and were linked to early neurodevelopment and pathways associated with neuropsychiatric risk. Opposing gradations of decreased surface area and increased thickness were associated with common inversion polymorphisms. Inferior frontal regions, encompassing Broca’s area, which is important for speech, were enriched for human-specific genomic elements. Thus, a mixed genetic landscape of conserved and human-specific features is concordant with brain hierarchy and morphogenetic gradients.