We report effects of age, age2, sex and additive genetic effects on variability in gray matter thickness, surface area and white matter integrity in 1,010 subjects from the Genetics of Brain Structure and Function Study. Age was primarily associated with gray matter thickness and fractional anisotropy of water diffusion in white matter tracts, while sex was primarily associated with affected gray matter surface area; age2 was only significantly associated with average white matter integrity. Widespread heritability of neuroanatomic traits was observed, suggesting that brain structure is under strong genetic control. Furthermore, our findings indicate that neuroimaging-based measurements of cerebral variability are sensitive to genetic mediation. Further fundamental studies of genetic influence on the brain will help inform gene discovery initiatives in both clinical and normative samples.