Identification of genes associated with brain aging should markedly improve our understanding of the biological processes that govern normal age-related decline. However, challenges to identifying genes that facilitate successful brain aging are considerable, including a lack of established phenotypes and difficulties in modeling the effects of aging per se, rather than genes that influence the underlying trait. In a large cohort of randomly selected pedigrees ( n = 1,129 subjects), we documented profound aging effects from young adulthood to old age (18–83 y) on neurocognitive ability and diffusion-based white-matter measures. Despite significant phenotypic correlation between white-matter integrity and tests of processing speed, working memory, declarative memory, and intelligence, no evidence for pleiotropy between these classes of phenotypes was observed. Applying an advanced quantitative gene-by-environment interaction analysis where age is treated as an environmental factor, we demonstrate a heritable basis for neurocognitive deterioration as a function of age. Furthermore, by decomposing gene-by-aging (G × A) interactions, we infer that different genes influence some neurocognitive traits as a function of age, whereas other neurocognitive traits are influenced by the same genes, but to differential levels, from young adulthood to old age. In contrast, increasing white-matter incoherence with age appears to be nongenetic. These results clearly demonstrate that traits sensitive to the genetic influences on brain aging can be identified, a critical first step in delineating the biological mechanisms of successful aging.