Significance Brain and cognition change with age, with early gains and later declines. Attempts have been made to identify age-specific mechanisms, focusing on when and how declines begin in adults. However, even though general cognitive ability declines with age, there is a high stability in individuals’ cognitive ability relative to their same-age peers. Here we show that the relation between brain and cognition appears remarkably stable through the human lifespan. The cortical area change trajectories of higher and lower cognitive ability groups were parallel through life. Birth weight and parental education were identified as predictors, which provides novel evidence for stability in brain–cognition relationships throughout life, and indicates that early life factors impact brain and cognition for the entire life course.