<b><i>Background:</i></b> Cognitive capabilities change in later life, although their onset and rate of decline, and how they are shaped by lifetime socioeconomic position, childhood cognition and adult health status are all unclear. <b><i>Methods:</i></b> From the Medical Research Council National Survey of Health and Development, we analysed 3,192 participants undergoing one or more cognitive assessments at ages 43, 53, 60–64 and 69. Linear mixed models described cognitive trajectories, adjusting for factors across the life course. <b><i>Results:</i></b> For both search speed and verbal memory, better performance at age 43 (the intercept) was associated with higher paternal and own education, childhood cognition, and adult occupational class. For search speed, the trajectory was best described as a quadratic function (decline of 45.6 letters/5-years + 4.6 letters). Verbal memory showed a linear decline of 0.20 words/5-years between ages 43 and 60 and a steeper linear decline of 0.95 words/5-years between ages 60 and 69. Decline in verbal memory in the latter period was steeper in those with higher educational achievements at age 26 (additional 0.28 words/5-years for highest attainment). <b><i>Conclusions:</i></b> Decline in verbal memory and search speed across midlife is evident, though with different non-linear trajectories. By implication, pathways to cognitive impairment and dementia in older age may have their origins in this period.