The extent to which social relationships influence cognitive aging is unclear. In this study, we investigated the association of midlife quality of close relationships with subsequent cognitive decline. Participants in the Whitehall II Study (n = 5,873; ages 45–69 years at first cognitive assessment) underwent executive function and memory tests 3 times over a period of 10 years (1997–1999 to 2007–2009). Midlife negative and positive aspects of close relationships were assessed twice using the Close Persons Questionnaire during the 8 years preceding cognitive assessment. Negative aspects of close relationships, but not positive aspects, were associated with accelerated cognitive aging. Participants in the top third of reported negative aspects of close relationships experienced a faster 10-year change in executive function (−0.04 standard deviation, 95% confidence interval: −0.08, −0.01) than those in the bottom third, which was comparable with 1 extra year of cognitive decline for participants aged 60 years after adjustment for sociodemographic and health status. Longitudinal analysis found no evidence of reverse causality. This study highlights the importance of differentiating aspects of social relationships to evaluate their unique associations with cognitive aging.