Abstract Objectives Midlife stressors may be particularly consequential for cognitive performance and disparities in cognitive decline. This study examined Black–White differences in trajectories of cognition among middle-aged adults and the effects of acute and chronic stressors on these trajectories. Methods Data come from 4,011 cognitively healthy individuals aged 51–64 (620 Black and 3,391 White) who participated in the 2006–2018 waves of the Health and Retirement Study. Stressors included a count of recent life events and measures of financial strain and everyday discrimination. Global cognition was assessed using a modified version of the Telephone Interview for Cognitive Status. Linear mixed models with random slopes and intercepts assessed change in cognition over time. Race-by-time, race-by-stressor, time-by-stressor, and race-by-stressor-by-time interactions were assessed as were quadratic terms for time and each stressor. Results After adjusting for sociodemographic, health behaviors, and health-related factors, Black respondents had lower initial cognitive performance scores (b = −1.75, p < .001) but experienced earlier but slower decline in cognitive performance over time (Black × Time2 interaction: b = 0.02, p < .01). Financial strain, discrimination, and recent life events each had distinct associations with cognitive performance but did not influence racial differences in levels of or change in cognition over time. Discussion Middle-aged Black adults have lower initial cognition levels and experience earlier but less accelerated cognitive decline compared to White middle-aged adults. Midlife acute and chronic stressors influence baseline cognition but do so in different ways. Future research should examine the influence of other stressors on racial differences in cognitive decline at other points in the life course.