Abstract Background Consumption of a prudent dietary pattern rich in healthy nutrients is associated with enhanced cognitive performance in older adulthood, while a Western dietary pattern low in healthy nutrients is associated with poor age-related cognitive function. Sex differences exist in dietary intake among older adults; however, there is a paucity of research examining the relationship between sex-specific dietary patterns and cognitive function in later life. Methods The current study aimed to investigate sex differences in the relationship between sex-specific dietary pattern adherence and global cognitive function at baseline and over a 3-year follow-up in 1268 community-dwelling older adults (M_age = 74 years, n = 664 women, n = 612 men) from the Quebec Longitudinal Study on Nutrition and Successful Aging (NuAge). A 78-item Food Frequency Questionnaire was used to estimate dietary intake over the previous year. Sex-specific dietary pattern scores were derived using principal component analysis. Global cognition was assessed using the Modified Mini-Mental State Examination (3MS). Results Adjusted linear mixed effects models indicated that a healthy, prudent dietary pattern was not associated with baseline cognitive performance in men or women. No relationship was found between Western dietary pattern adherence and baseline cognitive function in women. Among men, adherence to an unhealthy, Western dietary pattern was associated with poorer baseline cognitive function (β = − 0.652, p = 0.02, 95% CI [− 1.22, − 0.65]). No association was found between prudent or Western dietary patterns and cognitive change over time in men or women. Conclusions These findings highlight the importance of conducting sex-based analyses in aging research and suggest that the relationship between dietary pattern adherence and cognitive function in late life may be sex-dependent.