Abstract Objective: To examine the association between leisure activity (LA) frequency and cognitive trajectories over 5 years across adulthood, and whether gender and age moderate these associations. Method: A total of 234 cognitively healthy adults (21–80 years) completed a LA questionnaire at baseline and neuropsychological measures at baseline and after 5 years. Latent change score analysis was applied to generate latent variables estimating changes in different cognitive domains. For a secondary analysis, LA components’ scores were calculated, reflecting cognitive-intellectual, social, and physical activities. Regression analysis examined the association between baseline LA and cognitive change, and potential moderation of gender and age. In addition, we tested the influence of cortical gray matter thickness on the results. Results: We found that higher LA engagement was associated with slower cognitive decline for reasoning, speed, and memory, as well as better vocabulary across two time points. Regarding LA components, higher Social-LA and Intellectual-LA predicted slower rates of cognitive decline across different domains, while Physical-LA was not associated with cognitive change. Gender, but not age, moderated some of the associations observed. Our results remained the same after controlling for cortical gray matter thickness. Conclusions: We demonstrated a protective effect of LA engagement on cognitive trajectories over 5 years, independent from demographics and a measure of brain health. The effects were in part moderated by gender, but not age. Results should be replicated in larger and more diverse samples. Our findings support cognitive reserve hypothesis and have implications for future reserve-enhancing interventions.