Abstract Background Previous studies have suggested that the association between APOE ɛ 4 and dementia is moderated by physical activity (PA), but the results remain inconclusive and longitudinal data on cognitive decline are missing. In this study, we examine whether there is a gene–environment interaction between APOE and PA on cognitive decline in older adults using 9-year follow-up data of three cohort studies. Methods We followed 7,176 participants from three longitudinal cohort studies: Longitudinal Aging Study Amsterdam (LASA), InCHIANTI, and Rotterdam Study for 9 years. PA was assessed with self-reported questionnaires and was categorized in low, moderate, and high PA. Cognitive function was assessed with the Mini-Mental State Examination (MMSE) and cognitive decline was defined as a decrease of three points or more on the MMSE during 3 years follow-up. We fitted logistic regression models using generalized estimating equations adjusting for age, sex, education, depressive symptoms, and number of chronic disease. Interaction between APOE and PA was tested on multiplicative and additive scale. Results Cohorts were similar in most aspects but InCHIANTI participants were on average older and had lower education. APOE ɛ 4 carriers had higher odds of cognitive decline (odds ratio [OR] = 1.46, 95% confidence interval [CI]: 1.29–1.64) while PA was not significantly associated with cognitive decline overall (moderate PA: OR = 0.87, 0.67–1.13; high PA: OR = 0.71, 0.36–1.40). There was no evidence for an interaction effect between PA and APOE ɛ 4 in cognitive decline in older adults (APOE × moderate PA: p = .83; APOE × high PA: p = .90). Conclusions Previous claims of a gene–environment interaction between APOE ɛ 4 and PA in cognitive decline are not supported by our results.