Given the increase in the aging population and thus in the prevalence of dementia, the identification of protective factors against cognitive decline is necessary. In a cohort of 1076 non-demented adults ≥ 65 years old (59.7% women) from the HELIAD study, we assessed whether changes in body mass index (BMI) were associated with changes in cognition over a 3-year follow-up period separately for those ≤ 75 and >75 years old. We identified six BMI trajectory groups based on participants’ BMI status at baseline and at the first follow-up visit; normal to normal BMI was the reference group. Major cognitive domains were evaluated, and a composite index reflecting global cognition was calculated. In participants aged ≤75 years, weight loss—moving from obesity to overweight or normal BMI—was associated with less decline in the memory composite score over time (β = 0.141; p = 0.035), while 3-year maintenance of a BMI ≥ 25 kg/m2 was related to greater reduction in the visuospatial composite score over time (β = −0.093; p = 0.020). Regarding participants aged >75 years, 3-year maintenance of a BMI ≥ 30 kg/m2 contributed to a slower rate of decline in the memory composite score over time (β = 0.102; p = 0.042), whereas weight loss—from overweight to normal BMI—was associated with a decreased attention/processing speed composite score longitudinally (β = −0.275; p = 0.043). Our findings indicated that the association between changes in BMI and cognitive functioning was modified by age. Weight management may have the potential to delay cognitive decline in older adults.