Abstract Background Lower physical activity (PA) is associated with greater perceived fatigability, a person-centered outcome. The association between change in PA and fatigability with advanced age has yet to be established. Methods Community-dwelling older men (N = 1 113, age = 84.1 ± 3.9 years at Year 14) had free-living PA assessed using SenseWear Armband prospectively at Year 7 (2007–2009) and Year 14 (2014–2016) of Osteoporotic Fractures in Men Study, a longitudinal cohort established in 2000 (baseline). We categorized percent changes in PA into groups (large decline → large increase) for 4 metrics: step count, light intensity PA (LIPA, metabolic equivalents [METs] >1.5 to <3.0), moderate-to-vigorous PA (MVPA, METs ≥ 3.0), and sedentary behavior (SB, METs ≤ 1.5, excluding sleep). Perceived physical and mental fatigability were measured (Year 14) with the Pittsburgh Fatigability Scale (PFS, higher score = greater fatigability; range = 0–50). Associations between each metric of percent changes in PA and fatigability were examined using linear regression, adjusted for demographics, change in health conditions, and Year 7 step count or total PA (METs > 1.5). Results Men declined 2 336 ± 2 546 (34%) steps/d, 24 ± 31 (25%) LIPA min/d, 33 ± 58 (19%) MVPA min/d, and increased 40 ± 107 (6%) SB min/d over 7.2 ± 0.7 years. Compared to large decline (% change less than −50%), those that maintained or increased step count had 3–8 points lower PFS Physical scores; those who maintained or increased LIPA and MVPA had 2–3 and 2–4 points lower PFS Physical scores, respectively (all p ≤ .01). Associations were similar, but smaller, for PFS Mental scores. Conclusion Older men who maintained or increased PA had lower fatigability, independent of initial PA. Our findings inform the types and doses of PA that should be targeted to reduce fatigability in older adults.