Abstract Background Lower physical activity has been cross-sectionally associated with greater perceived fatigability, defined as self-reported fatigue anchored to activity intensity and duration. The temporality of this relationship, and whether it differs by activity type or marital status, remains unclear. Methods In the Osteoporotic Fractures in Men Study (N = 1 759), self-reported total, exercise, and household activity were assessed using the Physical Activity Scale for the Elderly across 7 visits (2000–2016). The Pittsburgh Fatigability Scale (range: 0–50; higher scores = greater fatigability) measured physical (mean = 16.6 ± 9.7) and mental (mean = 7.8 ± 8.3) fatigability at Year 14. Least absolute deviation and linear regression were used to examine associations between baseline and change in activity over 14 years with subsequent fatigability. Models were adjusted for demographic, health, and lifestyle factors. Results After adjustment, lower baseline (β= −0.08, 95% confidence interval [CI]: −0.12, −0.04) and greater annual declines in total activity (β = −0.09, 95% CI: −0.14, −0.05) were prospectively associated with higher Pittsburgh Fatigability Scale (PFS) Physical scores. Associations were similar for mental fatigability (both p < .05). Lower baseline leisure exercise, but not baseline household activity, predicted higher PFS Physical scores (β = −0.10 vs −0.001). In contrast, greater declines in household activity, but not declines in exercise, were associated with higher PFS Physical scores (β = −0.09 vs −0.03). Lower baseline household activity predicted higher PFS Mental scores only for unmarried men (β = −0.15, 95% CI: −0.29, −0.01, interaction p = .019). Conclusions Baseline total activity and leisure exercise, and declines in total and household activity, were associated with higher subsequent perceived fatigability in older men. Marital status may mitigate the contribution of household activity to subsequent fatigability.