Background There is broad consensus that regular physical activity yields major health benefits. However, current guidelines on physical activity are mainly aimed at middle-aged adults. It is unclear whether physical activity also translates into cardiovascular health benefits in older adults. Therefore, we aimed to compare the association between different levels of physical activity and the risk of cardiovascular disease (CVD) in elderly to middle-aged individuals. Methods We analysed data from the EPIC Norfolk prospective population study. Cox proportional hazards models were used to analyse the association between physical activity levels and time to CVD events in three age categories (<55, 55–65 and >65 years). Interaction between age categories and physical activity levels was assessed. Results Analyses were based on 24,502 study participants aged 39–79 years. A total of 5240 CVD events occurred during 412,954 person-years follow-up (median follow-up was 18.0 years). Among individuals aged over 65 years, hazard ratios for CVD were 0.86 (95% confidence interval (CI) 0.78–0.96), 0.87 (95% CI 0.77–0.99) and 0.88 (95% CI 0.77–1.02) in moderately inactive, moderately active and active people, respectively, compared to inactive people. Among people aged 55–65 and less than 55 years, the associations were directionally similar, but not statistically significant. The interaction term between physical activity levels and age categories was not significant ( P = 0.38). Conclusion The inverse association between physical activity and the risk of CVD was significant in elderly and comparable with middle-aged individuals. In addition, we observed that modest levels of physical activity confer benefits in terms of CVD risk, compared to being completely inactive.