Abstract Background The aim of this study was to analyze the association of physical activity and its intensity with arterial stiffness and vascular aging and differences by sex in a Spanish population with intermediate cardiovascular risk. Methods Cross-sectional study. A total of 2475 individuals aged 35–75 years participated in the study. Brachial–ankle pulse wave velocity (baPWV) was measured using a VaSera VS-1500^® device. Based on the age and sex percentile presented by the participants, the latter were classified as follows: those with a percentile above 90 and presenting established cardiovascular disease were classified as early vascular aging (EVA); those with a percentile between 10 and 90 were classified as normal vascular aging (NVA) and those with a percentile below 10 were classified as healthy vascular aging (HVA). Physical activity was analyzed through the short version of the Minnesota Leisure Time Physical Activity Questionnaire (MLTPAQ). Results The mean age of the participants was 61.34 ± 7.70 years, with 61.60% men. Of the total sample, 86% were sedentary (83% men vs 90% women). The total physical activity showed a negative association with baPWV (β = − 0.045; 95% CI − 0.080 to − 0.009). Intense physical activity showed a negative relationship with baPWV (β = − 0.084; 95% CI − 0.136 to − 0.032). The OR of the total physical activity and the intense physical activity carried out by the subjects classified as NVA with respect to those classified as HVA was OR = 0.946; (95% CI 0.898 to 0.997) and OR = 0.903; (95% CI 0.840 to 0.971), and of those classified as EVA it was OR = 0.916; (95% CI 0.852 to 0.986) and OR = 0.905; (95% CI 0.818 to 1.000). No association was found with moderate- or low-intensity physical activity. Conclusions The results of this study suggest that, when intense physical activity is performed, the probability of presenting vascular aging is lower. In the analysis by sex, this association is only observed in men.