(1) Background: We aim to assess the time trend from 2014 to 2020 in the prevalence of physical activity (PA), identify gender differences and sociodemographic and health-related factors associated with PA among people with diabetes, and compare PA between people with and without diabetes. (2) Methods: We conducted a cross-sectional and a case–control study using as data source the European Health Interview Surveys for Spain (EHISS) conducted in years 2014 and 2020. The presence of diabetes and PA were self-reported. Covariates included socio-demographic characteristics, health-related variables, and lifestyles. To compare people with and without diabetes, we matched individuals by age and sex. (3) Results: The number of participants aged ≥18 years with self-reported diabetes were 1852 and 1889 in the EHISS2014 and EHISS2020, respectively. The proportion of people with diabetes that had a medium or high frequency of PA improved from 48.3% in 2014 to 52.6% in 2020 (p = 0.009), with 68.5% in 2014 and 77.7% in 2020 being engaged in two or more days of PA (p < 0.001). Males with diabetes reported more PA than females with diabetes in both surveys. After matching by age and gender, participants with diabetes showed significantly lower engagement in PA than those without diabetes. Among adults with diabetes, multivariable logistic regression showed confirmation that PA improved significantly from 2014 to 2020 and that male sex, higher educational level, and better self-rated health were variables associated to more PA. However, self-reported comorbidities, smoking, or BMI > 30 were associated to less PA. (4) Conclusions: The time trend of PA among Spanish adults with diabetes is favorable but insufficient. The prevalence of PA in this diabetes population is low and does not reach the levels of the general population. Gender differences were found with significantly more PA among males with diabetes. Our result could help to improve the design and implementation of public health strategies to improve PA among people with diabetes.