Background: Diabetes is an important public health problem due to its health impairments and high costs for health services. We analyzed the relationship between the domains of physical activity at leisure-time (LTPA) and at commuting (CPA) with diabetes and pre-diabetes in an ELSA-Brasil study. Methods: Data from 11,797 participants (52.5% women, 49.1 ± 7.2 years) were analyzed. LTPA and CPA were measured using the International Physical Activity Questionnaire. Diabetes and pre-diabetes were defined by medical history, medication use to treat diabetes or blood glucose. Logistic regression models were performed to estimate the association between LTPA and CPA with diabetes and pre-diabetes after adjustment for sociodemographic and cardiovascular risk factors. Results: The prevalence of LTPA and CPA was 24.4% and 34%, respectively. Physically active participants at LTPA were less likely to have pre-diabetes (OR = 0.86 [95% CI = 0.77–0.95]) and diabetes (OR = 0.80 [95% CI = 0.69–0.93]), compared with inactive participants. No association between CPA and diabetes/pre-diabetes was observed. LTPA was inversely associated with diabetes among men (OR = 0.73 [95% CI = 0.60–0.89]), but was not associated among women. Women who were active (OR = 0.78 [95% CI = 0.67–0.90]) (OR = 0.79 [95% CI = 0.65–0.95]) at LTPA were less likely to have pre-diabetes, than inactive women. Conclusion: LTPA was inversely associated with diabetes and pre-diabetes in the ELSA-Brasil participants. A different behavior was observed between genders.