Background To determine whether biological and sociodemographic factors at birth and current factors are associated with insufficient physical activity during leisure among Brazilian adolescents. Methods A school-based cross-sectional study with national coverage was conducted involving Brazilian adolescents 12 to 17 years of age in municipalities with more than 100 thousand residents. The sample consisted of 74,589 adolescents who participated in the Study of Cardiovascular Risk in Adolescents. Insufficient leisure-time physical activity was categorized based on total volume (<300 minutes/week = insufficiently active; >300 minutes/week = sufficiently active). Poisson regression models were used to assess associated factors. Results Most adolescents were classified as insufficiently active (54.8%; 95%CI: 53.7–55.9). The variables associated with insufficient physical activity during leisure were the female sex (70.4%; 95%CI: 68.8–71.9), age between 15 and 17 years (57.8%; 95%CI: 56.3–59.2), pertaining to the low or middle class (54.5%; 95%CI: 52.8–56.1), and not being overweight (55.9%; 95%CI: 54.6–57.1). Conclusion Contrary to our hypothesis, birth related factors (e.g., low birth weight, preterm birth and exclusive breastfeeding until 6 months of age) are not associated with physical inactivity. The prevalence of insufficient physical activity during leisure was high among the adolescents evaluated and was associated with sociodemographic characteristics as well as nutritional status. It is necessary to implement strategies focused on physical activity at schools.