To reduce transmission of the coronavirus, the Brazilian government adopted containment measures to control the virus’ spread, including limitations to the practice of physical activity. It was aimed to analyze the effects of COVID-19 quarantine on physical activity levels, energy expenditure, quality of life, and level of stress in a sample of the Brazilian population. The sample included 426 participants (7 to 80 years). The International Physical Activity Questionnaire, Short form survey-36 (SF-36), and Stress Perception Scale, were used to assess the level of physical activity, quality of life and stress, respectively. The anthropometric data was used to the assessment of body mass index and basal metabolic rate. Body weight increased significantly in all sample (p < 0.05). During the pandemic, 84% of the sample indicated a low level of physical activity. Furthermore, weekly energy expenditure decreased significantly in all age groups during the pandemic (children p < 0.0001; adolescents: p < 0.0001; adults p < 0.001, and elderly p < 0.0001). All aspects of quality of life, significantly reduced in both sexes in all age groups (p < 0.05). With the exception of children, stress levels increased significantly during the pandemic (adolescents: male: p = 0.003, female: p < 0.05; adults: male: p = 0.003, female: p = 0.01, and elderly: male: p = 0.02, female: p = 0.03). Findings from the present study suggests that COVID-19 social isolation has negatively affected Brazilian’ physical activity and quality of life.