Most countries affected by the COVID-19 pandemic have repeatedly restricted public life to control the contagion. However, the health impact of confinement measures is hitherto unclear. We performed a multinational survey investigating changes in mental and physical well-being (MWB/PWB) during the first wave of the pandemic. A total of 14,975 individuals from 14 countries provided valid responses. Compared to pre-restrictions, MWB, as measured by the WHO-5 questionnaire, decreased considerably during restrictions (68.1 ± 16.9 to 51.9 ± 21.0 points). Whereas 14.2% of the participants met the cutoff for depression screening pre-restrictions, this share tripled to 45.2% during restrictions. Factors associated with clinically relevant decreases in MWB were female sex (odds ratio/OR = 1.20, 95% CI: 1.11–1.29), high physical activity levels pre-restrictions (OR = 1.29, 95% CI 1.16–1.42), decreased vigorous physical activity during restrictions (OR = 1.14, 95% CI: 1.05–1.23), and working (partially) outside the home vs. working remotely (OR = 1.29, 95% CI: 1.16–1.44/OR = 1.35, 95% CI: 1.23–1.47). Reductions, although smaller, were also seen for PWB. Scores in the SF-36 bodily pain subscale decreased from 85.8 ± 18.7% pre-restrictions to 81.3 ± 21.9% during restrictions. Clinically relevant decrements of PWB were associated with female sex (OR = 1.62, 95% CI: 1.50–1.75), high levels of public life restrictions (OR = 1.26, 95% CI: 1.18–1.36), and young age (OR = 1.10, 95% CI: 1.03–1.19). Study findings suggest lockdowns instituted during the COVID-19 pandemic may have had substantial adverse public health effects. The development of interventions mitigating losses in MWB and PWB is, thus, paramount when preparing for forthcoming waves of COVID-19 or future public life restrictions.