Sporting events were cancelled, and sports training was banned to prevent the spread of COVID-19. These changes during the COVID-19 pandemic decreased the physical activity levels, increased sedentary time, and also impaired the mental health of elite and sub-elite athletes. The impact on body composition and physical performance is not clear, however, especially considering a systematic review with meta-analysis. Thus, our objective was to conduct a review in accordance with the PRISMA Statement studies published in scientific journals (PubMed, Web of Science, or Scopus databases) that investigated the effect that social distancing during the COVID-19 pandemic had on the physical performance (muscle power, cardiorespiratory capacity, and sprint) or body composition (body weight, percentage of fat, fat mass, and fat-free mass) of athletes. Data from 24 studies indicate that, throughout the global lockdown, the athletes maintained muscle power, cardiorespiratory capacity, and sprint, and prevented significant changes in fat mass and fat-free mass. However, the total body weight (meta-analysis with 18 studies), showed a significant increase (p = 0.006), with a small ES = 0.12; 95% CI = 0.04 to 0.21. Furthermore, the time of follow-up, level of training, and the age of the athletes were possible moderators of these effects. The data reinforce the importance of general strength and endurance exercises sessions to maintain physical fitness during non-competitive periods or due to the mandatory lockdown.