Background—Patients affected by Long COVID often report shorter times spent doing physical activity (PA) after COVID-19. The aim of the present study was to evaluate potential associations between PA levels and parameters of physical function in a cohort of COVID-19 survivors with post-acute symptoms, with a particular focus on individuals aged 65 and older. Materials and methods—PA levels before and after COVID-19 were assessed in a sample of patients that had recovered from COVID-19 and were admitted to a post-acute outpatient service at the Fondazione Policlinico Universitario Agostino Gemelli IRCCS (Rome, Italy). Participation in PA was operationalized as the engagement in leisure-time PA for at least 150 min per week in the last 3 months. Self-rated health and measures of physical performance and muscle strength were assessed. Results—Mean age of 1846 participants was 55.2 ± 14.4 years and 47% were women. Before COVID-19, inactivity was detected in 47% of the whole study population; only 28% maintained pre-COVID-19 PA engagement. Inactivity was more frequent in women. The stopping of physical activity was associated with increased BMI and CRP levels, lower vitamin D levels and a higher prevalence of post-COVID-19 fatigue, dyspnea, arthralgia, and myalgia. Active participants had higher handgrip strength and performed better on both the six-minute walking test (6MWT) and at the one-minute sit-to-stand test (1MSTST). In particular, at the 6MWT, participants 65 and older that were still active after COVID-19 walked 32 m more than sedentary peers. Moreover, the distance covered was 28 m more than those who were active only before COVID-19 (p = 0.05). Formerly active subjects performed similarly at the 6MWT to inactive participants. PA was associated with better self-rated health. Conclusions—Our findings reveal that inactivity is frequent in the post-acute COVID-19 phase. Stopping physical activity after COVID-19 results in measures of performance that are comparable to those who were never active. Relevant differences in the distance covered at the 6MWT were found between older active subjects and their sedentary peers.