Due to the dramatic impact of the COVID-19 pandemic, Spain underwent a strict lockdown (March–May 2020). How the lockdown modified older adults’ physical activity (PA) has been poorly described. This research assesses the effect of the lockdown on PA levels and identifies predictors of sufficient/insufficient PA in frail older community-dwellers. Community-dwelling participants from the +ÀGIL Barcelona frailty intervention program, suspended during the pandemic, underwent a phone-assessment during the lockdown. PA was measured before and after the lockdown using the Brief Physical Activity Assessment Tool (BPAAT). We included 98 frail older adults free of COVID-19 (mean age = 82.7 years, 66.3% women, mean Short Physical Performance Battery = 8.1 points). About one third of participants (32.2%) were not meeting sufficient PA levels at the end of the lockdown. Depressive symptoms (OR = 0.12, CI95% = 0.02–0.55) and fatigue (OR = 0.11, CI95% = 0.03–0.44) decreased the odds of maintaining sufficient PA, whereas maintaining social networks (OR = 5.07, CI95% = 1.60–16.08) and reading (OR = 6.29, CI95% = 1.66–23.90) increased it. Living alone was associated with the reduction of PA levels (b = −1.30, CI95% = −2.14–−0.46). In our sample, pre-lockdown mental health, frailty-related symptoms and social relationships were consistently associated with both PA levels during-lockdown and pre-post change. These data suggest considering specific plans to maintain PA levels in frail older community-dwellers.