There is a global ageing of the world’s population. Ageing is associated with multiple pathologies, reductions in physical activity, and losses in cognitive function. This study aimed to analyse the associations between the frequency of leisure-time physical activity (PAF) in middle-aged Spaniards and subjective cognitive limitations (SCLs): self-reported problems for remembering or concentrating (data extracted from the 2017 National Health Survey and the 2020 European Health Survey in Spain). Furthermore, the study aimed to evaluate risk factors that could be related to a higher probability of developing SCLs. This was a cross-sectional study with 15,866 middle-aged Spaniards. The associations between FAP and SCLs were analysed using chi-square. Also, the risk factors for SCLs were evaluated using binary multiple logistic regression. The median age of participants was 55 years, with 49% men and 51% women. Associations were found between PAF and SCLs (p < 0.001). The highest prevalence of SCLs was found in physically inactive people and the lowest in very active people (13.7% vs. 5.8%, p < 0.05), and people with SCLs had a higher prevalence of inactivity than those without SCLs (47.2% vs. 33.8%, p < 0.05). Physical inactivity, low educational level, low social class, and being female were the main risk factors for SCLs. Among the actions to prevent cognitive limitations, as well as interventions in people with cognitive limitations, it would be advisable to include physical activity programmes, both as a preventive measure to delay cognitive limitations and to reduce the risk of other pathologies in people who already have them.