Depression and anxiety are two of the most common mental diseases both in formal and nonformal caregivers. Physical activity during leisure time seems to have benefits on their practitioner’s mental health. This study aimed to analyze the associations between physical activity frequency (PAF) and depression and anxiety status, as well as antidepressant and anxiolytic use in Spanish nonformal caregivers. A cross-sectional study with data from the 2014 and 2020 European Health Interview Surveys in Spain (EHSS) including 4520 Spanish nonformal caregivers was carried out. The PAF was found to be related to depression and anxiety, as well as antidepressants and anxiolytics use (p < 0.001), with the highest proportions of these variables found in the inactive population (p < 0.05), while the active and very active populations showed the lowest proportions (p < 0.05). Weak but statistically significant correlations were found between all variables of interest (p < 0.001). Being female, older, and dedicating more hours per week to caregiving and caring for nonfamily members were found to have increased risks of depression, anxiety and antidepressants or anxiolytics use. Nonformal caregivers who were not physically active during their leisure time had higher mental disorders and psychotropic drug use proportions than the active and very active caregivers. Thus, increasing nonformal caregivers’ PAF could be a protective tool.