The physical activity (PA) and inactivity of family caregivers of cancer patients were investigated and related to burden and quality of life through a cross-sectional multicentre study. A total of 75 caregivers were recruited from June 2020 to March 2021. The levels of PA and inactivity were estimated with a wrist accelerometer, 24 h a day, for 7 consecutive days. The Quality of Life Family Version, the Caregiver Strain Index, the total duration of care, the average number of hours spent in care, and the assistance received were registered. Our results showed that moderate-to-vigorous PA was 96.40 ± 46.93 min/day, with 90.7% of participants performing more than 150 min/week of physical activity, and this was significantly associated with age (r = −0.237). Daily inactivity was 665.78 ± 94.92 min, and inactivity for 20–30 min was significantly associated with caregiver burden (r = 0.232) and quality of life (r = −0.322). Compliance with the World Health Organization recommendations was significantly associated with a lower quality of life (r = −0.269). The strength of these associations was limited (r ~0.2). In conclusion, the PA performed by most caregivers met the established recommendations, although older caregivers (>65 years old) performed lower moderate-to-vigorous PA than younger ones. In addition, the mean inactive time was high (11 h/day), showing slight relationships with the burden and quality of life of caregivers.