The COVID-19 pandemic has had a strong influence on people’s quality of life, and the different restrictive measures during the phases of the pandemic have had consequences for physical and mental health. This study determined the changes in health-related quality of life (HRQoL), sleep quality, depression symptoms (DS), and physical activity (PA) level of middle-aged Colombian university personnel during the COVID-19 pandemic. The analysis was performed following a longitudinal design on a sample of 354 people at four points before and during the pandemic. The International Physical Activity Questionnaire (IPAQ), the SF-12v2, the Pittsburgh Sleep Quality Index (PSQI), and the Zung Self-Rating Depression Scale (ZSDS) were included in an online survey. The measurements showed a decrease in quality of life with respect to PCS from 91.66 ± 10.78 to 83.64 ± 17.22 (p = 0.000) and MCS from 87.57 ± 12.23 to 80.90 ± 17.31, while poor sleep quality increased from 25.99% to 47.46% (p = 0.000), with DS reaching the peak during mandatory confinement (14.69%). Regarding the level of physical activity, the period of mandatory confinement inverted the proportion, highlighting active people as a minority (32.2%). In the multivariate analysis, we adjusted for age, sex, BMI, and socioeconomic level, discarding confounding effects and their interactions with the results obtained. In conclusion, this study showed that the pandemic has had a negative impact on sleep quality, HRQoL, DS, and PA level.