(1) Background: People with HIV (PWH) may perform more than one type of exercise cumulatively. The objective of this study is to investigate recreational exercise and its association with health-related quality of life (HRQOL) and comorbidities in relation to potential covariates. (2) Methods: The HIBES study (HIV-Begleiterkrankungen-Sport) is a cross-sectional study for people with HIV. The differences between non-exercisers versus exercisers (cumulated vs. single type of exercises) were investigated using regression models based on 454 participants. (3) Results: Exercisers showed a higher HRQOL score compared to non-exercisers (Wilcox r = 0.2 to 0.239). Psychological disorders were identified as the main covariate. Participants performing exercise cumulatively showed higher scores in duration, frequency, and intensity when compared to participants performing only one type of exercise. The mental health summary score was higher for the cumulated and single type of exercise if a psychological disorder existed. Duration and intensity were associated with an increase of HRQOL, whilst a stronger association between psychological disorders and exercise variables were evident. Exercise duration (minutes) showed a significant effect on QOL (standardized beta = 0.1) and for participants with psychological disorders (standardized beta = 0.3), respectively. (4) Conclusions: Psychological disorders and other covariates have a prominent effect on HRQOL and its association with exercise. For PWH with a psychological disorder, a stronger relationship between HRQOL with exercise duration and intensity emerged. However, differentiation of high-HRQOL individuals warrants further investigation by considering additional factors.