Abstract Background Self-reported time-use in relation to health-related quality of life (HRQoL) has been widely studied, yet less is known about the directionality of the association and how it compares across genders when controlling for sociodemographic confounders. Methods This study focused on the working population of the most recent waves (2013–2018) of the Core-Study of the German Socio-Economic Panel (N = 30,518, 46.70% female, M = 39.24 years). It examined the relationship between three time-use categories (contracted, committed, & leisure time) and HRQoL (self-rated health & life satisfaction) in men and women via multigroup fixed effects cross-lagged panel models. The models controlled for sociodemographic background (age, household income, number of children living in household, employment status, education, & marital status), which was associated with time-use and psychosocial health in previous research. Results Contracted time showed consistent positive relationships with HRQoL across genders while associations with the other types of time use differed significantly between men and women and across indicators of HRQoL. Conclusions The way we spend our time directly predicts our health perceptions, but in the same vein our health also predicts how we can spend our time. Contracted time in particular was associated with positive HRQoL, across genders, and beyond sociodemographic predictors, highlighting the important role of employment in health, for men and women alike. The impact of commitments beyond contracted time-use—like household chores and childcare—however, continues to affect mainly women, which ultimately reflects in poorer health outcomes.