Background Problems with health-related quality of life can affect physicians’ ability to work effectively. This study compared the health-related quality of life of Hong Kong physicians to the general population and explored the factors associated with mental and physical health-related quality of life. Methods This cross-sectional study was conducted from January to April 2016. Medical graduates from the University of Hong Kong participated in a survey containing the Short Form-12 Item Health survey version 2, Patient Health Questionnaire-9, Copenhagen Burnout Inventory, and items on lifestyle behaviors, career satisfaction, and socio-demographics. Results 496 responses were received. The mean physical component summary score was 53.2 (SD = 7.6), similar to the general population. The mean mental component summary score was 43.6 (SD = 11.8), significantly worse than the general population (P<0.01). Compared to the general population, all Short-Form 12 Health Survey version 2 domains were worse in doctors, aside from bodily pain and general health. Regular exercise was positively associated with physical component summary scores (Coeff 2.024; P = 0.047); but having children and higher personal burnout scores were negatively associated with it (Coeff -1.890; P = 0.036; and Coeff -0.045; P = 0.027, respectively). Poorer mental component summary scores correlated with worse personal (Coeff -0.284; P< 0.001), work-related (Coeff -0.135; P = 0.040), and patient-related burnout (Coeff -0.060; P = 0.041), and higher Patient Health Questionnaire-9 scores (Coeff -9.170; P<0.001). There were significant differences in mental health (P = 0.042) and mental component summary scores (P = 0.012) across age groups, but not with gender. Conclusion Hong Kong physicians are less impacted by physical health than mental health. Compared to the general population, doctors’ mental health has a more significant impact on their lives. Interventions aimed to improve burnout and depression rates in physicians may improve physicians’ mental health-related quality of life.