Peer abuse (PA) is a widespread and gender-sensitive form of Adverse Childhood Experiences (ACEs). However, research on its influence on physical and mental health in adulthood remains scarce. The aim of this study was to investigate gender-specific associations between PA and physical and mental health in adulthood in a sample of general hospital patients. A cross-sectional study at the University Hospital of Innsbruck was conducted. Data on ACEs, physical and mental health were collected using self-report questionnaires. We compared patients with no ACEs, PA only, ACEs without PA, and ACEs with PA using gender-specific binary logistic regressions to investigate the association of PA with physical and mental health. A total of 2,392 patients were included in the analyses. Women reported more emotional PA (13.1% vs. 9.4%; p = 0.006), while men reported more physical PA (8.3% vs. 5.2%; p = 0.003). PA was associated with a higher likelihood for depression (OR = 2.6), somatization (OR = 2.1), as well as worse physical health (OR = 2.1) in women but not in men. This study is the first to present data on the gender-specific detrimental effect of PA on physical and mental health in adulthood. Especially for women, PA poses a significant health risk. Thus, we should be aware of these effects and offer adequate support for affected individuals.