Abstract Background Aggression in psychiatric hospitals has been of interest to researchers. Information on how different stakeholders perceive patient aggression remains equivocal. Even less is known about possible similarities or differences in stakeholders’ perceptions of how aggressive behaviour is understood, managed and prevented in psychiatric hospitals. We aimed to explore multiple viewpoints on patient aggression, its possible causes and outcomes, and development ideas for prevention and management. Methods A qualitative design was adopted. The data were collected using focus group interviews. A thematic approach was used for interpretation. The data were collected on 15 adult wards in two inpatient psychiatric settings in Hong Kong. Participants were nurses working on the psychiatric inpatient wards, patients admitted to the wards, and informal caregivers visiting inpatient wards (N = 94). Results Commonalities between all groups were found on how patient aggression is perceived, and why it occurs. Patients and especially nurses described how patient aggression occurred with no clear reason or forewarning and how patients were physically controlled or restricted after aggressive events. Only nurses and patients expressed experiencing physical burden, while all groups considered psychological burden to be a consequence of aggression. All groups proposed that helpful attitudes among nurses, better communication, structural changes, and better self-management skills would prevent patient aggression. Risk assessment was proposed only by nurses and patients, while safety measures were proposed by nurses and informal caregivers only. The use of restrictive interventions to manage aggressive events was proposed by all groups. Conclusions Despite the complex diversity of perspectives in different stakeholder groups regarding patient aggression, the findings highlighted that it is possible to achieve some mutual understanding of aggression in psychiatric hospitals and identify areas to be developed. Staffs’ attitudes and skills for engagement and communication with patients and informal caregivers should be improved. There is also still room to develop the therapeutic environment and culture toward meaningful activities during the treatment period.