Background: The COVID-19 pandemic has had substantial impacts on citizens’ daily living. Concerns over mental health issues are rising. Recent studies assessing the psychosocial impact of COVID-19 on the general public revealed alarming results. Meanwhile, the impact of the COVID-19 pandemic on mental health among patients with pre-existing psychiatric disorders remained unclear. Methods: Patients diagnosed with anxiety disorders, depressive disorders, bipolar disorders, or schizophrenia were invited to complete a survey between July and October 2020. The survey collected information on subjects’ demographics, accommodation status, changes in mental health status during the COVID-19 outbreak, and the factors that affect subjects’ mental health during COVID-19. The primary outcome of this study was the change in mental health, defined by psychiatric symptom change and patient satisfaction on symptom control. The secondary outcomes were patients’ emotional status—measured by the Depression, Anxiety and Stress Scale (DASS-21)—during the COVID-19 pandemic and factors that impacted patients’ mental health during the COVID-19 pandemic. Results: Out of the 294 patients recruited, 65.0% were living in hostel while 35.0% were living in the community. The proportion of patients with ‘unsatisfied’ or ‘very unsatisfied’ mental disease control increased from 10.2% to 17.1% after the COVID-19 outbreak (p < 0.001). Under the DASS-21 questionnaire, 24.2% subjects, 32.6% subjects, and 18.9% subjects were classified as severe or extremely severe in terms of the level of depression, anxiety, and stress they experienced, respectively. Patients living in the community, patients with mood disorders, and female patients reported significantly worse control over anxiety and mood symptoms. The three major factors that affected patients’ mental health during COVID-19 were ‘reduced social activities’, ‘worries over people around getting infected’, and ‘reduced exercise’. Conclusion: Psychiatric patients in general have poorer disease control after the COVID-19 outbreak. Patients in the community appeared to be more affected than patients residing in hostels. More efforts should be directed to screening patients with pre-existing mental health disorders to enable timely interventions.