With the impact of the COVID-19 pandemic, the number of patients suffering from depression is rising around the world. It is important to diagnose depression early so that it may be treated as soon as possible. The self-response questionnaire, which has been used to diagnose depression in hospitals, is impractical since it requires active patient engagement. Therefore, it is vital to have a system that predicts depression automatically and recommends treatment. In this paper, we propose a smartphone-based depression prediction system. In addition, we propose depressive features based on multimodal sensor data for predicting depressive mood. The multimodal depressive features were designed based on depression symptoms defined in the Diagnostic and Statistical Manual of Mental Disorders (DSM-5). The proposed system comprises a “Mental Health Protector” application that collects data from smartphones and a big data-based cloud platform that processes large amounts of data. We recruited 106 mental patients and collected smartphone sensor data and self-reported questionnaires from their smartphones using the proposed system. Finally, we evaluated the performance of the proposed system’s prediction of depression. As the test dataset, 27 out of 106 participants were selected randomly. The proposed system showed 76.92% on an f1-score for 16 patients with depression disease, and in particular, 15 patients, 93.75%, were successfully predicted. Unlike previous studies, the proposed method has high adaptability in that it uses only smartphones and has a distinction of evaluating prediction accuracy based on the diagnosis.