Depression is a leading cause of the global burden of disease and has a multifactorial etiology that includes nutrients. Magnesium status has been associated with depression with inconclusive results. The impact of chronic latent magnesium deficiency (CLMD, 0.75 ≤ serum magnesium < 0.85 mmol/L) on depression has not yet been investigated. We assessed the association between serum magnesium levels/dietary magnesium intake and depressive symptoms by analyzing nationally representative data from Taiwan (Nutrition and Health Survey in Taiwan, NAHSIT). We used the 5-item Brief Symptom Rating Scale to measure depressive symptoms. Subgroup analysis by sex was also performed. Serum magnesium levels had a low correlation with dietary magnesium intake. Higher serum magnesium levels were associated with lower depressive scores and a lower risk of depressive symptoms, but dietary magnesium intake showed no association. Sex differences were found. Compared with subjects with serum magnesium <0.75 mmol/L, those with ≥0.85 mmol/L had lower depressive scores. In conclusion, serum magnesium was inversely associated with depressive symptoms, but dietary magnesium intake was not. Subjects with CLMD showed similar depressive scores and were at a similar risk of depressive symptoms to those with serum magnesium < 0.75 mmol/L. CLMD should be considered while assessing the association between magnesium status and depressive symptoms.