Depression is a profound public health concern, yet its etiology remains unclear. A body’s magnesium status and low-grade systemic inflammation are associated with depression. However, the interaction of magnesium status and inflammation on depression/depressive symptoms is unknown. We assessed the association between serum magnesium levels and depressive symptoms by analyzing data from the Nutrition and Health Survey in Taiwan 2005–2008. In total, 2196 participants aged ≥20 years were included. Depressive symptoms were assessed using the 5-item Brief-Symptom Rating Scale. We performed logistic regression and multiple linear regression analyses to examine the association. A dose–response analysis was performed using restricted cubic spline models, and stratification by chronic inflammation was also performed. We found that higher serum magnesium levels were associated with lower depression scores and a lower risk of depression. In the subgroup analysis, serum magnesium levels were inversely associated with depressive symptoms more prominently among people with higher CRP levels, with a threshold at 5 mg/L (≥5 vs. <5) showing a greater difference than at 3 mg/L (≥3 vs. <3). Conclusions: Serum magnesium levels were inversely associated with depressive symptoms. This inverse association was affected by inflammation level. A dose–response relationship was also observed.