While studies have documented the influence of caregiver and care recipient factors on caregiver health, it is important to address the potential impact of neighborhood contexts. This study estimated the cross-sectional associations between neighborhood characteristics and mental health among caregivers cohabiting with Alzheimer’s disease care recipients that were experiencing severe or non-severe neuropsychiatric symptoms (NPSs) (e.g., aggression/anxiety). We obtained data collected in 2010 on caregivers and care recipients (n = 212) from a subset of South Carolina’s Alzheimer’s Disease Registry. Neighborhood measures (within 1 mile of the residence) came from the American Community Survey and the Rural-Urban Commuting Area Code. We categorized the neighborhood median household income into tertiles, namely, “low” (<$31,000), “medium” ($31,000–40,758), and “high” (>$40,758), and rurality as “large urban,” “small urban,” and “rural.” We used negative binomial regression to estimate the prevalence ratios (PRs) and 95% confidence intervals (CIs) for caregiver mental health using neighborhood characteristics. The mean age was 58 ± 10.3 years, 85% were women, and 55% were non-Hispanic Black. Among the caregivers cohabiting with a recipient experiencing severe NPS, higher distress was experienced by caregivers living in low- (PR = 1.61 (95% CI = 1.26–2.04)) and medium- (PR = 1.45 (95% CI = 1.17–1.78)) vs. high-income neighborhoods after an adjustment. These results suggest that neighborhood characteristics may amplify other social stressors experienced by caregivers.