We investigated the relationship of anthropometric markers of obesity with quantitative magnetic resonance imaging markers of brain aging, including measures of total brain volume (TBV), gray matter volume (GMV), hippocampal volume, white matter hyperintensity volume (WMHV), and brain infarcts, and examined causality using Mendelian randomization (MR). Analyses were performed in 1779 individuals (60.4% women, 72.8 ± 4.1 years of age) from the 3C-Dijon population-based cohort study (N = 1555 for the MR). Larger waist-to-hip-ratio (WHR) and waist circumference (WC) were associated with lower TBV (p = 0.0001 and p = 0.005), and lower GMV (p = 0.0008 and p = 0.003), independently of age, gender, body mass index (BMI), and vascular risk factors. Higher BMI, WC, and WHR were associated with larger WMHV and WC with brain infarcts, before adjusting for vascular risk factors only. We used MR to investigate the inverse relationship between WHR and GMV. One valid instrumental variable was available in women only (rs6905288), which was associated with GMV (p = 0.015). Age and BMI-adjusted effect estimates from the MR analysis confirmed the inverse association between GMV and WHR and are in favor of a causal association.