Despite associations of regular coffee consumption with fewer neurodegenerative disorders, its association with microstructural brain alterations is unclear. To address this, we examined the association of coffee consumption with brain MRI parameters representing vascular brain damage, neurodegeneration, and microstructural integrity in 2316 participants in the population-based Hamburg City Health Study. Cortical thickness and white matter hyperintensity (WMH) load were measured on FLAIR and T1-weighted images. Microstructural white matter integrity was quantified as peak width of skeletonized mean diffusivity (PSMD) on diffusion-weighted MRI. Daily coffee consumption was assessed in five groups (<1 cup, 1–2 cups, 3–4 cups, 5–6 cups, >6 cups). In multiple linear regressions, we examined the association between brain MRI parameters and coffee consumption (reference group <1 cup). After adjustment for covariates, 3–4 cups of daily coffee were associated with lower PSMD (p = 0.028) and higher cortical thickness (p = 0.015) compared to <1 cup. Moreover, 1–2 cups per day was also associated with lower PSMD (p = 0.022). Associations with WMH load or other groups of coffee consumption were not significant (p > 0.05). The findings indicate that regular coffee consumption is positively associated with microstructural white matter integrity and cortical thickness. Further research is necessary to determine longitudinal effects of coffee on brain microstructure.