Elevated glucocorticoid (GC) levels are hypothesized to be deleterious to some brain regions, including white matter (WM). Older age is accompanied by increased between-participant variation in GC levels, yet relationships between WM integrity and cortisol levels in older humans are underexplored. Moreover, it is unclear whether GC-WM associations might be general or pathway specific. We analyzed relationships between salivary cortisol (diurnal and reactive) and general measures of brain WM hyperintensity (WMH) volume, fractional anisotropy (gFA), and mean diffusivity (gMD) in 90 males, aged 73 years. Significant associations were predominantly found between cortisol measures and WMHs and gMD but not gFA. Higher cortisol at the start of a mild cognitive stressor was associated with higher WMH and gMD. Higher cortisol at the end was associated with greater WMHs. A constant or increasing cortisol level during cognitive testing was associated with lower gMD. Tract-specific bases of these associations implicated anterior thalamic radiation, uncinate, and arcuate and inferior longitudinal fasciculi. The cognitive sequelae of these relationships, above other covariates, are a priority for future study.