Abstract Background Physical frailty is associated with increased risk for dementia and other neurologic sequelae. However, the neurobiological changes underlying frailty and frailty risk remain unknown. We examined the association of cerebral white matter structure with current and future frailty. Methods Atherosclerosis Risk in Communities Study Neurocognitive Study participants who underwent 3T brain MRI were included. Frailty status was classified according to the Fried criteria. Cerebral white matter integrity was defined using white matter hyperintensity (WMH) volume and microstructure, measured using diffusion tensor imaging fractional anisotropy (FA) and mean diffusivity (MD). Multivariable linear regression was used to relate baseline frailty to white matter structure; multivariable logistic regression was used to relate baseline white matter to frailty risk among participants nonfrail at baseline. Results In the cross-sectional analysis (N = 1 754; mean age: 76 years), frailty was associated with greater WMH volume, lower FA, and greater MD. These associations remained consistent after excluding participants with a history of stroke or dementia. Among participants nonfrail at baseline who completed follow-up frailty assessment (N = 1 379; 6.6-year follow-up period), each standard deviation increase in WMH volume was associated with 1.46 higher odds of frailty at follow-up. Composite FA and MD measures were not associated with future frailty; however, secondary analyses found several significant white matter tract-specific associations with frailty risk. Conclusion The current study demonstrates a robust association of WMH volume with current and future frailty. Although measures of white matter microstructure were altered in frail individuals, these measures were not generally associated with progression from nonfrail to frail status.