Although the cross-sectional relationship of arterial stiffness with cerebral small vessel disease is consistently shown in middle-aged and young-old adults, it is less clear whether these associations remain significant over time in very old adults. We hypothesize that arterial stiffness is longitudinally associated with white matter characteristics, and associations are stronger within watershed areas. Neuroimaging was obtained in 2006–2008 from 303 elderly (mean age 82.9 years, 59% women, 41% black) with pulse wave velocity (PWV) measures in 1997–1998. Multivariable regression models estimated the coefficients for PWV (cm/sec) in relationship to presence, severity, and spatial distribution of white matter hyperintensities (WMH), gray matter volume, and fractional anisotropy from diffusion tensor, adjusting for demographic, cardiovascular risk factors, and diseases from 1997–1998 to 2006–2008. Higher PWV in 1997–1998 was associated with greater WMH volume in 2006–2008 within the left superior longitudinal fasciculus (age and total brain WMH adjusted, P =0.023), but not with WMH in other tracts or with fractional anisotropy or gray matter volume from total brain ( P >0.2). Associations were stronger in blacks than in whites, remaining significant in fully adjusted models. Elderly with WMH in tracts related to processing speed and memory are more likely to have had higher PWV values 10 years prior, before neuroimaging data being available. Future studies should address whether arterial stiffness can serve as an early biomarker of covert brain structural abnormalities and whether early arterial stiffness control can promote successful brain aging, especially in black elderly.