Background and Purpose— The mechanisms by which leukoariosis impacts on clinical and cognitive functions are not yet fully understood. We hypothesized that ultrastructural abnormalities of the normal-appearing brain tissue (NABT) assessed by diffusion-weighted imaging played a major and independent role. Methods— In addition to a comprehensive clinical, neuropsychologic, and imaging work-up, diffusion-weighted imaging was performed in 340 participants of the multicenter leukoariosis and disability study examining the impact of white matter hyperintensities (WMH) on 65- to 85-year old individuals without previous disability. WMH severity was rated according to the Fazekas score. Multivariate regression analysis served to assess correlations of histogram metrics of the apparent diffusion coefficient (ADC) of whole-brain tissue, NABT, and of the mean ADC of WMH with cognitive functions. Results— Increasing WMH scores were associated with a higher frequency of hypertension, a greater WMH volume, more brain atrophy, worse overall cognitive performance, and changes in ADC. We found strong associations between the peak height of the ADC histogram of whole-brain tissue and NABT with memory performance, executive dysfunction, and speed, which remained after adjustment for WMH lesion volume and brain atrophy and were consistent among centers. No such association was seen with the mean ADC of WMH. Conclusions— Ultrastructural abnormalities of NABT increase with WMH severity and have a strong and independent effect on cognitive functions, whereas diffusion-weighted imaging metrics within WMH have no direct impact. This should be considered when defining outcome measures for trials that attempt to ameliorate the consequences of WMH progression.