White matter hyperintensities (WMH) are associated with cognitive decline. We aimed to identify the spatial specificity of WMH impact on cognition in non-demented, healthy elderly. We quantified WMH volume among healthy participants of a community dwelling cohort ( n = 702, age range 60 – 82 years, mean age = 69.5 years, 46% female) and investigated the effects of WMH on cognition and behavior, specifically for executive function, memory, and motor speed performance. Lesion location influenced their effect on cognition and behavior: Frontal WMH in the proximity of the frontal ventricles mainly affected executive function and parieto-temporal WMH in the proximity of the posterior horns deteriorated memory, while WMH in the upper deep white matter—including the corticospinal tract—compromised motor speed performance. This study exposes the subtle and subclinical yet detrimental effects of WMH on cognition in healthy elderly, and strongly suggests a causal influence of WMH on cognition by demonstrating the spatial specificity of these effects.