Cerebral small vessel disease, including white matter hyperintensities (WMH) and lacunes of presumed vascular origin, is common in elderly people and is related to cognitive impairment and dementia. One possible mechanism could be the disruption of white matter tracts (both within WMH and normal-appearing white matter) that connect distributed brain regions involved in cognitive functions. Here, we investigated the relation between microstructural integrity of the white matter and cognitive functions in patients with small vessel disease. The Radboud University Nijmegen Diffusion tensor and Magnetic resonance Cohort study is a prospective cohort study among 444 independently living, non-demented elderly with cerebral small vessel disease, aged between 5500 and 85 years. All subjects underwent magnetic resonance imaging and diffusion tensor imaging scanning and an extensive neuropsychological assessment. We showed that loss of microstructural integrity of the white matter at specific locations was related to specific cognitive disturbances, which was mainly located in the normal-appearing white matter (p < 0.05, FWE-corrected for multiple comparisons). The microstructural integrity in the genu and splenium showed the highest significant relation with global cognitive function and executive functions, in the cingulum bundle with verbal memory performance. Associations between diffusion tensor imaging parameters and most cognitive domains remained present after adjustment for WMH and lacunes. In conclusion, cognitive disturbances in subjects with cerebral small vessel disease are related to microstructural integrity of multiple white matter fibers (within WMH and normal-appearing white matter) connecting different cortical and subcortical regions.