<i>Background:</i> Cross-sectional studies have indicated that subcortical ischemic vascular disease (SIVD), as defined according to imaging criteria, is associated with a specific clinical and cognitive profile. Much less is known about the long-term cognitive consequences of SIVD. The aim of the study was to investigate the longitudinal cognitive performance and incident dementia in subjects with and without SIVD in a sample of older adults with white matter lesions. <i>Methods:</i> In the Leukoaraiosis and Disability (LADIS) study, 639 participants were examined with annual clinical and neuropsychological evaluations for 3 years. The subjects meeting the MRI criteria of SIVD at baseline were compared to the other subjects of the sample with linear mixed models. <i>Results:</i> The overall level of cognitive performance over the follow-up period was inferior in multiple cognitive domains in SIVD subjects as compared to the reference group. The subjects with SIVD presented significantly steeper decline of performance in the Stroop test (parts I and II), Trail Making A test, Verbal fluency test, and Mini-Mental State Examination. They also had a threefold risk of developing dementia during follow-up independently of age, sex, education and medial temporal lobe atrophy. <i>Conclusions:</i> SIVD, as a manifestation of cerebral small vessel disease, is related to progressive cognitive impairment and a considerable risk of developing dementia. SIVD seems to specifically contribute to the deterioration of psychomotor speed, executive control, and global cognitive function.