Background Cerebral autosomal dominant arteriopathy with subcortical infarcts and leukoencephalopathy (CADASIL) is the most prevalent monogenic cerebral small‐vessel disease. Phenotype variability in CADASIL suggests the possible role of genetic modifiers. We aimed to investigate the contributions of the APOE genotype and Neurogenic locus notch homolog protein 3 ( NOTCH3 ) variant position to cognitive impairment associated with CADASIL. Methods and Results Patients with the cysteine‐altering NOTCH3 variant were enrolled in a cross‐sectional study, including the Mini‐Mental State Examination (MMSE), brain magnetic resonance imaging, and APOE genotyping. Cognitive impairment was defined as an MMSE score <24. The associations between the MMSE score and genetic factors were assessed using linear regression models. Bayesian adjustment for confounding was used to identify clinical confounders. A total of 246 individuals were enrolled, among whom 210 (85%) harbored the p.R544C variant, 96 (39%) had cognitive impairment, and 150 (61%) had a history of stroke. The APOE ɛ2 allele was associated with a lower MMSE score (adjusted B , −4.090 [95% CI, −6.708 to −1.473]; P =0.023), whereas the NOTCH3 p.R544C variant was associated with a higher MMSE score (adjusted B , 2.854 [95% CI, 0.603–5.105]; P =0.0132) after adjustment for age, education, and history of ischemic stroke. Mediation analysis suggests that the associations between the APOE ɛ2 allele and MMSE score and between the NOTCH3 p.R544C variant and MMSE score are mediated by mesial temporal atrophy and white matter hyperintensity, respectively. Conclusions APOE genotype may modify cognitive impairment in CADASIL, whereby individuals carrying the APOE ɛ2 allele may present a more severe cognitive impairment.