Abstract Alzheimer’s disease is a highly heritable, common neurodegenerative disease characterized neuropathologically by the accumulation of β-amyloid plaques and tau-containing neurofibrillary tangles. In addition to the well-established risk associated with the APOE locus, there has been considerable success in identifying additional genetic variants associated with Alzheimer’s disease. Major challenges in understanding how genetic risk influences the development of Alzheimer’s disease are clinical and neuropathological heterogeneity, and the high level of accompanying comorbidities. We report a multimodal analysis integrating longitudinal clinical and cognitive assessment with neuropathological data collected as part of the Brains for Dementia Research study to understand how genetic risk factors for Alzheimer’s disease influence the development of neuropathology and clinical performance. Six hundred and ninety-three donors in the Brains for Dementia Research cohort with genetic data, semi-quantitative neuropathology measurements, cognitive assessments and established diagnostic criteria were included in this study. We tested the association of APOE genotype and Alzheimer’s disease polygenic risk score—a quantitative measure of genetic burden—with survival, four common neuropathological features in Alzheimer’s disease brains (neurofibrillary tangles, β-amyloid plaques, Lewy bodies and transactive response DNA-binding protein 43 proteinopathy), clinical status (clinical dementia rating) and cognitive performance (Mini-Mental State Exam, Montreal Cognitive Assessment). The APOE ε4 allele was significantly associated with younger age of death in the Brains for Dementia Research cohort. Our analyses of neuropathology highlighted two independent pathways from APOE ε4, one where β-amyloid accumulation co-occurs with the development of tauopathy, and a second characterized by direct effects on tauopathy independent of β-amyloidosis. Although we also detected association between APOE ε4 and dementia status and cognitive performance, these were all mediated by tauopathy, highlighting that they are a consequence of the neuropathological changes. Analyses of polygenic risk score identified associations with tauopathy and β-amyloidosis, which appeared to have both shared and unique contributions, suggesting that different genetic variants associated with Alzheimer’s disease affect different features of neuropathology to different degrees. Taken together, our results provide insight into how genetic risk for Alzheimer’s disease influences both the clinical and pathological features of dementia, increasing our understanding about the interplay between APOE genotype and other genetic risk factors.