Abstract Background Macrostructural brain alterations in the form of brain atrophy or cortical thinning typically occur during the prodromal Alzheimer’s disease stage. Mixed findings largely dependent on the age of the examined cohorts have been reported during the preclinical, asymptomatic disease stage. In the present study, our aim was to examine the association of midlife dementia risk with brain macrostructural alterations. Methods Structural 3T MRI scans were acquired for 647 cognitively normal middle-aged (40–59 years old) participants in the PREVENT-Dementia study. Cortical thickness, volumes of subcortical structures, the hippocampus and hippocampal subfields were quantified using Freesurfer version 7.1. The clarity of the hippocampal molecular layer was evaluated based on T2-weighted hippocampal scans. Associations of structural measures with apolipoprotein ε4 (APOE4) genotype and dementia family history (FHD), were investigated using linear regression. Correlations between the CAIDE dementia risk score (incorporating information about blood pressure, cholesterol, physical activity, body mass index, education, age and sex) and structural measures were further investigated. Results A higher CAIDE score was associated with thinner cortex and a larger hippocampal fissure. APOE4 genotype was associated with reduced molecular layer clarity. Conclusions Our findings suggest that a higher CAIDE score is associated with widespread cortical thinning. Conversely, APOE4 carriers and participants with FHD do not demonstrate prominent macrostructural alterations at this age range. These findings indicate that cardiovascular and not inherited risk factors for dementia are associated with macrostructural brain alterations at midlife.