Genetic susceptibility and lifestyle affect the risk of dementia but there is little direct evidence for their associations with preclinical changes in brain structure. We investigated the association of genetic dementia risk and healthy lifestyle with brain morphometry, and whether effects from elevated genetic risk are modified by lifestyle changes. We used prospective data from up to 25,894 UK Biobank participants (median follow-up of 8.8 years), and defined healthy lifestyle according to American Heart Association criteria as BMI < 30, no smoking, healthy diet and regular physical activity). Higher genetic risk was associated with lower hippocampal volume (beta −0.16 cm3, 95% CI −0.22, −0.11) and total brain volume (−4.34 cm3, 95% CI −7.68, −1.01) in participants aged ≥60 years but not <60 years. Healthy lifestyle was associated with higher total brain, grey matter and hippocampal volumes, and lower volume of white matter hyperintensities, with no effect modification by age or genetic risk. In conclusion, adverse effects of high genetic risk on brain health were only found in older participants, while adhering to healthy lifestyle recommendations is beneficial regardless of age or genetic risk.