Abstract Human longevity is moderately heritable and is hence influenced by both genetic and environmental factors. However, there remains considerable uncertainty regarding its relationship with brain aging. Here, we used a discovery sample (N = 19 136, aged 45–81 years) from the UK Biobank and a replication sample (N = 809, aged 66–93 years) from the Sydney Memory and Ageing Study and the Older Australian Twins Study to investigate the associations between both parental life span (parental age at death) and polygenic risk score (PRS) for longevity (longevity-PRS) and structural magnetic resonance imaging brain metrics, which are considered to reflect the brain aging process, namely white matter hyperintensities (WMHs), total gray matter, and cortical volumes. We found lower volumes of WMHs to be significantly associated with longer parental life span in the discovery (whole WMH, β = −0.0323, padj = .0002) and replication samples (whole WMH, β = −0.0871, padj = .0208) and higher longevity-PRS in the discovery sample (whole WMH, β = −0.0331, padj = .0015) and a similar trend in the replication sample (significant before multiple comparison adjustment). The association of longevity-PRS with WMH remained significant after removing the influence of the apolipoprotein E locus (whole WMH, β = −0.0297, padj = .0048). While total gray matter and cortical volumes were related to parental life span in the discovery sample, they were not significantly associated with longevity-PRS. Additionally, the effects of longevity-PRS on the association were more prominent in males. Our findings suggest that enrichment of longevity-related alleles may provide some protection against WMH burden and highlight the important aspect of genetic relationship between longevity and WMH.