Background We investigated whether cardiac parameters in young adulthood are associated with indicators of brain health in midlife. Methods and Results This study includes 648 participants from the CARDIA (Coronary Artery Risk Development in Young Adults) study (52% women, 38% black). We studied associations of cardiac parameters assessed by echocardiography (left ventricular ejection fraction, left atrial volume, and left ventricular mass) in young adulthood (mean age: 30 years) with brain measures obtained by magnetic resonance imaging (total brain, gray and white matter volume, white matter integrity, abnormal white matter) in midlife (mean age: 50 years). In 406 individuals with complete measurements, higher left atrial volume was associated with lower white matter fractional anisotropy, independent of traditional cardiovascular risk factors (β=−0.002; P <0.02). The association was strongest in black participants and in men. Conclusions Higher left atrial volume in early adulthood is associated with impairment of white matter integrity in midlife. Interventions to improve cardiac function in young adults may benefit brain health and should be targeted in particular at black men.