Background Atrial fibrillation (AF) is associated with increased stroke risk and accelerated cognitive decline, but the association of early manifestations of left atrial (LA) impairment with subclinical changes in brain structure is unclear. We investigated whether abnormal LA structure and function, greater supraventricular ectopy, and intermittent AF are associated with small vessel disease on magnetic resonance imaging of the brain. Methods and Results In the Multi‐Ethnic Study of Atherosclerosis, 967 participants completed 14‐day ambulatory electrocardiographic monitoring, speckle tracking echocardiography and, a median 17 months later, magnetic resonance imaging of the brain. We assessed associations of LA volume index and reservoir strain, supraventricular ectopy, and prevalent AF with brain magnetic resonance imaging measures of small vessel disease and atrophy. The mean age of participants was 72 years; 53% were women. In multivariable models, LA enlargement was associated with lower white matter fractional anisotropy and greater prevalence of microbleeds; reduced LA strain, indicating worse LA function, was associated with more microbleeds. More premature atrial contractions were associated with lower total gray matter volume. Compared with no AF, intermittent AF (prevalent AF with <100% AF during electrocardiographic monitoring) was associated with lower white matter fractional anisotropy (−0.25 SDs [95% CI, −0.44 to −0.07]) and greater prevalence of microbleeds (prevalence ratio: 1.42 [95% CI, 1.12–1.79]). Conclusions In individuals without a history of stroke or transient ischemic attack, alterations of LA structure and function, including enlargement, reduced strain, frequent premature atrial contractions, and intermittent AF, were associated with increased markers of small vessel disease. Detailed assessment of LA structure and function and extended ECG monitoring may enable early identification of individuals at greater risk of small vessel disease.