The Apolipoprotein E (APOE) ɛ4 allele is the best-established genetic risk factor for sporadic Alzheimer's disease, and is also associated with structural gray matter and functional brain changes in healthy young, middle-aged and elderly subjects. Because APOE is implicated in brain mechanisms associated with white matter (WM) development and repair, we investigated the potential role played by the APOE polymorphism on WM structure in healthy younger (aged 20-35 years) and older (aged 50-78 years) adults using diffusion tensor imaging. General reduction of fractional anisotropy and increase in mean diffusivity values was found in carriers of the APOE ɛ4 allele relative to non-carriers. No significant interactions between genotype and age were observed, suggesting that differences in WM structure between APOE ɛ4-carriers and non-carriers do not undergo significant differential changes with age. This result was not explained by differences in brain morphology or cognitive measures. The APOE ɛ4 allele modulates brain WM structure before any clinical or neurophysiological expression of impending disease.