Increased performance variability has been demonstrated in several groups and conditions, including aging and cognitive decline. Structural brain characteristics underlying this phenomenon have so far been elusive. However, there is reason to expect that disconnectivity in associative pathways, whether caused by immature or degraded white matter (WM) tracts, will increase performance variability by neural noise. The aim of this study was to test whether the quality of WM, measured by diffusion tensor imaging, is related to performance variability in healthy adults. Intraindividual standard deviation of the reaction time (sdRT) across trials and median reaction time (mRT) from 270 participants were obtained from a speeded continuous performance task (Eriksen flanker task) with two conditions (congruent, incongruent). Tract-based spatial statistics was used to test the relationship with diffusion characteristics [fractional anisotropy (FA), mean diffusion (MD), radial diffusion (RD), axial diffusion (AD)]. Robust relationships between sdRT and all diffusion measures were found in most WM areas, independently of mRT, age, and sex. The effects were anatomically more widespread in the congruent than the incongruent condition, covering almost 50% of the voxels for RD and MD, and >25% of the voxels for FA and AD. Partial betas were in the range 0.45–0.55, and the strength of the relationships increased significantly with age. For mRT, the effects were smaller and unstable across condition. We concluded that performance variability is a likely consequence of individual differences in WM integrity, and that it is a promising behavioral correlate of individual differences in WM microstructure.