Left visual neglect is a dramatic neurological condition that impairs awareness of left-sided events. Neglect has been classically reported after strokes in the territory of the right middle cerebral artery. However, the precise lesional correlates of neglect within this territory remain discussed. Recent evidence strongly suggests an implication of dysfunction of large-scale perisylvian networks in chronic neglect, but the quantitative relationships between neglect signs and damage to white matter (WM) tracts have never been explored. In this prospective study, we used diffusion tensor imaging (DTI) tractography in twelve patients with a vascular stroke in the right hemisphere. Six of these patients showed signs of neglect. Nonparametric voxel-based comparisons between neglect and controls on fractional anisotropy maps revealed clusters in the perisylvian WM and in the external capsule. Individual DTI tractography identified specific disconnections of the fronto-parietal and fronto-occipital pathways in the neglect group. Voxel-based correlation statistics highlighted correlations between patients' performance on two visual search tasks and damage to WM clusters. These clusters were located in the anterior limb of the internal capsule and in the WM underlying the inferior frontal gyrus, along the trajectory of the anterior segment of the arcuate fasciculus (asAF). These results indicate that chronic visual neglect can result from, and correlate with, damage to fronto-parietal connections in the right hemisphere, within large-scale cortical networks important for orienting of spatial attention, arousal and spatial working memory.