The apolipoprotein E4 (ApoE4) is an established risk factor for Alzheimer's disease (AD). Previous work has shown that this allele is associated with functional (fMRI) changes as well structural grey matter (GM) changes in healthy young, middle-aged and older subjects. Here, we assess the diffusion characteristics and the white matter (WM) tracts of healthy young (20–38 years) ApoE4 carriers and non-carriers. No significant differences in diffusion indices were found between young carriers (ApoE4+) and non-carriers (ApoE4−). There were also no significant differences between the groups in terms of normalised GM or WM volume. A feature selection algorithm (ReliefF) was used to select the most salient voxels from the diffusion data for subsequent classification with support vector machines (SVMs). SVMs were capable of classifying ApoE4 carrier and non-carrier groups with an extremely high level of accuracy. The top 500 voxels selected by ReliefF were then used as seeds for tractography which identified a WM network that included regions of the parietal lobe, the cingulum bundle and the dorsolateral frontal lobe. There was a non-significant decrease in volume of this WM network in the ApoE4 carrier group. Our results indicate that there are subtle WM differences between healthy young ApoE4 carriers and non-carriers and that the WM network identified may be particularly vulnerable to further degeneration in ApoE4 carriers as they enter middle and old age.