Background: Fatigue in multiple sclerosis (MS) has been inconsistently associated with disruption of specific brain circuitries. Temporal fluctuations of fatigue have not been considered. Objective: The aim of this study was to investigate the association of fatigue with brain diffusion abnormalities, using robust criteria for patient stratification based on longitudinal patterns of fatigue. Methods: Patient stratification: (1) sustained fatigue (SF, n = 26): latest two Modified Fatigue Impact Scale (MFIS) ⩾ 38; (2) reversible fatigue (RF, n = 25): latest MFIS < 38 and minimum one previous MFIS ⩾ 38; and (3) never fatigued (NF, n = 42): MFIS always < 38 (five assessments minimum). 3T brain magnetic resonance imaging (MRI) was used to perform voxel-wise comparison of fractional anisotropy (FA) between the groups controlling for age, sex, disease duration, physical disability, white matter lesion load (T2LV), and depression. Results: SF and, to a lesser extent, RF patients showed lower FA in multiple brain regions compared to NF patients, independent of age, sex, disease duration, and physical disability. In cingulo-postcommissural-striato-thalamic regions, the differences in FA between SF and NF (but not between RF and NF or SF) patients were independent of T2LV, and in ventromedial prefronto-precommissuro-striatal and temporo-insular areas, independent of T2LV and depression. Conclusion: Damage to ventromedial prefronto-precommissuro-striatal and temporo-insular pathways appears to be a specific substrate of SF in MS.