Purpose To compare the diffusion tensor imaging (DTI) measures of multiple sclerosis (MS) patients and healthy subjects in every brain voxel and to correlate them with Paced Auditory Serial Addition Test (PASAT) scores. Materials and Methods Fractional anisotropy (FA), and mean, longitudinal, and transverse diffusivity are compared between control subjects and MS patients, which were subdivided as mildly and moderately impaired. In addition, PASAT scores are correlated for both MS groups with the diffusion measures. An optimized voxel based analysis (VBA) method, in terms of coregistration, atlas construction, and image smoothing, was thereby used. Results: Diffusion differences between the control subjects and the patients with MS were found in the corpus callosum, inferior longitudinal fasciculus, cortico spinal tracts, forceps major, superior longitudinal fasciculus, and cingulum. In addition, we observed significant correlations of the FA and PASAT scores in the left inferior longitudinal fasciculus, the forceps minor, the capsula interna and externa, the genu of the corpus callosum, the left cingulum, the superior longitudinal fasciculus, and the corona radiata. Conclusion: Diffusion differences were observed between the mildly impaired MS patients and control subjects. In addition, different diffusion measures correlated with PASAT scores for cognitive decline in parietal, frontal, as well as temporal white matter (WM) regions.