Purpose CNS-directed chemotherapy (CT) and cranial radiotherapy (CRT) for childhood acute lymphoblastic leukemia or lymphoma have various neurotoxic properties. This study aimed to assess their impact on the maturing brain 20 to 30 years after diagnosis, providing a much stronger perspective on long-term quality of life than previous studies. Patients and Methods Ninety-three patients treated between 1978 and 1990 at various intensities, with and without CRT, and 49 healthy controls were assessed with magnetic resonance diffusion tensor imaging (DTI) and neuropsychological tests. Differences in fractional anisotropy (FA)—a DTI measure describing white matter (WM) microstructure—were analyzed by using whole brain voxel-based analysis. Results CRT-treated survivors demonstrated significantly decreased FA compared with controls in frontal, parietal, and temporal WM tracts. Trends for lower FA were seen in the CT-treated survivors. Decreases in FA correlated well with neuropsychological dysfunction. In contrast to the CT group and controls, the CRT group showed a steep decline of FA with age at assessment. Younger age at cranial irradiation and higher dosage were associated with worse outcome of WM integrity. Conclusion CRT-treated survivors show decreased WM integrity reflected by significantly decreased FA and associated neuropsychological dysfunction 25 years after treatment, although effects of CT alone seem mild. Accelerated aging of the brain and increased risk of early onset dementia are suspected after CRT, but not after CT.