Cognitive deficits are common in glioma patients following multimodality therapy, but the relative impact of different types and locations of treatment-related brain damage and recurrent tumors on cognition is not well understood. In 121 WHO Grade III/IV glioma patients, structural MRI, O-(2-[18F]fluoroethyl)-L-tyrosine FET-PET, and neuropsychological testing were performed at a median interval of 14 months (range, 1–214 months) after therapy initiation. Resection cavities, T1-enhancing lesions, T2/FLAIR hyperintensities, and FET-PET positive tumor sites were semi-automatically segmented and elastically registered to a normative, resting state (RS) fMRI-based functional cortical network atlas and to the JHU atlas of white matter (WM) tracts, and their influence on cognitive test scores relative to a cohort of matched healthy subjects was assessed. T2/FLAIR hyperintensities presumably caused by radiation therapy covered more extensive brain areas than the other lesion types and significantly impaired cognitive performance in many domains when affecting left-hemispheric RS-nodes and WM-tracts as opposed to brain tissue damage caused by resection or recurrent tumors. Verbal episodic memory proved to be especially vulnerable to T2/FLAIR abnormalities affecting the nodes and tracts of the left temporal lobe. In order to improve radiotherapy planning, publicly available brain atlases, in conjunction with elastic registration techniques, should be used, similar to neuronavigation in neurosurgery.