Sex-specific differences have been increasingly recognized in many human diseases including brain cancer, namely glioblastoma. Primary CNS lymphoma (PCNSL) is an exceedingly rare type of brain cancer that tends to have a higher incidence and worse outcomes in male patients. Yet, relatively little is known about the reasons that contribute to these observed sex-specific differences. Using a population-representative cohort of patients with PCNSL with dense magnetic resonance (MR) imaging and digital pathology annotation (n = 74), we performed sex-specific cluster and survival analyses to explore possible associations. We found three prognostically relevant clusters for females and two for males, characterized by differences in (i) patient demographics, (ii) tumor-associated immune response, and (iii) MR imaging phenotypes. Upon a multivariable analysis, an enhanced FoxP3+ lymphocyte-driven immune response was associated with a shorter overall survival particularly in female patients (HR 1.65, p = 0.035), while an increased extent of contrast enhancement emerged as an adverse predictor of outcomes in male patients (HR 1.05, p < 0.01). In conclusion, we found divergent prognostic constellations between female and male patients with PCNSL that suggest differential roles of tumor-associated immune response and MR imaging phenotypes. Our results further underline the importance of continued sex-specific analyses in the field of brain cancer.