While the clinical importance of CD8+ and CD3+ cells in colorectal cancer (CRC) is well established, the impact of other immune cell subsets is less well described. We sought to provide a detailed overview of the immune landscape of CRC in the largest study to date in terms of patient numbers and in situ analyzed immune cell types. Tissue microarrays from 536 patients were stained using multiplexed immunofluorescence panels, and fifteen immune cell subclasses, representing adaptive and innate immunity, were analyzed. Overall, therapy-naïve CRC patients clustered into an ‘inflamed’ and a ‘desert’ group. Most T cell subsets and M2 macrophages were enriched in the right colon (p-values 0.046–0.004), while pDC cells were in the rectum (p = 0.008). Elderly patients had higher infiltration of M2 macrophages (p = 0.024). CD8+ cells were linked to improved survival in colon cancer stages I-III (q = 0.014), while CD4+ cells had the strongest impact on overall survival in metastatic CRC (q = 0.031). Finally, we demonstrated repopulation of the immune infiltrate in rectal tumors post radiation, following an initial radiation-induced depletion. This study provides a detailed analysis of the in situ immune landscape of CRC paving the way for better diagnostics and providing hints to better target the immune microenvironment.