The role of the local tumour and stromal immune landscape is increasingly recognised to be important in cancer development, progression and response to therapy. The composition, function, spatial orientation and gene expression profile of the infiltrate of the innate and adaptive immune system at the tumour and surrounding tissue has an established prognostic role in colorectal cancer (CRC). Multiple studies have confirmed that a tumour immune microenvironment (TIME) reflective of a type 1 adaptive immune response is associated with improved prognosis. There have been significant efforts to evolve these observations into validated, histopathology-based prognostic biomarkers, such as the Immunoscore. However, the clinical need lies much more in the development of predictive, not prognostic, biomarkers which have the potential to improve patient outcomes. This is particularly pertinent to help guide cytotoxic chemotherapy use in CRC, which remains the standard of care. Cytotoxic chemotherapy has recognised immunomodulatory activity distinct from its antimitotic effects, including mechanisms such as immunogenic cell death (ICD) and induction/inhibition of key immune players. Response to chemotherapy may differ with regard to molecular subtype of CRC, which are strongly associated with immune phenotypes. Thus, immune markers are potentially useful, though under-reported, predictive biomarkers. In this review, we discuss the impact of the TIME on response to cytotoxic chemotherapy in CRC, with a focus on baseline immune markers, and associated genomic and transcriptomic signatures.