Colorectal cancer remains one of the most important health challenges in our society. The development of cancer immunotherapies has fostered the need to better understand the anti-tumor immune mechanisms at play in the tumor microenvironment and the strategies by which the tumor escapes them. In this review, we provide an overview of the molecular interactions that regulate tumor inflammation. We particularly discuss immunomodulatory cell-cell interactions, cell-soluble factor interactions, cell-extracellular matrix interactions and cell-microbiome interactions. While doing so, we highlight relevant examples of tumor immunomodulation in colorectal cancer.