The Epstein–Barr virus (EBV) can cause a wide variety of cancers upon infection of different cell types and induces a highly variable composition of the tumor microenvironment (TME). This TME consists of both innate and adaptive immune cells and is not merely an aspecific reaction to the tumor cells. In fact, latent EBV-infected tumor cells utilize several specific mechanisms to form and shape the TME to their own benefit. These mechanisms have been studied largely in the context of EBV+ Hodgkin lymphoma, undifferentiated nasopharyngeal carcinoma, and EBV+ gastric cancer. This review describes the composition, immune escape mechanisms, and tumor cell promoting properties of the TME in these three malignancies. Mechanisms of susceptibility which regularly involve genes related to immune system function are also discussed, as only a small proportion of EBV-infected individuals develops an EBV-associated malignancy.