Tumor-associated macrophages (TAMs) and tumor-associated neutrophils (TANs) have been extensively studied. Their pleotropic roles were observed in multiple steps of tumor progression and metastasis, and sometimes appeared to be inconsistent across different studies. In this review, we collectively discussed many lines of evidence supporting the mutual influence between cancer cells and TAMs/TANs. We focused on how direct interactions among these cells dictate co-evolution involving not only clonal competition of cancer cells, but also landscape shift of the entire tumor microenvironment (TME). This co-evolution may take distinct paths and contribute to the heterogeneity of cancer cells and immune cells across different tumors. A more in-depth understanding of the cancer-TAM/TAN co-evolution will shed light on the development of TME that mediates metastasis and therapeutic resistance.