Prostate cancer (PCa) accounts as the most common non-cutaneous disease affecting males, and as the first cancer, for incidence, in male. With the introduction of the concept of immunoscore, PCa has been classified as a cold tumor, thus driving the attention in the development of strategies aimed at blocking the infiltration/activation of immunosuppressive cells, while favoring the infiltration/activation of anti-tumor immune cells. Even if immunotherapy has revolutionized the approaches to cancer therapy, there is still a window failure, due to the immune cell plasticity within PCa, that can acquire pro-tumor features, subsequent to the tumor microenvironment (TME) capability to polarize them. This review discussed selected relevant soluble factors [transforming growth factor-beta (TGFβ), interleukin-6 (IL-6), IL-10, IL-23] and cellular components of the innate immunity, as drivers of tumor progression, immunosuppression, and angiogenesis within the PCa-TME.