The tumor microenvironment (TME) is comprised of different cellular components, such as immune and stromal cells, which co-operate in unison to promote tumor progression and metastasis. In the last decade, there has been an increasing focus on one specific component of the TME, the stromal component, often referred to as Cancer-Associated Fibroblasts (CAF). CAF modulate the immune response and alter the composition of the extracellular matrix with a decisive impact on the response to immunotherapies and conventional chemotherapy. The most recent publications based on single-cell analysis have underlined CAF heterogeneity and the unique plasticity that strongly impact the TME. In this review, we focus not only on the characterization of CAF based on the most recent findings, but also on their impact on the immune system. We also discuss clinical trials and preclinical studies where targeting CAF revealed controversial results. Therefore, future efforts should focus on understanding the functional properties of individual subtypes of CAF, taking into consideration the peculiarities of each pathological context.