Heterogeneity in breast carcinomas can be appreciated at various levels, from morphology to molecular alterations, and there are well-known genotypic-phenotypic correlations. Clinical decision-making is strictly focused on the evaluation of tumor cells and is based on the assessment of hormone receptors and of the HER2 status, by means of a combination of immunohistochemical and in situ hybridization techniques. The tumor microenvironment (TME) also shows a multifaceted nature stemming from the different actors populating the intratumoral and the peritumoral stroma of breast carcinomas. Of note, we have now evidence that tumor-infiltrating lymphocytes (TILs) are clinically meaningful as their quantification in the intratumoral stroma strongly correlates with good prognosis, in particular in triple-negative and HER2-positive breast cancer patients. Nevertheless, TILs are just one of the many actors orchestrating the complexity of the TME, which is populated by immune and non-immune cells (cancer-associated fibroblasts, cancer-associated adipocytes), as well as non-cellular components such as chemical inflammation mediators. In this review article we will overview the main features of the distinct cell compartments by discussing (i) the potential impact the TME may have on the prognostic stratification of breast cancers and (ii) the possible predictive value of some markers in the context of immunotherapy in light of the recent results of phase III studies in advanced and early triple-negative breast cancer patients.