Abstract Background Breast cancer (BC) immune infiltrates play a critical role in tumor progression and response to treatment. Besides stromal tumor infiltrating lymphocytes (sTILs) which have recently reached level 1B evidence as a prognostic marker in triple negative BC, a plethora of methods to assess immune infiltration exists, and it is unclear how these compare to each other and if they can be used interchangeably. Methods Two experienced pathologists scored sTIL, intra-tumoral TIL (itTIL), and 6 immune cell types (CD3⁺, CD4⁺, CD8⁺, CD20⁺, CD68⁺, FOXP3⁺) in the International Cancer Genomics Consortium breast cancer cohort using hematoxylin and eosin-stained (n = 243) and immunohistochemistry-stained tissue microarrays (n = 254) and whole slides (n = 82). The same traits were evaluated using transcriptomic- and methylomic-based deconvolution methods or signatures. Results The concordance correlation coefficient (CCC) between pathologists for sTIL was very good (0.84) and for cell-specific immune infiltrates slightly lower (0.63–0.66). Comparison between tissue microarray and whole slide pathology scores revealed systematically higher values in whole slides (ratio 2.60–5.98). The Spearman correlations between microscopic sTIL and transcriptomic- or methylomic-based assessment of immune infiltrates were highly variable (r = 0.01–0.56). Similar observations were made for cell type-specific quantifications (r = 0.001–0.54). We observed a strong inter-method variability between the omics-derived estimations, which is further cell type dependent. Finally, we demonstrated that most methods more accurately identify highly infiltrated (sTIL ≥ 60%; area under the curve, AUC, 0.64–0.99) as compared to lowly infiltrated tumors (sTIL ≤ 10%; AUC 0.52–0.82). Conclusions There is a lower inter-pathologist concordance for cell-specific quantification as compared to overall infiltration quantification. Microscopic assessments are underestimated when considering small cores (tissue microarray) instead of whole slides. Results further highlight considerable differences between the microscopic-, transcriptomic-, and methylomic-based methods in the assessment of overall and cell-specific immune infiltration in BC. We therefore call for extreme caution when assessing immune infiltrates using current methods and emphasize the need for standardized immune characterization beyond TIL.