Abstract Our understanding is limited concerning the tumor immune microenvironment of inflammatory breast cancer (IBC), an aggressive form of primary cancer with low rates of pathologic complete response to current neoadjuvant chemotherapy (NAC) regimens. We retrospectively identified pretreatment (N = 86) and matched posttreatment tissue (N = 27) from patients with stage III or de novo stage IV IBC who received NAC followed by a mastectomy. Immune profiling was performed including quantification of lymphoid and myeloid infiltrates by IHC and T-cell repertoire analysis. Thirty-four of 86 cases in this cohort (39.5%) achieved a pathologic complete response. Characterization of the tumor microenvironment revealed that having a lower pretreatment mast cell density was significantly associated with achieving a pathologic complete response to NAC (P = 0.004), with responders also having more stromal tumor-infiltrating lymphocytes (P = 0.035), CD8+ T cells (P = 0.047), and CD20+ B cells (P = 0.054). Spatial analysis showed close proximity of mast cells to CD8+ T cells, CD163+ monocytes/macrophages, and tumor cells when pathologic complete response was not achieved. PD-L1 positivity on tumor cells was found in fewer than 2% of cases and on immune cells in 27% of cases, but with no correlation to response. Our results highlight the strong association of mast cell infiltration with poor response to NAC, suggesting a mechanism of treatment resistance and a potential therapeutic target in IBC. Proximity of mast cells to immune and tumor cells may suggest immunosuppressive or tumor-promoting interactions of these mast cells.