Introduction: Inflammatory breast cancer (IBC) is a rare but aggressive form of breast cancer (BC) in which the (prognostic) role of stromal tumour-infiltrating lymphocytes (sTIL) and the peripheral circulating immune cells in patients with residual disease (RD) after neo-adjuvant chemotherapy (NACT) is not clearly established. Methodology: To describe the evolution of sTIL and some peripheral inflammation markers (Neutrophil-to-lymphocyte ratio, Platelet-to-lymphocyte ratio and Lymphocyte-to-monocyte ratio) after NACT in IBC, we retrospectively collected clinicopathological variables for 125 stage III IBC patients. sTILs were scored by three different researchers on an H&E slide of the mastectomy specimen. A cohort of subtype-matched non-IBC breast cancer patients (nIBC) treated with NACT was included for comparison. Results: There was no significant difference in the pre- and posttreatment sTIL scores between IBC and nIBC and in both groups the number of sTIL was significantly lower after NACT. However, the IBC phenotype did correlate with a stronger decrease of sTIL after NACT (OR: 0.25, 95% CI: 0.073–0.76, p = 0.018). The change in the peripheral immune markers was not significantly different between IBC and nIBC. After NACT, 75 patients had residual disease. In this group, a high number of sTIL before NACT (HR: 0.23, 95% CI: 0.05–1.02, p = 0.05) was prognostic for a longer OS, while a low number of sTIL after NACT (HR: 0.33, 95% CI: 0.11–0.98, p = 0.046) and a low residual cancer cellularity (HR: 0.20, 95% CI: 0.08–0.52, p < 0.001) was associated with a longer DFS. Conclusions: IBC is associated with a significantly stronger decrease of sTIL after NACT compared to nIBC. Furthermore, a high number of sTIL after NACT was associated with a worse prognosis in IBC.