Shear-wave elastography (SWE) is an effective tool in discriminating malignant lesions of breast and axillary lymph node metastasis in patients with breast cancer. However, the association between the baseline elasticity value of breast cancer and the treatment response of neoadjuvant chemotherapy is yet to be elucidated. Baseline SWE measured mean stiffness (E-mean) and maximum stiffness (E-max) in 830 patients who underwent neoadjuvant chemotherapy and surgery from January 2012 to December 2022. Association of elasticity values with breast pCR (defined as ypTis/T0), pCR (defined as ypTis/T0, N0), and tumor-infiltrating lymphocytes (TILs) was analyzed. Of 830 patients, 356 (42.9%) achieved breast pCR, and 324 (39.0%) achieved pCR. The patients with low elasticity values had higher breast pCR and pCR rates than those with high elasticity values. A low E-mean (adjusted odds ratio (OR): 0.620; 95% confidence interval (CI): 0.437 to 0.878; p = 0.007) and low E-max (adjusted OR: 0.701; 95% CI: 0.494 to 0.996; p = 0.047) were independent predictive factors for breast pCR. Low elasticity values were significantly correlated with high TILs. Pretreatment elasticity values measured using SWE were significantly associated with treatment response and inversely correlated with TILs, particularly in HR+HER2- breast cancer and TNBC.