Few studies have examined the role of BAG2 in malignancies. We investigated the prognostic value of BAG2-expression in cancer-associated fibroblasts (CAFs) and tumor cells in predicting metastasis-free survival in patients with breast cancer. Tissue-microarray was constructed using human breast cancer tissues obtained by surgical resection between 1992 and 2015. BAG2 expression was evaluated by immunohistochemistry in CAFs or the tumor cells. BAG2 expression in the CAFs and cytoplasm of tumor cells was classified as positive and negative, and low and high, respectively. BAG2-CAF was evaluated in 310 patients and was positive in 67 (21.6%) patients. Kaplan–Meier plots showed that distant metastasis-free survival (DMFS) was lesser in patients with BAG2(+) CAF than in patients with BAG2(−) CAF (p = 0.039). Additionally, we classified the 310 patients into two groups: 109 in either BAG2-high or BAG2(+) CAF and 201 in BAG2-low and BAG2(−) CAF. DMFS was significantly reduced in patients with either BAG2-high or BAG2(+) CAF than in the patients of the other group (p = 0.005). Multivariable analysis demonstrated that DMFS was prolonged in patients with BAG2(−) CAF or BAG2-low. Evaluation of BAG2 expression on both CAFs and tumor cells could help in determining the risk of metastasis in breast cancer.