Breast cancer is one of the most common female malignancies worldwide. Due to its early metastases formation and a high degree of malignancy, the 10 year-survival rate of metastatic breast cancer does not exceed 30%. Thus, more precise biomarkers are urgently needed. In our study, we first estimated the tumor microenvironment (TME) infiltration using the xCell algorithm. Based on TME infiltration, the three main TME clusters were identified using consensus clustering. Our results showed that the three main TME clusters cause significant differences in survival rates and TME infiltration patterns (log-rank test, p = 0.006). Then, multiple machine learning algorithms were used to develop a nine-pathway-based TME-related risk model to predict the prognosis of breast cancer (BRCA) patients (the immune-related pathway-based risk score, defined as IPRS). Based on the IPRS, BRCA patients were divided into two subgroups, and patients in the IPRS-low group presented significantly better overall survival (OS) rates than the IPRS-high group (log-rank test, p < 0.0001). Correlation analysis revealed that the IPRS-low group was characterized by increases in immune-related scores (cytolytic activity (CYT), major histocompatibility complex (MHC), T cell-inflamed immune gene expression profile (GEP), ESTIMATE, immune, and stromal scores) while exhibiting decreases in tumor purity, suggesting IPRS-low patients may have a strong immune response. Additionally, the gene-set enrichment analysis (GSEA) result confirmed that the IPRS-low patients were significantly enriched in several immune-associated signaling pathways. Furthermore, multivariate Cox analysis revealed that the IPRS was an independent prognostic biomarker after adjustment by clinicopathologic characteristics. The prognostic value of the IPRS model was further validated in three external validation cohorts. Altogether, our findings demonstrated that the IPRS was a powerful predictor to screen out certain populations with better prognosis in breast cancer and may serve as a potential biomarker guiding clinical treatment decisions.