Abstract Background Given the high heterogeneity among breast tumors, associations between common germline genetic variants and survival that may exist within specific subgroups could go undetected in an unstratified set of breast cancer patients. Methods We performed genome-wide association analyses within 15 subgroups of breast cancer patients based on prognostic factors, including hormone receptors, tumor grade, age, and type of systemic treatment. Analyses were based on 91,686 female patients of European ancestry from the Breast Cancer Association Consortium, including 7531 breast cancer-specific deaths over a median follow-up of 8.1 years. Cox regression was used to assess associations of common germline variants with 15-year and 5-year breast cancer-specific survival. We assessed the probability of these associations being true positives via the Bayesian false discovery probability (BFDP < 0.15). Results Evidence of associations with breast cancer-specific survival was observed in three patient subgroups, with variant rs5934618 in patients with grade 3 tumors (15-year-hazard ratio (HR) [95% confidence interval (CI)] 1.32 [1.20, 1.45], P = 1.4E−08, BFDP = 0.01, per G allele); variant rs4679741 in patients with ER-positive tumors treated with endocrine therapy (15-year-HR [95% CI] 1.18 [1.11, 1.26], P = 1.6E−07, BFDP = 0.09, per G allele); variants rs1106333 (15-year-HR [95% CI] 1.68 [1.39,2.03], P = 5.6E−08, BFDP = 0.12, per A allele) and rs78754389 (5-year-HR [95% CI] 1.79 [1.46,2.20], P = 1.7E−08, BFDP = 0.07, per A allele), in patients with ER-negative tumors treated with chemotherapy. Conclusions We found evidence of four loci associated with breast cancer-specific survival within three patient subgroups. There was limited evidence for the existence of associations in other patient subgroups. However, the power for many subgroups is limited due to the low number of events. Even so, our results suggest that the impact of common germline genetic variants on breast cancer-specific survival might be limited.