Background Genetic factors are important in determining breast density, and heritable factors account for 60% of the variation. Certain single nucleotide polymorphisms (SNPs) are associated with density and risk of breast cancer but the association with prognosis is not clear. Purpose To investigate associations between selected SNPs and breast cancer survival in the Malmö Diet and Cancer Study (MDCS). Material and Methods A total of 724 unrelated women with breast cancer and registered radiological and pathological data were identified in MDCS 1991–2007, with genotyping available for 672 women. Associations among 15 SNPs, density, and breast cancer-specific survival were analyzed using logistic/Cox regression, adjusted for factors affecting density and survival. Variants significantly associated with either density or survival were validated in a large independent breast cancer cohort (LIBRO-1). Results Minor homozygotes of SNPs rs9383589, CCDC170 and rs6557161, ESR1 were associated with high breast density (adjusted odds ratio [AOR] 8.97, 95% confidence interval [CI] 1.35–59.57; AOR 2.08, 95% CI 1.19–3.65, respectively) and poorer breast cancer survival (adjusted hazard ratio [HR_adj] 6.46, 95% CI 1.95–21.39; HR_adj 2.30, 95% CI 1.33–3.96, respectively) compared to major homozygotes. For SNP rs3757318, ESR1, minor homozygotes (HR_adj 7.46, 95% CI 2.28–24.45) were associated with poorer survival. We confirmed that rs6557161, ESR1 was significantly associated with both density and survival in the LIBRO-1 study. Conclusion These findings support a shared genetic basis for density and breast cancer survival. The SNP significantly associated with both density and survival in both cohorts may be of interest in future research investigating polygenic risk scores for breast cancer risk and screening stratification purposes.