Mammographic breast density (MD) reflects breast fibroglandular content. Its decline following adjuvant tamoxifen treated, estrogen receptor (ER)-positive breast cancer has been associated with improved outcomes. Breast cancers arise from structures termed lobules, and lower MD is associated with increased age-related lobule involution. We assessed whether pre-treatment involution influenced associations between MD decline and risk of breast cancer-specific death. ER-positive tamoxifen treated patients diagnosed at Kaiser Permanente Northwest (1990–2008) were defined as cases who died of breast cancer (n = 54) and matched controls (remained alive over similar follow-up; n = 180). Lobule involution was assessed by examining terminal duct lobular units (TDLUs) in benign tissues surrounding cancers as TDLU count/mm2, median span and acini count/TDLU. MD (%) was measured in the unaffected breast at baseline (median 6-months before) and follow-up (median 12-months after tamoxifen initiation). TDLU measures and baseline MD were positively associated among controls (p < 0.05). In multivariable regression models, MD decline (≥10%) was associated with reduced risk of breast cancer-specific death before (odds ratio (OR): 0.41, 95% CI: 0.18–0.92) and after (OR: 0.41, 95% CI: 0.18–0.94) adjustment for TDLU count/mm2, TDLU span (OR: 0.34, 95% CI: 0.14–0.84), and acini count/TDLU (OR: 0.33, 95% CI: 0.13–0.81). MD decline following adjuvant tamoxifen is associated with reduced risk of breast cancer-specific death, irrespective of pre-treatment lobule involution.