Abstract Background: Early-life and adult anthropometrics are associated with breast density and breast cancer risk. However, little is known about whether these factors also influence breast tissue composition beyond what is captured by breast density among women with benign breast disease (BBD). Methods: This analysis included 788 controls from a nested case–control study of breast cancer within the Nurses' Health Study BBD subcohorts. Body fatness at ages 5 and 10 years was recalled using a 9-level pictogram. Weight at age 18, current weight, and height were reported via questionnaires. A deep-learning image analysis was used to quantify the percentages of epithelial, fibrous stromal, and adipose tissue areas within BBD slides. We performed linear mixed models to estimate beta coefficients (β) and 95% confidence intervals (CI) for the relationships between anthropometrics and the log-transformed percentages of individual tissue type, adjusting for confounders. Results: Childhood body fatness (level ≥ 4.5 vs. 1), BMI at age 18 (≥23 vs. <19 kg/m2), and current adult BMI (≥30 vs. <21 kg/m2) were associated with higher proportions of adipose tissue [β (95% CI) = 0.34 (0.03, 0.65), 0.19 (−0.04–0.42), 0.40 (0.12, 0.68), respectively] and lower proportions of fibrous stromal tissue [−0.05 (−0.10, 0.002), −0.03 (−0.07, 0.003), −0.12 (−0.16, −0.07), respectively] during adulthood (all Ptrend < 0.04). BMI at age 18 was also inversely associated with epithelial tissue (Ptrend = 0.03). Adult height was not associated with any of the individual tissue types. Conclusions: Our data suggest that body fatness has long-term impacts on breast tissue composition. Impact: This study contributes to our understanding of the link between body fatness and breast cancer risk. See related commentary by Oskar et al., p. 590