Experimental studies have shown that dioxin-like chemicals may interfere with aspects of the endocrine system including growth. However, human background population studies are, however, scarce. We aimed to investigate whether early exposure of healthy infants to dioxin-like chemicals was associated with changes in early childhood growth and serum IGF1. In 418 maternal breast milk samples of Danish children (born 1997–2001) from a longitudinal cohort, we measured polychlorinated dibenzo-p-dioxins, polychlorinated dibenzofurans, and polychlorinated biphenyls (pg or ng/g lipid) and calculated total toxic equivalent (total TEQ). SDS and SDS changes over time (ΔSDS) were calculated for height, weight, BMI, and skinfold fat percentage at 0, 3, 18, and 36 months of age. Serum IGF1 was measured at 3 months. We adjusted for confounders using multivariate regression analysis. Estimates (in parentheses) correspond to a fivefold increase in total TEQ. TEQ levels in breast milk increased significantly with maternal age and fish consumption and decreased with maternal birth year, parity, and smoking. Total TEQ was associated with lower fat percentage (−0.45 s.d., CI: −0.89; −0.04), non-significantly with lower weight and length at 0 months, accelerated early height growth (increased ΔSDS) (ΔSDS 0–18 months: +0.77 s.d., CI: 0.34; 1.19) and early weight increase (ΔSDS 0–18: +0.52 s.d., CI: 0.03; 1.00), and increased IGF1 serum levels at 3 months (+13.9 ng/ml, CI: 2.3; 25.5). Environmental exposure to dioxin-like chemicals was associated with being skinny at birth and with higher infant levels of circulating IGF1 as well as accelerated early childhood growth (rapid catch-up growth).