Objective— Atherosclerosis begins early in life and obesity is a key determinant. We investigated the role of body mass index (BMI) and height from infancy to adulthood in presenting with high adulthood carotid intima-media thickness. Approach and Results— Odds ratios of BMI, and height Z scores at 2, 4, 6, 7, 11, 15, and 20 years, and changes between 2 and 4, 4 and 7, 7 and 15, and 15 and 20 years, for carotid intima-media thickness at 60 to 64 years in the upper quartile were estimated for 604 men and 669 women. Confounding by early-life environments, mediating by body size and cardiometabolic measures at 60 to 64 years, and effect modification were investigated. In men, there was positive association of BMI at 4 years (odds ratio, 1.256; 95% confidence interval, 1.026–1.538) and 20 years (1.282; 1.022–1.609), negative association of height at 4 years (0.780; 0.631–0.964), and negative association of height growth between 2 and 4 years (0.698; 0.534–0.913) with high carotid intima-media thickness. The childhood estimates were robust, but the estimate for BMI at 20 years was attenuated by adjustment for BMI at 60 to 64 years. The protective influence of greater early childhood height was strongest in those with the lowest systolic blood pressure at 60 to 64 years. In women, there was no pattern of association and all confidence intervals crossed 1. Conclusions— Early childhood in men might be a sensitive developmental period for atherosclerosis, in which changes in BMI and height represent 2 distinct biological mechanisms. The maintenance of healthy weight in men from adolescence onward may be a useful strategy to avoid the atherosclerotic complications of adiposity tracking.