Here we examine the effects of both self-reported and independent observer-reported environmental risk indices, the serotonin transporter gene promoter (5HTTLPR) polymorphism, and their interaction on self-esteem. This trait was assessed during early and mid adolescence (mean age=14 and 16.5, respectively) and young adulthood (mean age=21.8) in a prospective cohort of 1214 unrelated participants in the Longitudinal Study of Adolescent Health (Add Health). Using structural equation modeling we identified a gene-environment (G×E) interaction using observer-report but not self-report measures of environmental stress exposure during adolescence: 5HTTLPR genotype and observer-reports of home and neighborhood quality (HNQ) during adolescence interacted to predict self-esteem levels in young adulthood (p<.004). Carriers of the s allele who lived in poor HNQ conditions during adolescence reported lower self-esteem in young adulthood than those with a good HNQ during adolescence. In contrast, among individuals with the l/l genotype, adolescent HNQ did not predict adulthood self-esteem. Genes may moderate the effect of adolescent environmental conditions on adulthood self-esteem.