In this study we aim to examine gene–environment interactions (GxEs) between genes involved with estrogen metabolism and environmental factors related to estrogen exposure. GxE analyses were conducted with 1970 Korean breast cancer cases and 2052 controls in the case-control study, the Seoul Breast Cancer Study (SEBCS). A total of 11,555 SNPs from the 137 candidate genes were included in the GxE analyses with eight established environmental factors. A replication test was conducted by using an independent population from the Breast Cancer Association Consortium (BCAC), with 62,485 Europeans and 9047 Asians. The GxE tests were performed by using two-step methods in GxEScan software. Two interactions were found in the SEBCS. The first interaction was shown between rs13035764 of NCOA1 and age at menarche in the GE|2df model (p-2df = 1.2 × 10−3). The age at menarche before 14 years old was associated with the high risk of breast cancer, and the risk was higher when subjects had homozygous minor allele G. The second GxE was shown between rs851998 near ESR1 and height in the GE|2df model (p-2df = 1.1 × 10−4). Height taller than 160 cm was associated with a high risk of breast cancer, and the risk increased when the minor allele was added. The findings were not replicated in the BCAC. These results would suggest specificity in Koreans for breast cancer risk.