Based on the multiple logistic regression analysis of data from a random sample of 1,023 old adults collected in Taiwan in 2000, we found that interactions between carrying the APOE4 allele and one of four life stress factors (relocated mainlander, living in a crowded household with six or more persons, living in an earthquake-damaged house, and monthly financial difficulty) significantly increased the odds ratio of poor self-reported health. Correlations between carrying the APOE4 allele and the life stress factors were ruled out by statistical tests. These life stress factors had a substantially larger adverse impact on self-reported health in APOE4 allele carriers than in noncarriers. This study provides evidence that interaction between carrying APOE4 allele and chronic life stressors has significant impacts on self-reported health while controlling for various sociodemographic and health behavior factors. Further studies with richer biomarkers are warranted for deeper understanding of the biological mechanisms.