Variants in the APOE gene region may explain ethnic differences in the association of Alzheimer’s disease (AD) with ε4. Ethnic differences in allele frequencies for three APOE region SNPs (single nucleotide polymorphisms) were identified and tested for association in 19,398 East Asians (EastA), including Koreans and Japanese, 15,836 European ancestry (EuroA) individuals, and 4985 African Americans, and with brain imaging measures of cortical atrophy in sub-samples of Koreans and EuroAs. Among ε4/ε4 individuals, AD risk increased substantially in a dose-dependent manner with the number of APOE promoter SNP rs405509 T alleles in EastAs (TT: OR (odds ratio) = 27.02, p = 8.80 × 10−94; GT: OR = 15.87, p = 2.62 × 10−9) and EuroAs (TT: OR = 18.13, p = 2.69 × 10−108; GT: OR = 12.63, p = 3.44 × 10−64), and rs405509-T homozygotes had a younger onset and more severe cortical atrophy than those with G-allele. Functional experiments using APOE promoter fragments demonstrated that TT lowered APOE expression in human brain and serum. The modifying effect of rs405509 genotype explained much of the ethnic variability in the AD/ε4 association, and increasing APOE expression might lower AD risk among ε4 homozygotes.