Background: The association between the apolipoprotein E (APOE) ɛ4 allele and high risk of developing Alzheimer’s disease (AD) dementia before the age of 80 has been recognized for over 30 years. However, the timing and mode of action of APOE is not understood, nor has there been a detailed analysis of the effect of APOE genotype on memory, hippocampal volume, and amyloid-β (Aβ) levels in cognitively normal adults. Objective: Examine the effect of APOE allelic genotype on the relationship between Aβ levels, hippocampal volume, and memory in cognitively normal adults. Methods: This is a cross-sectional study of 989 cognitively normal older adults enrolled in the Australian Imaging, Biomarkers and Lifestyle (AIBL) study, all of whom underwent APOE genotyping and memory assessment. A subset of this group underwent PET neuroimaging for Aβ (n = 585) and MRI for hippocampal volume (n = 303). Results: APOE ɛ4 homozygotes (ɛ4/ɛ4) showed significantly worse episodic memory and higher Aβ levels than ɛ4 heterozygotes. The relationship between increasing Aβ levels and worse episodic memory was significant for ɛ3 homozygotes (ɛ3/ɛ3), ɛ4 heterozygotes, and strongest for ɛ4 homozygotes. There were no differences in hippocampal volume between APOE groups; the relationship between smaller hippocampal volume and worse episodic memory was significant only for ɛ4 homozygotes. Conclusion: APOE acts in a co-dominant fashion on Aβ levels, episodic memory, and hippocampal volume in cognitively normal older adults. APOE ɛ4 is central to the events that lead to AD in cognitively normal older adults, likely through a quantitative role in the disruption of Aβ clearance.