High levels of β-amyloid (Aβ) in the brain and carriage of the APOE ε4 allele have each been linked to cognitive impairment in cognitively normal (CN) older adults. However, the relationship between these two biomarkers and cognitive decline is unclear. The aim of this study was to investigate the relationship between cerebral Aβ level, APOE ε4 carrier status, and cognitive decline over 18 months, in 317 cognitively healthy (CN) older adults (47.6% males, 52.4% females) aged between 60 and 89 years (Mean = 69.9, SD = 6.8). Cognition was assessed using the Cogstate Brief Battery (CBB) and the California Verbal Learning Test, Second Edition (CVLT-II). Planned comparisons indicated that CN older adults with high Aβ who were also APOE ε4 carriers demonstrated the most pronounced decline in learning and working memory. In CN older adults who were APOE ε4 non-carriers, high Aβ was unrelated to cognitive decline in learning and working memory. Carriage of APOE ε4 in CN older adults with low Aβ was associated with a significantly increased rate of decline in learning and unexpectedly, improved cognitive performance on measures of verbal episodic memory over 18 months. These results suggest that Aβ and APOE ε4 interact to increase the rate of cognitive decline in CN older adults and provide further support for the use of Aβ and APOE ε4 as biomarkers of early Alzheimer's disease.