Studies suggest that tau deposition starts in the anterolateral entorhinal cortex (EC) with normal aging, and that the presence of β-amyloid (Aβ) facilitates its spread to neocortex, which may reflect the beginning of Alzheimer's disease (AD). Functional connectivity between the anterolateral EC and the anterior-temporal (AT) memory network appears to drive higher tau deposition in AT than in the posterior-medial (PM) memory network. Here, we investigated whether this differential vulnerability to tau deposition may predict different cognitive consequences of EC, AT, and PM tau. Using¹⁸F-flortaucipir (FTP) and¹¹C-Pittsburgh compound-B (PiB) positron emission tomography (PET) imaging, we measured tau and Aβ in 124 cognitively normal human older adults (74 females, 50 males) followed for an average of 2.8 years for prospective cognition. We found that higher FTP in all three regions was individually related to faster memory decline, and that the effects of AT and PM FTP, but not EC, were driven by Aβ+ individuals. Moreover, when we included all three FTP measures competitively in the same model, only AT FTP significantly predicted memory decline. Our data support a model whereby tau, facilitated by Aβ, transits from EC to cortical regions that are most closely associated with the anterolateral EC, which specifically affects memory in the initial stage of AD. Memory also appears to be affected by EC tau in the absence of Aβ, which may be less clinically consequential. These findings may provide clarification of differences between normal aging and AD, and elucidate the transition between the two stages.SIGNIFICANCE STATEMENTTau and β-amyloid (Aβ) are hallmarks of Alzheimer's disease (AD) but are also found in cognitively normal people. It is unclear whether, and how, this early deposition of tau and Aβ may affect cognition in normal aging and the asymptomatic stage of AD. We show that tau deposition in the entorhinal cortex (EC), which is common in advanced age, predicts memory decline in older adults independent of Aβ, likely reflecting normal, age-related memory loss. In contrast, tau in anterior-temporal (AT) regions is most predictive of memory decline in Aβ+ individuals. These data support the idea that tau preferentially spreads to specific cortical regions, likely through functional connections, which plays a primary role in memory decline in the early stage of AD.