Abstract Background To assess the effects of apolipoprotein E (ApoE) ε4 genotype on amyloid-β (Aβ) and tau burden and their longitudinal changes in Alzheimer’s disease (AD) spectrum. Methods Among 272 individuals who underwent PET scans (¹⁸F-florbetaben for Aβ and ¹⁸F-flortaucipir for tau) and ApoE genotyping, 187 individuals completed 2-year follow-up PET scans. After correcting for the partial volume effect, we compared the standardized uptake value ratio (SUVR) for Aβ and tau burden between the ε4+ and ε4− groups. By using a linear mixed-effect model, we measured changes in SUVR in the ApoE ε4+ and ε4− groups. Results The ε4+ group showed greater baseline Aβ burden in the diffuse cortical regions and greater tau burden in the lateral, and medial temporal, cingulate, and insula cortices. Tau accumulation rate was higher in the parietal, occipital, lateral, and medial temporal cortices in the ε4+ group. In Aβ+ individuals, baseline tau burden was greater in the medial temporal cortex, while Aβ burden was conversely greater in the ε4− group. Tau accumulation rate was higher in the ε4+ group in a small region in the lateral temporal cortex. The effect of ApoE ε4 on enhanced tau accumulation persisted even after adjusting for the global cortical Aβ burden. Conclusions Progressive tau accumulation may be more prominent in ε4 carriers, particularly in the medial and lateral temporal cortices. ApoE ε4 allele has differential effects on the Aβ burden depending on the existing amyloidosis and may enhance vulnerability to progressive tau accumulation in the AD spectrum independent of Aβ.