Abstract Background Given that tau accumulation, not amyloid-β (Aβ) burden, is more closely connected with cognitive impairment in Alzheimer’s disease (AD), a detailed understanding of the tau-related characteristics of cognitive function is critical in both clinical and research settings. We investigated the association between phosphorylated tau (p-Tau) level and cognitive impairment across the AD continuum and the mediating role of medial temporal lobe (MTL) atrophy. We also developed a prediction model for abnormal tau accumulation. Methods We included participants from the Gwangju Alzheimer’s Disease and Related Dementia Cohort in Korea, who completed cerebrospinal fluid analysis and clinical evaluation, and corresponded to one of three groups according to the biomarkers of A and T profiles based on the National Institute on Aging and Alzheimer’s Association research framework. Multiple linear and logistic regression analyses were performed to examine the association between p-Tau and cognition and to develop prediction models. Receiver operating characteristic curve analysis was performed to examine the discrimination ability of the models. Results Among 185 participants, 93 were classified as A-T-, 23 as A+T-, and 69 as A+T+. There was an association between decreased visuospatial delayed memory performance and p-Tau level (B = − 0.754, β = − 0.363, p < 0.001), independent of other relevant variables (e.g., Aβ). MTL neurodegeneration was found to mediate the association between the two. Prediction models with visuospatial delayed memory alone (area under the curve [AUC] = 0.872) and visuospatial delayed memory and entorhinal thickness (AUC = 0.921) for abnormal tau accumulation were suggested and they were validated in an independent sample (AUC = 0.879 and 0.891, respectively). Conclusion It is crucial to identify sensitive cognitive measures that capture subtle cognitive impairment associated with underlying pathological changes. Preliminary findings from the current study might suggest that abnormal tau accumulation underlies episodic memory impairment, particularly visuospatial modality, in the AD continuum. Suggested models are potentially useful in predicting tau pathology, and might be utilized practically in the field.