Abstract Alzheimer’s disease is the leading cause of dementia worldwide and is characterized by a long preclinical phase in which amyloid-β and tau accumulate in the absence of cognitive decline. In vivo biomarkers for Alzheimer’s disease are expensive, invasive and inaccessible, yet are critical for accurate disease diagnosis and patient management. Recent ultrasensitive methods to measure plasma phosphorylated tau 181 (p-tau181) display strong correlations with tau positron emission tomography, p-tau181 in CSF, and tau pathology at autopsy. The clinical utility of plasma-based p-tau181 biomarkers is unclear. In a longitudinal multicentre observational study, we assessed 1113 non-demented individuals (509 cognitively unimpaired elderly and 604 individuals with mild cognitive impairment) from the Alzheimer’s Disease Neuroimaging Initiative who underwent neuropsychological assessments and were evaluated for plasma p-tau181. The primary outcome was a memory composite z-score. Mixed-effect models assessed rates of memory decline in relation to baseline plasma p-tau181, and whether plasma p-tau181 significantly predicted memory decline beyond widely available clinical and genetic data (age, sex, years of education, cardiovascular and metabolic conditions, and APOEε4 status). Participants were followed for a median of 4.1 years. Baseline plasma p-tau181 was associated with lower baseline memory (β estimate: −0.49, standard error: 0.06, t-value: −7.97), as well as faster rates of memory decline (β estimate: −0.11, standard error: 0.01, t-value: −7.37). Moreover, the inclusion of plasma p-tau181 resulted in improved prediction of memory decline beyond clinical and genetic data (marginal R2 of 16.7–23%, χ2 = 100.81, P < 0.00001). Elevated baseline plasma p-tau181 was associated with higher rates of clinical progression to mild cognitive impairment (hazard ratio = 1.82, 95% confidence interval: 1.2–2.8) and from mild cognitive impairment to dementia (hazard ratio = 2.06, 95% confidence interval: 1.55–2.74). Our results suggest that in elderly individuals without dementia at baseline, plasma p-tau181 biomarkers were associated with greater memory decline and rates of clinical progression to dementia. Plasma p-tau181 improved prediction of memory decline above a model with currently available clinical and genetic data. While the clinical importance of this improvement in the prediction of memory decline is unknown, these results highlight the potential of plasma p-tau181 as a cost-effective and scalable Alzheimer’s disease biomarker.