Abstract Background Medicare fee-for-service (FFS) claims data are increasingly leveraged for dementia research. Few studies address the validity of recent claim data to identify dementia, or carefully evaluate characteristics of those assigned the wrong diagnosis in claims. Methods We used claims data from 2014 to 2018, linked to participants administered rigorous, annual dementia evaluations in 5 cohorts at the Rush Alzheimer’s Disease Center. We compared prevalent dementia diagnosed through the 2016 cohort evaluation versus claims identification of dementia, applying the Bynum-standard algorithm. Results Of 1 054 participants with Medicare Parts A and B FFS in a 3-year window surrounding their 2016 index date, 136 had prevalent dementia diagnosed during cohort evaluations; the claims algorithm yielded 217. Sensitivity of claims diagnosis was 79%, specificity 88%, positive predictive value 50%, negative predictive value 97%, and overall accuracy 87%. White participants were disproportionately represented among detected dementia cases (true positive) versus cases missed (false negative) by claims (90% vs 75%, respectively, p = .04). Dementia appeared more severe in detected than missed cases in claims (mean Mini-Mental State Exam = 15.4 vs 22.0, respectively, p < .001; 28% with no limitations in activities of daily living versus 45%, p = .046). By contrast, those with “over-diagnosis” of dementia in claims (false positive) had several worse health indicators than true negatives (eg, self-reported memory concerns = 51% vs 29%, respectively, p < .001; mild cognitive impairment in cohort evaluation = 72% vs 44%, p < .001; mean comorbidities = 7 vs 4, p < .001). Conclusions Recent Medicare claims perform reasonably well in identifying dementia; however, there are consistent differences in cases of dementia identified through claims than in rigorous cohort evaluations.