Abstract Background although type 2 diabetes increases risk of dementia by 2-fold, whether optimizing glycemic level in late life can reduce risk of dementia remains uncertain. We examined if achieving the glycemic goal recommended by the American Diabetes Association (ADA) within a year was associated with lower risk of dementia in 6 years. Methods in this population-based observational study, we examined 2246 community-living dementia-free Chinese older adults with type 2 diabetes who attended the Elderly Health Centres in Hong Kong at baseline and followed their HbA1c level and cognitive status for 6 years. In line with the ADA recommendation, we defined the glycemic goal as HbA1c < 7.5%. The study outcome was incident dementia in 6 years, diagnosed according to the 10th revision of the International Statistical Classification of Diseases and Related Health Problems (ICD-10) or Clinical Dementia Rating of 1–3. Results those with HbA1c ≥ 7.5% at baseline and HbA1c < 7.5% in 1 year were associated with higher rather than lower incidence of dementia, independent of severe hypoglycemia, glycemic variability and other health factors. Sensitivity analyses showed that a relative reduction of ≥10%, but not 5–10%, in HbA1c within a year was associated with higher incidence of dementia in those with high (≥8%) and moderate (6.5–7.9%) HbA1c at baseline. Conclusion a large reduction in HbA1c could be a potential predictor and possibly a risk factor for dementia in older adults with type 2 diabetes. Our findings suggest that optimizing or intensifying glycemic control in this population requires caution.