Abstract Background Biologic studies have suggested that tea may have neuroprotective activity. However, tea’s protective effect on cognitive function is controversial in human epidemiological studies, and no data, including the middle-aged, are available. The objective of this study was to investigate the association of habit, frequency, and types of tea consumption with incident cognitive impairment in middle-aged and older adults. Methods Data from the Asymptomatic Polyvascular Abnormalities in Community study were used (aged over 40y). We gathered information on tea consumption, including habit, frequency, and types, via a standardized questionnaire and assessed cognitive function by Mini-Mental State Examination (MMSE) and/or Montreal Cognitive Assessment (MoCA). Three thousand eight hundred sixty-eight and 806 participants were selected in MMSE and MoCA subgroups. Multivariate logistic regression models were utilized to examine associations between tea consumption and cognitive impairment in middle-aged and older participants. Results In MMSE analyses, after adjustment for potential confounding factors, habitual (odds ratio (OR) 0.47, [95% confidence interval (CI) 0.33–0.68], p < 0.001) and high frequency (p trend < 0.001) of tea intake were associated with a lower risk of cognitive impairment. The risk of cognitive impairment was lower in green tea consumption (OR 0.36, [95% CI 0.22–0.61], p < 0.001) than other types (OR 0.59, [95% CI 0.38–0.91], p = 0.017). In MoCA analyses, we got similar results. Conclusions Habitual tea consumption, especially high-frequency and green tea consumption, was significantly associated with a lower prevalence of cognitive impairment in middle-aged and older individuals.