Studies related to air pollution exposure and neurocognitive disorders, specifically cognitive impairment, among older adults are limited. We investigated the association between short-term and long-term exposure to ambient air pollution (i.e., particulate matter with an aerodynamic diameter of <10 μm and ozone) and the effects of their interaction on cognitive function in a community-dwelling, free-living elderly population. Study participants were in a multiple-wave representative sample, namely the Taiwan Longitudinal Study on Aging (n = 2241). In four surveys between 1996 and 2007, their cognitive function was assessed using the Short Portable Mental Status Questionnaire (SPMSQ). We estimated air pollution from 1993 to 2007, including daily concentrations of PM10 and O3 from air quality monitoring stations, based on the administrative zone of each participant’s residence. Generalized linear mixed models were used to examine these associations after adjusting for covariates. We found that long-term exposure to PM10 and O3 was significantly associated with cognitive impairment (OR = 1.094, 95% CI: 1.020, 1.174 for PM10; OR = 1.878, 95% CI: 1.363, 2.560 for O3). The joint effect of exposure to PM10 and O3 was associated with cognitive impairment (p < 0.001). Co-exposure to ambient PM10 and O3 may deteriorate cognitive function in older adults.