Previous research suggests specific deficits in face perception and memory in older adults, which could reflect a dedifferentiation in the context of a general broadening of cognitive architecture with advanced age. Such dedifferentiation could manifest in a less specialized face processing system. A promising tool to investigate the fine-tuning of face processing in older age is the own-race bias (ORB), a phenomenon reflecting more accurate memory for own-relative to other-race faces, which is related to an expertise-based specialization of early perceptual stages. To investigate whether poor face memory in older age is accompanied by reduced expertise-based specialization of face processing, we assessed event-related brain potential correlates of the ORB in high- vs. low-performing older adults (mean age = 69 years; N = 24 per group). Intriguingly, both older groups demonstrated an equivalent pattern of a behavioral ORB, and a parallel increase in N170 for other-race faces, reflecting less efficient early perceptual processing for this face category. Group differences only emerged independent of face ethnicity: whereas low-performers exhibited a right-lateralized N170, high-performers showed a more bilateral response. This finding may suggest a compensatory mechanism counteracting age-related decline in face perception enabling more efficient encoding into memory in high performers. Overall, our results demonstrate that even a less efficient face processing system in older adults can exhibit preserved expertise-related specialization toward own-race faces.