Apolipoprotein-E4 (APOE-E4) is a major genetic risk factor for developing Alzheimer’s disease (AD). The verbal fluency task (VFT), especially the subtask category fluency, has shown to provide a good discrimination between cognitively normal controls and subjects with AD. Interestingly, APOE-E4 seems to have no effect on the behavioral performance during a VFT in healthy elderly. Thus, the purpose of the present study was to reveal possible compensation mechanisms by investigating the effect of APOE-E4 on the hemodynamic response in non-demented elderly during a VFT by using functional near-infrared spectroscopy (fNIRS). We compared performance and hemodynamic response of high risk APOE-E4/E4, -E3/E4 carriers with neutral APOE-E3/E3 non-demented subjects (N = 288; 70–77 years). No difference in performance was found. APOE-E4/E4, -E3/E4 carriers had a decreased hemodynamic response in the right inferior frontal junction (IFJ) with a corresponding higher response in the left middle frontal gyrus (MFG) during category fluency. Performance was correlated with the hemodynamic response in the MFG. We assume a compensation of decreased IFJ brain activation by utilizing the MFG during category fluency and thus resulting in no behavioral differences between APOE-groups during the performance of a VFT.