When brain activity is measured by neuroimaging, the canonical hemodynamic response (increase in oxygenated hemoglobin ([O2Hb]) and decrease in deoxygenated hemoglobin ([HHb]) is not always seen in every subject. The reason for this intersubject-variability of the responses is still not completely understood. This study is performed with 32 healthy subjects, using the systemic physiology augmented functional near-infrared spectroscopy (SPA-fNIRS) approach. We investigate the intersubject variability of hemodynamic and systemic physiological responses, due to a verbal fluency task (VFT) under colored light exposure (CLE; blue and red). Five and seven different hemodynamic response patterns were detected in the subgroup analysis of the blue and red light exposure, respectively. We also found that arterial oxygen saturation and mean arterial pressure were positively correlated with [O2Hb] at the prefrontal cortex during the CLE-VFT independent of the color of light and classification of the subjects. Our study finds that there is substantial intersubject-variability of cerebral hemodynamic responses, which is partially explained by subject-specific systemic physiological changes induced by the CLE-VFT. This means that both subgroup analyses and the additional assessment of systemic physiology are of crucial importance to achieve a comprehensive understanding of the effects of a CLE-VFT on human subjects.