This study aimed to investigate the association between atrophy in the prefrontal cortex with executive function and verbal fluency in elderly male and female patients poststroke. Thirty elderly female patients with non-aphasic ischemic stroke aged ≥ 60 years and 30 age-matched non-aphasic male patients with ischemic stroke were recruited. Automatic magnetic resonance imaging segmentation was used to assess the volume of the whole prefrontal cortex, along with its subdivisions: anterior cingulate cortex, orbitofrontal cortex and dorsolateral prefrontal cortex. The Semantic Verbal Fluency Test was administered at 3 and 15 months poststroke. At 3 months poststroke, left dorsolateral prefrontal cortex volume was significantly correlated with Verbal Fluency Test score in female patients only (partial coefficient = 0.453, P = 0.045), after controlling for age, education, diabetes, neurological deficit, white matter lesions volume, as well as the location and volume of infarcts. At 15 months poststroke, there remained a significant association between the left dorsolateral prefrontal cortex volume and Verbal Fluency Test (partial coefficient = 0.661, P = 0.001) and between the left prefrontal cortex volume and Verbal Fluency Test (partial coefficient = 0.573, P = 0.004) in female patients after the same adjustments. These findings indicate that atrophy of the left dorsolateral prefrontal cortex contributes to the impairment of verbal fluency in elderly female patients with stroke. Sex differences may be present in the neuropsychological mechanisms of verbal fluency impairment in patients with stroke. Research Highlights (1) Automatic MRI volumetry is an accurate method to quantify regional brain atrophy in stroke patients. (2) The standardized volume of the left dorsolateral prefrontal cortex correlated positively with the performance of semantic verbal fluency test at 3 and 15 months poststroke in both elderly male and female patients. (3) In elderly patients with ischemic stroke, performance on executive function and semantic verbal fluency tests did not correlate with location of infarcts, regional brain atrophy or white matter lesions. (4) These results suggest that sex differences may be present in the neuropsychological mechanisms of verbal fluency impairment in elderly patients with stroke.