Abstract Dynamic whole-brain changes occur following stroke, and not just in association with recovery. We tested the hypothesis that the presence of a specific behavioural deficit after stroke would be associated with structural decline (atrophy) in the brain regions supporting the affected function, by examining language deficits post-stroke. We quantified whole-brain structural volume changes longitudinally (3–12 months) in stroke participants with (N = 32) and without aphasia (N = 59) as assessed by the Token Test at 3 months post-stroke, compared with a healthy control group (N = 29). While no significant difference in language decline rates (change in Token Test scores from 3 to 12 months) was observed between groups and some participants in the aphasic group improved their scores, stroke participants with aphasia symptoms at 3 months showed significant atrophy (>2%, P = 0.0001) of the left inferior frontal gyrus not observed in either healthy control or non-aphasic groups over the 3–12 months period. We found significant group differences in the inferior frontal gyrus volume, accounting for age, sex, stroke severity at baseline, education and total intracranial volume (Bonferroni-corrected P = 0.0003). In a subset of participants (aphasic N = 14, non-aphasic N = 36, and healthy control N = 25) with available diffusion-weighted imaging data, we found significant atrophy in the corpus callosum and the left superior longitudinal fasciculus in the aphasic compared with the healthy control group. Language deficits at 3 months post-stroke are associated with accelerated structural decline specific to the left inferior frontal gyrus, highlighting that known functional brain reorganization underlying behavioural improvement may occur in parallel with atrophy of brain regions supporting the language function.