Abstract Prior studies have reported inconsistency in the lesion sites associated with verbal short-term memory impairments. Here we asked: How many different lesion sites can account for selective impairments in verbal short-term memory that persist over time, and how consistently do these lesion sites impair verbal short-term memory? We assessed verbal short-term memory impairments using a forward digit span task from the Comprehensive Aphasia Test. First, we identified the incidence of digit span impairments in a sample of 816 stroke survivors (541 males/275 females; age at stroke onset 56 ± 13 years; time post-stroke 4.4 ± 5.2 years). Second, we studied the lesion sites in a subgroup of these patients (n = 39) with left hemisphere damage and selective digit span impairment—defined as impaired digit span with unimpaired spoken picture naming and spoken word comprehension (tests of speech production and speech perception, respectively). Third, we examined how often these lesion sites were observed in patients who either had no digit span impairments or digit span impairments that co-occurred with difficulties in speech perception and/or production tasks. Digit span impairments were observed in 222/816 patients. Almost all (199/222 = 90%) had left hemisphere damage to five small regions in basal ganglia and/or temporo-parietal areas. Even complete damage to one or more of these five regions was not consistently associated with persistent digit span impairment. However, when the same regions were spared, only 5% (23/455) presented with digit span impairments. These data suggest that verbal short-term memory impairments are most consistently associated with damage to left temporo-parietal and basal ganglia structures. Sparing of these regions very rarely results in persistently poor verbal short-term memory. These findings have clinical implications for predicting recovery of verbal short-term memory after stroke.