The study of patients with semantic dementia, a variant of frontotemporal lobar degeneration, has emerged over the last two decades as an important lesion model for studying human semantic memory. Although it is well-known that semantic dementia is associated with temporal lobe degeneration, controversy remains over whether the semantic deficit is due to diffuse temporal lobe damage, damage to only a sub-region of the temporal lobe or even less severe damage elsewhere in the brain. The manner in which the right and left temporal lobes contribute to semantic knowledge is also not fully elucidated. In this study we used unbiased imaging analyses to correlate resting cerebral glucose metabolism and behavioural scores in tests of verbal and non-verbal semantic memory. In addition, a region of interest analysis was performed to evaluate the role of severely hypometabolic areas. The best, indeed the only, strong predictor of semantic scores across a set of 21 patients with frontotemporal lobar degeneration with semantic impairment was degree of hypometabolism in the anterior fusiform region subjacent to the head and body of the hippocampus. As hypometabolism in the patients' rostral fusiform was even more extreme than the abnormality in other regions with putative semantic relevance, such as the temporal poles, the significant fusiform correlations cannot be attributed to floor-level function in these other regions. More detailed analysis demonstrated more selective correlations: left anterior fusiform function predicted performance on two expressive verbal tasks, whereas right anterior fusiform metabolism predicted performance on a non-verbal test of associative semantic knowledge. This pattern was further supported by an additional behavioural study performed on a wider cohort of patients with semantic dementia, in which the patients with more extensive right-temporal atrophy (when matched on degree of naming deficit to a set of cases with more extensive left temporal atrophy) were significantly more impaired on the test of non-verbal semantics. Our preferred interpretation of this laterality effect involves differential strength of connectivity between different regions of a widespread semantic network in the human brain.