It has traditionally been held that the hippocampus is not part of the neural substrate of working memory (WM), and that WM is preserved in Temporal Lobe Epilepsy (TLE). Recent imaging and neuropsychological data suggest this view may need revision. The aim of this study was to investigate the neural correlates of WM in TLE using functional MRI (fMRI). We used a visuo-spatial 'n-back' paradigm to compare WM network activity in 38 unilateral hippocampal sclerosis (HS) patients (19 left) and 15 healthy controls. WM performance was impaired in both left and right HS groups compared to controls. The TLE groups showed reduced right superior parietal lobe activity during single- and multiple-item WM. No significant hippocampal activation was found during the active task in any group, but the hippocampi progressively deactivated as the task demand increased. This effect was bilateral for controls, whereas the TLE patients showed progressive unilateral deactivation only contralateral to the side of the hippocampal sclerosis and seizure focus. Progressive deactivation of the posterior medial temporal lobe was associated with better performance in all groups. Our results suggest that WM is impaired in unilateral HS and the underlying neural correlates of WM are disrupted. Our findings suggest that hippocampal activity is progressively suppressed as the WM load increases, with maintenance of good performance. Implications for understanding the role of the hippocampus in WM are discussed.