The episodic buffer accommodates formation and maintenance of unitary multidimensional representations based on information in different codes from different sources. Formation, based on submorphemic units, engages posterior brain regions, while maintenance engages frontal regions. Using a hybrid fMRI design, that allows separate analysis of transient and sustained components, an n-back task and an experimental group of 13 hearing native signers, with experience of Swedish Sign Language and Swedish since birth, we investigated binding of lexical signs and words in working memory. Results show that the transient component of these functions is supported by a buffer-specific network of posterior regions including the right middle temporal lobe, possibly relating to binding of phonological loop representations with semantic representations in long-term memory, as well as a loop-specific network, in line with predictions of a functional relationship between loop and buffer. The left hippocampus was engaged in transient and sustained components of buffer processing, possibly reflecting the meaningful nature of the stimuli. Only a minor role was found for executive functions in line with other recent work. A novel representation of the sustained component of working memory for audiovisual language in the right inferior temporal lobe may be related to perception of speech-related facial gestures. Previous findings of sign and speech loop representation in working memory were replicated and extended. Together, these findings support the notion of a module that mediates between codes and sources, such as the episodic buffer, and further our understanding of its nature.