Significance Here, we show that deaf individuals activate a specific and discrete subregion of the temporal cortex, typically selective to voices in hearing people, for visual face processing. This reorganized “voice” region participates in face identity processing and responds selectively to faces early in time, suggesting that this area becomes an integral part of the face network in early deaf individuals. Observing that face processing selectively colonizes a region of the hearing brain that is functionally related to identity processing evidences the intrinsic constraints imposed to cross-modal plasticity. Our work therefore supports the view that, even if brain components modify their sensory tuning in case of deprivation, they maintain a relation to the computational structure of the problems they solve.