Significance The human voice is a major source for auditorily communicated social signals. The voice in general, and emotional cues embedded in vocalizations in particular, receive enhanced decoding in sensory cortical areas of the auditory system. This enhanced cortical decoding is assumed to be remotely driven by the amygdala, which responds to socially and emotionally meaningful stimuli. Here, we tested for the first time, to our knowledge, how damage to either the left or right amygdala impairs the cortical processing of human voices and vocal affect. Amygdala damage generally leads to reduced cortical processing of human voices in the hemisphere corresponding to the side of the amygdala damage, whereas only left amygdala damage impaired the cortical processing of vocal affect.