Studies using event related potentials have shown that men are more likely than women to rely on semantic cues when understanding emotional speech. In a previous functional Magnetic Resonance Imaging (fMRI) study, using an affective sentence classification task, we were able to separate areas involved in semantic processing and areas involved in the processing of affective prosody (Beaucousin et al., 2007). Here we searched for sex-related differences in the neural networks active during emotional speech processing in groups of men and women. The ortholinguistic abilities of the participants did not differ when evaluated with a large battery of tests. Although the neural networks engaged by men and women during emotional sentence classification were largely overlapping, sex-dependent modulations were detected during emotional sentence classification, but not during grammatical sentence classification. Greater activity was observed in men, compared with women, in inferior frontal cortical areas involved in emotional labeling and in attentional areas. In conclusion, at equivalent linguistic abilities and performances, men activate semantic and attentional cortical areas to a larger extent than women during emotional speech processing.