Brain regions associated with the processing of emotional stimuli are often also associated with the processing of social stimuli. Therefore, this network consisting of the amygdala, the anterior insula, the superior temporal sulcus (STS), and the orbitofrontal cortex (OFC) may rather be involved in more general relevance detection which should be independent of the sensory modality of the stimuli. In the current study, we used functional MRI to measure brain activations while participants either viewed pictures that varied in their emotional and social content or listened to sounds that varied along the same dimensions. The amygdala, the anterior insula, the STS, and the OFC showed increased activation during processing of emotional as well as social stimuli independent of the sensory modality in which the stimuli were presented. Moreover, social emotional stimuli elicited more pronounced activity in this network than stimuli with solely emotional or social content. These results indicate that the proposed network involved in relevance detection works independently of the source of relevance (emotional or social information mediated by the stimulus) and modality.