Significance How do we infer absence of emotion? As opposed to the brain mechanisms for recognizing emotions, this issue has remained underinvestigated. Present findings bond communication between the amygdala and insula with the ability to infer the absence of emotion in body language. Previously, both the amygdala and insula have been associated with emotion perception. The outcome of this study extends the conceptualization of these brain regions and their interplay to the processing of social signals without emotional content. This may have wide-ranging implications for better understanding neuropsychiatric conditions characterized by misinterpretation of neutral for emotional signals, such as depression or schizophrenia. The findings call for interdisciplinary research on how the brain infers the lack of emotional valence in social signals.