The question of how affective processing is organized in the brain is still a matter of controversial discussions. Based on previous initial evidence, several suggestions have been put forward regarding the involved brain areas: (a) right-lateralized dominance in emotional processing, (b) hemispheric dominance according to positive or negative valence, (c) one network for all emotional processing and (d) region-specific discrete emotion matching. We examined these hypotheses by investigating intrinsic functional connectivity patterns that covary with results of the Positive and Negative Affective Schedule (PANAS) from 65 participants. This approach has the advantage of being able to test connectivity rather than activation, and not requiring a potentially confounding task. Voxelwise functional connectivity from 200 regions-of-interest covering the whole brain was assessed. Positive and negative affect covaried with functional connectivity involving a shared set of regions, including the medial prefrontal cortex, the anterior cingulate, the visual cortex and the cerebellum. In addition, each affective domain had unique connectivity patterns, and the lateralization index showed a right hemispheric dominance for negative affect. Therefore, our results suggest a predominantly right-hemispheric network with affect-specific elements as the underlying organization of emotional processes.