Previous investigations showed that the impact of negative distraction on cognitive processing is linked to increased activation in a ventral affective system (VAS) and simultaneous deactivation in a dorsal executive system (DES). However, less in known about the influences of positive valence and different arousal levels on these effects. FMRI data were recorded while participants performed a working memory (WM) task, with positive and negative pictures presented as distracters during the delay between the memoranda and probes. First, positive distraction had reduced impact on WM performance, compared to negative distraction. Second, fMRI results identified valence-specific effects in DES regions and overlapping arousal and valence effects in VAS regions, suggesting increased impact of negative distraction and enhanced engagement of coping mechanisms for positive distraction. Third, a valence-related rostro-caudal dissociation was identified in medial frontal regions associated with the default-mode network (DMN). Finally, these DMN regions showed increased functional connectivity with DES regions for negative compared to positive distraction. Overall, these findings suggest that, while both positive and negative distraction engage partly similar arousal-dependent mechanisms, their differential impact on WM performance is linked to dissociations in the engagement of, and coupling between, regions associated with emotion processing and higher-lever cognitive control.