Neuroimaging studies have consistently shown functional brain abnormalities in patients with Bipolar Disorder (BD) and Major Depressive Disorder (MDD). However, the extent to which these two disorders are associated with similar or distinct neural changes remains unclear. We conducted a systematic review of functional magnetic resonance imaging studies comparing BD and MDD patients to healthy participants using facial affect processing paradigms. Relevant spatial coordinates from twenty original studies were subjected to quantitative Activation Likelihood Estimation meta-analyses based on 168 BD and 189 MDD patients and 344 healthy controls. We identified common and distinct patterns of neural engagement for BD and MDD within the facial affect processing network. Both disorders were associated with increased engagement of limbic regions. Diagnosis-specific differences were observed in cortical, thalamic and striatal regions. Decreased ventrolateral prefrontal cortical engagement was associated with BD while relative hypoactivation of the sensorimotor cortices was seen in MDD. Increased responsiveness in the thalamus and basal ganglia were associated with BD. These findings were modulated by stimulus valence. These data suggest that whereas limbic overactivation is reported consistently in patients with mood disorders, future research should consider the relevance of a wider network of regions in formulating conceptual models of BD and MDD.