Individual differences in affective and social processes may arise from variability in amygdala-medial prefrontal (mPFC) circuitry and related genetic heterogeneity. To explore this possibility in humans, we examined the structural correlates of trait negative affect in a sample of 1050 healthy young adults with no history of psychiatric illness. Analyses revealed that heightened negative affect was associated with increased amygdala volume and reduced thickness in a left mPFC region encompassing the subgenual and rostral anterior cingulate cortex. The most extreme individuals displayed an inverse correlation between amygdala volume and mPFC thickness, suggesting that imbalance between these structures is linked to negative affect in the general population. Subgroups of participants were further evaluated on social (n = 206) and emotional (n = 533) functions. Individuals with decreased mPFC thickness exhibited the poorest social cognition and were least able to correctly identify facial emotion. Given prior links between disrupted amygdala–mPFC circuitry and the presence of major depressive disorder (MDD), we explored whether the individual differences in anatomy observed here in healthy young adults were associated with polygenic risk for MDD (n = 438) using risk scores derived from a large genome-wide association analysis (n = 18,759). Analyses revealed associations between increasing polygenic burden for MDD and reduced cortical thickness in the left mPFC. These collective findings suggest that, within the healthy population, there is significant variability in amygdala–mPFC circuitry that is associated with poor functioning across affective and social domains. Individual differences in this circuitry may arise, in part, from common genetic variability that contributes to risk for MDD.