Klinefelter syndrome (47, XXY) is associated with several physical, cognitive, and behavioral consequences. In terms of social development, there is an increased risk of autism symptomatology. However, it remains unclear how social deficits are related to abnormal brain development and to what degree underlying mechanisms of social dysfunction in 47, XXY are similar to, or different from, those in idiopathic autism (ASD). This study was aimed at investigating the neural architecture of brain structures related to social information processing in boys with 47, XXY, also in comparison with boys with idiopathic ASD. MRI scans of 16 boys with 47, XXY, 16 with ASD, and 16 nonclinical, male controls were analyzed using voxel-based morphometry (VBM). A region of interest mask containing the superior temporal cortex, amygdala, orbitofrontal cortex (OFC), insular cortex, and medial frontal cortex was used. The Social Responsiveness Scale (SRS) was used to assess degree of autism spectrum symptoms. The 47, XXY group could not be distinguished from the ASD group on mean SRS scores, and their scores were significantly higher than in controls. VBM showed that boys with 47, XXY have significant gray matter volume reductions in the left and right insula, and the left OFC, compared with controls and boys with ASD. Additionally, boys with 47, XXY had significantly less gray matter in the right superior temporal gyrus than controls. These results imply social challenges associated with 47, XXY may be rooted in neural anatomy, and autism symptoms in boys with 47, XXY and boys with ASD might have, at least partially, different underlying etiologies.