Behavioral studies have shown that, at a population level, women perform better on tests of social cognition and empathy than men. Furthermore Autism Spectrum Disorders (ASDs), which are characterized by impairments in social functioning and empathy, occur more commonly in males than females. These findings have led to the hypothesis that differences in the functioning of the social brain between males and females contribute to the greater vulnerability of males to ASD and the suggestion that ASD may represent an extreme form of the male brain. Here we sought to investigate this hypothesis by determining: (i) whether males and females differ in social brain function, and (ii) whether any sex differences in social brain function are exaggerated in individuals with ASD. Using fMRI we show that males and females differ markedly in social brain function when making social decisions from faces (compared to simple sex judgements) especially when making decisions of an affective nature, with the greatest sex differences in social brain activation being in the inferior frontal cortex (IFC). We also demonstrate that this difference is exaggerated in individuals with ASD, who show an extreme male pattern of IFC function. These results show that males and females differ significantly in social brain function and support the view that sex differences in the social brain contribute to the greater vulnerability of males to ASDs.