Social attention is conveyed primarily by gaze, but also by head and body orientation. These cues not only signal a seen agent's direction of attention but are also used to infer their current goals and intentions. Here, we review recent research showing that different gaze, head and body orientations are represented by distinct neural mechanisms, and show that a statistical summary of recent neuroimaging studies reveals a widespread neural network for gaze processing. We discuss how this network subserves visual analysis of social attention cues, and imitative attention shifts and mental state attributions from these cues. We also review new research indicating that the posterior superior temporal sulcus region responds to the inferred intentionality of social cues, and consider the development of the gaze perception system.