Long-lived animals across a range of taxa display substantial social complexity that often includes hierarchical modularity of their social structures. A complete understanding of how their social systems function is achieved by understanding not only how individuals interact with each other, but also how their social groups relate to one another. Here, we examine social relationships across two levels of the hierarchical social structure of sperm whales, Physeter macrocephalus. Using an unparalleled data set of nine social units collected across a 6-year study (2005–2010), we calculate social differentiation (0 when relationships are completely homogeneous, and greater than 1 when there is considerable diversity among the relationships) to focus on the diversity of social relationships between the fundamental level of social structure, the unit. We contrast these patterns by comparing patterns between individuals within these units. Social relationships within units are diverse, with a mean social differentiation (S) ± SE of 0.80 ± 0.05 among adult females and 0.91 ± 0.05 when calves are included. Social differentiation was also high between units (1.11 ± 0.06). In addition, we identified long-term patterns of association between units that appear consistent over time, in two cases across more than a decade. Among the nine units, there were three strongly bonded pairs. Social preferences create complexity and diversity in the types of relationships formed at multiple levels of sperm whale social structure and across various timescales. Individuals show preferences for each other across hours, days and years; units form strong long-term bonds across decades; and vocal dialects mark social segregations between sperm whale cultures across generations.