AbstractRelatives of individuals with autism spectrum disorder (ASD) may display milder social traits of the broader autism phenotype (BAP) providing potential endophenotypic markers of genetic risk for ASD. We performed a case–control comparison to quantify social cognition and pragmatic language difficulties in the BAP (n = 25 cases; n = 33 controls) using the Faux Pas test (FPT) and the Goldman‐Eisler Cartoon task. Using deep phenotyping we then examined patterns of inheritance of social cognition in two large multiplex families and the spectrum of performance in 32 additional families (159 members; n = 51 ASD, n = 87 BAP, n = 21 unaffected). BAP individuals showed significantly poorer FPT performance and reduced verbal fluency with the absence of a compression effect in social discourse compared to controls. In multiplex families, we observed reduced FPT performance in 89% of autistic family members, 63% of BAP relatives and 50% of unaffected relatives. Across all affected families, there was a graded spectrum of difficulties, with ASD individuals showing the most severe FPT difficulties, followed by the BAP and unaffected relatives compared to community controls. We conclude that relatives of probands show an inherited pattern of graded difficulties in social cognition with atypical faux pas detection in social discourse providing a novel candidate endophenotype for ASD.