BackgroundThere has been substantial progress in developing approaches to measure mistreatment of women during childbirth. However, less is known about the differences in measurement approaches. In this study, we compare measures of mistreatment obtained from the same women using labour observations and community-based surveys in Ghana, Guinea and Nigeria.MethodsExperiences of mistreatment during childbirth are person-centred quality measures. As such, we assessed individual-level and population-level accuracy of labour observation relative to women’s self-report for different types of mistreatment. We calculated sensitivity, specificity, percent agreement and population-level inflation factor (IF), assessing prevalence of mistreatment in labour observation divided by ‘true’ prevalence in women’s self-report. We report the IF degree of bias as: low (0.75<IF<1.5), moderate (0.50<IF<0.75 or 1.5<IF<2.0) or high (IF≤0.50 or IF≥2.0).Results1536 women across Ghana (n=779), Guinea (n=425) and Nigeria (n=332) were included. Most mistreatment items demonstrated better specificity than sensitivity: observation of any physical abuse (44% sensitive, 89% specific), any verbal abuse (61% sensitive, 73% specific) and presence of a labour companion (19% sensitive, 93% specific). Items for stigma (IF 0.16), pain relief requested (IF 0.38), companion present (IF 0.32) and lack of easy access to fluids (IF 0.46) showed high risk of bias, meaning labour observations would substantially underestimate true prevalence. Other items showed low or moderate bias.ConclusionUsing self-report as the reference standard, labour observations demonstrated moderate-to-high specificity (accurately identifying lack of mistreatment) but low-to-moderate sensitivity (accurately identifying presence of mistreatment) among women. For overall prevalence, either women’s self-report or observations can be used with low-moderate bias for most mistreatment items. However, given the dynamicity, complexity, and limitations in ‘objectivity’, some experiences of mistreatment (stigma, pain relief, labour companionship, easy access to fluids) require measurement via women’s self-report. More work is needed to understand how subjectivity influences how well a measure represents individual’s experiences.