Non-adherence to medication remains a major barrier to recovery from mental illnesses. Identification of those patients likely to experience adherence difficulties is best undertaken during inpatient treatment, prior to discharge into the community. More objective assessments of adherence behaviours might assist staff to more effectively target support to those patients most in need. This study investigated the interrater reliability of an inpatient behavioural observation scale of patient engagement with medication: the Observer Rating of Medication Taking (ORMT) scale. Eight mental health nurses working in a psychiatric hospital inpatient setting were trained in the use of the ORMT using video-based vignettes. Working in pairs, staff then independently rated adherence behaviours of 13 inpatients in a rehabilitation unit (total of 160 ratings) over a 1-week period. Concurrently, two expert raters also undertook independent ratings of patient medication-taking behaviour using the ORMT. Interrater reliability was assessed across both staff and expert raters. The results indicated that the ORMT has satisfactory interrater reliability, and can be appropriately used in an inpatient setting. The observational location of raters on the ward influenced the range of medication-taking behaviours observed, and thus the extent of concordance between raters. Further research to determine if the ORMT predicts adherence in the community is warranted.