Abstract Background Procedure-based assessment (PBA) is used within most UK surgical training programmes for assessing trainees' procedural skills in the operating theatre. All postgraduate assessment methods require evidence to support their implementation. The aims were to evaluate the validity, reliability and acceptability of PBA. Methods Eighty-one trainees in six surgical specialties were assessed performing common procedures; 749 PBAs were provided across 348 operations by 57 clinical supervisors and four independent assessors. Results Construct validity was demonstrated by correlation of PBA scores with measures of surgical training and experience. Reliability (G exceeding 0·8) of the adjusted total item score and global summary score for a given procedure was achieved using four and three assessor judgements respectively; assessing a mix of procedures required more cases/assessors because performance is procedure specific. The acceptability of PBA for assessment and feedback within surgical training was predominantly positive among clinical supervisors and trainees. Conclusion PBA demonstrated good overall validity and acceptability, and exceptionally high reliability. Trainees should be assessed adequately for each given procedure.