Abstract The assessment of pilot performance relies on systematic observation and assessment by a single trained rater or instructor/evaluator (I/E). Due to the importance of aviat ion safety, it is imperative that the rating and evaluation processes used by these I/Es yield reliable and valid data. This paper describes a case study focused on improving the reliability and validity of crew assessment at a commercial,carrier. The process for improving reliability involved the evaluation of current training methods, the construction and evaluation of five metrics for assessing inter- rater reliability, and a standardized process for using these metrics to train I/Es. A separate set of interventions was developed and implemented,to improve structural validity. Data collected from two fleets in this airline over a three- year period suggest aspects of reliability and validity that should be the focal points of continuing I/E training. Improving Rater Calibration 3 Improving Rater Calibration and Performance in Aviation Robert W. Holt, Jeffrey T. Hansberger, and Deborah A. Boehm-Davis