Abstract Background Most tumour response scoring systems for resected pancreatic cancer after neoadjuvant therapy score tumour regression. However, whether treatment-induced changes, including tumour regression, can be identified reliably on haematoxylin and eosin-stained slides remains unclear. Moreover, no large study of the interobserver agreement of current tumour response scoring systems for pancreatic cancer exists. This study aimed to investigate whether gastrointestinal/pancreatic pathologists can reliably identify treatment effect on tumour by histology, and to determine the interobserver agreement for current tumour response scoring systems. Methods Overall, 23 gastrointestinal/pancreatic pathologists reviewed digital haematoxylin and eosin-stained slides of pancreatic cancer or treated tumour bed. The accuracy in identifying the treatment effect was investigated in 60 patients (30 treatment-naive, 30 after neoadjuvant therapy (NAT)). The interobserver agreement for the College of American Pathologists (CAP) and MD Anderson Cancer Center (MDACC) tumour response scoring systems was assessed in 50 patients using intraclass correlation coefficients (ICCs). An ICC value below 0.50 indicated poor reliability, 0.50 or more and less than 0.75 indicated moderate reliability, 0.75 or more and below 0.90 indicated good reliability, and above 0.90 indicated excellent reliability. Results The sensitivity and specificity for identifying NAT effect were 76.2 and 49.0 per cent respectively. After NAT in 50 patients, ICC values for both tumour response scoring systems were moderate: 0.66 for CAP and 0.71 for MDACC. Conclusion Identification of the effect of NAT in resected pancreatic cancer proved unreliable, and interobserver agreement for the current tumour response scoring systems was suboptimal. These findings support the recently published International Study Group of Pancreatic Pathologists recommendations to score residual tumour burden rather than tumour regression after NAT.