Abstract Background Ampullary adenocarcinoma (AAC) arises from the ampulla of Vater where the pancreatic duct and bile duct join and empty into the duodenum. It can be classified into intestinal and pancreatobiliary types based on histopathology or immunohistochemistry. However, there are no biomarkers for further classification of pancreatobiliary-type AAC which has important implications for its treatment. We aimed to identify the tumor origin of pancreatobiliary-type AAC by systematically analyzing whole-slide images (WSIs), survival data, and genome sequencing data collected from multiple centers. Methods This study involved three experiments. First, we extracted quantitative and highly interpretable features from the tumor region in WSIs and constructed a histologic classifier to differentiate between pancreatic adenocarcinoma (PAC) and cholangiocarcinoma. The histologic classifier was then applied to patients with pancreatobiliary-type AAC to infer the tumor origin. Secondly, we compared the overall survival of patients with pancreatobiliary-type AAC stratified by the adjuvant chemotherapy regimens designed for PAC or cholangiocarcinoma. Finally, we compared the mutation landscape of pancreatobiliary-type AAC with those of PAC and cholangiocarcinoma. Results The histologic classifier accurately classified PAC and cholangiocarcinoma in both the internal and external validation sets (AUC > 0.99). All pancreatobiliary-type AACs (n = 45) were classified as PAC. The patients with pancreatobiliary-type AAC receiving regimens designed for PAC showed more favorable overall survival than those receiving regimens designed for cholangiocarcinoma in a multivariable Cox regression (hazard ratio = 7.24, 95% confidence interval: 1.28–40.78, P = 0.025). The results of mutation analysis showed that the mutation landscape of AAC was very similar to that of PAC but distinct from that of cholangiocarcinoma. Conclusions This multi-center study provides compelling evidence that pancreatobiliary-type AAC resembles PAC instead of cholangiocarcinoma in different aspects, which can guide the treatment selection and clinical trials planning for pancreatobiliary-type AAC.