Abstract Esophageal cancer is a major cause of cancer mortality world-wide. Even with advanced treatment options available, patients with esophageal cancer have a poor 5-year survival rate of 16%. Genomic studies of esophageal cancer have revealed frequently mutated genes involved in different oncogenic pathways. In this study, we attempted to predict genomic alterations, pathological and clinical features directly from H&E-stained whole slide images (WSIs) of individual esophageal cancer cases by utilizing modern deep learning methods. We used WSIs with esophageal cancer obtained from TCGA and annotated cancerous epithelial and stromal components separately for esophageal adenocarcinoma (EAC) and squamous cell carcinoma (ESCC) (n= 25 cases for each) to create a training set. Subsequently, patches of size 256×256 pixels were extracted, and a U-Net convolutional neural network (CNN) was trained for semantic segmentation of tumor epithelium and stromal pixels. Tumor epithelium regions identified by the first U-Net CNN were fed into a Squeezenet model to learn to distinguish between EAC and ESCC in the training phase. Finally, for each subtype (EAC and ESCC), separate Densenet models were trained to predict gene mutation status. We also trained independent Densenet models to predict tumor stage, grade, and survival outcome from the tumor epithelial patches. After training, we tested these models on an independent validation set obtained from TCGA (n = 99) and UCSF (n = 25) patients, ensuring patient level separation between the training and the validation sets. We obtained an average AUC of 0.90 and 0.87 for epithelium vs. stromal and EAC vs. ESCC classification, respectively, on the validation datasets. These models enabled us to predict tumor stage, grade, and survival outcome with an accuracy of > 0.9. TP53 gene alterations were detected in over 50% of esophageal cancer cases, and our model was able to predict TP53 gene mutation status on WSIs with an AUC of 0.91. Overall, this study demonstrates how deep learning algorithms can aid in better understanding and predicting key biological and clinical features of aggressive malignancies at both population and individual patient level. This integration of artificial intelligence with molecular and histopathologic features could also help to accelerate precision cancer diagnosis and treatment in the clinical setting. Citation Format: Ruchika Verma, Wei Wu, Neeraj Kumar, Elizabeth Yu, Won-Tak Choi, Sarah Umetsu, Trever Bivona. Deep learning-based integration of esophageal cancer morphology with genomics [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2021; 2021 Apr 10-15 and May 17-21. Philadelphia (PA): AACR; Cancer Res 2021;81(13_Suppl):Abstract nr 3.