Introduction: Checkpoint inhibitors (CPI) have significantly improved survival among patients with various cancer types. Prior studies have shown a correlation between immune cell infiltration and poorly differentiated cancers. This study evaluated the impact of poorly differentiated histology on survival in patients with advanced gastrointestinal cancers treated with immunotherapy. Methods: This study was a retrospective, single-center analysis of patients with gastrointestinal cancers treated with CPIs between 2016 and 2021. Univariate and multivariable analyses were performed to identify factors associated with patient outcome. Kaplan-Meier analysis and Cox proportional hazards models were used to assess the association between tumor and patient characteristics, progression-free survival, and overall survival. Results: A total of 123 patients were eligible and included in the analysis. Median age was 66 years (23–88 years). Majority had stage IV disease (89%), were white (65.5%), and were male (64.5%). Most common diagnoses were hepatocellular carcinoma (30.5%), gastric adenocarcinoma (16.5%), esophageal adenocarcinoma (17%), and colorectal cancer (19.8%). About 32% of the tumors were microsatellite instability-high (MSI-High/dMMR), with BRAF V600E mutation rate of 10%. About 25% of the patients received CPIs as initial treatment, while 35.5% had received two or more prior lines of therapy. Compared with well and moderately differentiated histology, patients with poorly differentiated tumors had a shorter median overall survival (mOS) (not reached [NR] vs. NR vs. 9.3 months, p = 0.0264). There was no statistically significant difference in median progression-free survival (mPFS) between histology types (2.5 vs. 4.2 vs. 2 months, p = 0.1314). On univariate survival analysis, moderately differentiated tumors correlated with a significantly longer mOS (HR: 0.48, CI: 0.24–0.93, p = 0.030) and mPFS (HR: 0.62, 95% CI: 0.38–1.00, p = 0.048) compared to poorly differentiated histology. Female patients (HR: 0.55, 95% CI: 0.34–0.90, p = 0.018) and the Eastern Cooperative Oncology Group (ECOG) of 1 (vs. ≥2) had significantly longer mPFS (HR: 0.58, 95% CI: 0.35–0.97, p = 0.036). ECOG of 1 also correlated with longer mOS (HR: 0.47, 95% CI: 0.23–0.94, p = 0.034). Microsatellite stable (MSS) tumors had significantly shorter mPFS (HR: 5.74, 95% CI: 2.41–13.63, p < 0.001) and mOS (HR: 5.45, 95% CI: 1.64–18.12, p = 0.006). The number of prior systemic therapies was also associated with shorter mPFS (HR: 1.19, 95% CI: 1.03–1.39, p = 0.022) and mOS (HR: 1.23, 95% CI: 1.01–1.50, p = 0.045). On multivariable analyses, ECOG status of 0/1 versus ≥2 and MSI-High/dMMR versus MSS remained significantly associated with longer mPFS and mOS. There was no correlation with histologic differentiation status, race, or mutations such as BRAF V600E or KRAS. Conclusion: Results from this study demonstrate that poorly differentiated histology was associated with shorter mOS but was not associated with improved PFS in patients treated with CPI. Treatment-naïve patients, moderately differentiated tumors, female gender, ECOG 1, and MSI-High/dMMR were most likely to benefit from CPI.