Abstract Background Undifferentiated carcinoma (UC) of the pancreas is a rare subtype of pancreatic cancer. Although UC has been considered a highly aggressive malignancy, no clinical studies have addressed the efficacy of chemotherapy for unresectable UC. Therefore, we conducted multicenter retrospective study to investigate the efficacy of chemotherapy in patients with UC of the pancreas. Methods This multicenter retrospective cohort study was conducted at 17 institutions in Japan between January 2007 and December 2017. A total of 50 patients treated with chemotherapy were analyzed. Results The median overall survival (OS) in UC patients treated with chemotherapy was 4.08 months. The details of first-line chemotherapy were as follows: gemcitabine (n = 24), S-1 (n = 12), gemcitabine plus nab-paclitaxel (n = 6), and other treatment (n = 8). The median progression-free survival (PFS) was 1.61 months in the gemcitabine group, 2.96 months in the S-1 group, and 4.60 months in the gemcitabine plus nab-paclitaxel group. Gemcitabine plus nab-paclitaxel significantly improved PFS compared with gemcitabine (p = 0.014). The objective response rate (ORR) was 4.2% in the gemcitabine group, 0.0% in the S-1 group, and 33.3% in the gemcitabine plus nab-paclitaxel group. Gemcitabine plus nab-paclitaxel also showed a significantly higher ORR compared with both gemcitabine and S-1 (gemcitabine plus nab-paclitaxel vs. gemcitabine: p = 0.033; gemcitabine plus nab-paclitaxel vs. S-1: p = 0.034). A paclitaxel-containing first-line regimen significantly improved OS compared with a non-paclitaxel-containing regimen (6.94 months vs. 3.75 months, respectively; p = 0.041). After adjustment, use of a paclitaxel-containing regimen in any line was still an independent predictor of OS (hazard ratio for OS, 0.221; 95% confidence interval, 0.076–0.647; p = 0.006) in multiple imputation by chained equation. Conclusions The results of the present study indicate that a paclitaxel-containing regimen would offer relatively longer survival, and it is considered a reasonable option for treating patients with unresectable UC.