Intrahepatic cholangiocarcinoma (iCCA) is the most frequent subtype of cholangiocarcinoma (CCA), and the incidence has globally increased in recent years. In contrast to surgically treated iCCA, data on the impact of fibrosis on survival in patients undergoing palliative chemotherapy are missing. We retrospectively analyzed the cases of 70 patients diagnosed with iCCA between 2007 and 2020 in our tertiary hospital. Histopathological assessment of fibrosis was performed by an expert hepatobiliary pathologist. Additionally, the fibrosis-4 score (FIB-4) was calculated as a non-invasive surrogate marker for liver fibrosis. For overall survival (OS) and progression-free survival (PFS), Kaplan–Meier curves and Cox-regression analyses were performed. Subgroup analyses revealed a median OS of 21 months (95% CI = 16.7–25.2 months) and 16 months (95% CI = 7.6–24.4 months) for low and high fibrosis, respectively (p = 0.152). In non-cirrhotic patients, the median OS was 21.8 months (95% CI = 17.1–26.4 months), compared with 9.5 months (95% CI = 4.6–14.3 months) in cirrhotic patients (p = 0.007). In conclusion, patients with iCCA and cirrhosis receiving palliative chemotherapy have decreased OS rates, while fibrosis has no significant impact on OS or PFS. These patients should not be prevented from state-of-the-art first-line chemotherapy.