During the last decade, immunotherapy has radically changed perspectives on anti-tumor treatments. However, solid tumor treatment by immunotherapy has not met expectations. Indeed, poor clinical response to treatment has highlighted the need to understand and avoid immunotherapy resistance. Cholangiocarcinoma (CCA) is the second cause of hepatic cancer-related deaths because of drug inefficacy and chemo-resistance in a majority of patients. Thus, intense research is ongoing to better understand the mechanisms involved in the chemo-resistance processes. The tumor microenvironment (TME) may be involved in tumor therapy resistance by limiting drug access. Indeed, cells such as cancer-associated fibroblasts (CAFs) alter TME by producing in excess an aberrant extracellular matrix (ECM). Interestingly, CAFs are the dominant stromal component in CCA that secrete large amounts of stiff ECM. Stiff ECM could contribute to immune exclusion by limiting anti-tumor T-cells drop-in. Herein, we summarize features, functions, and interactions among CAFs, tumor-associated ECM, and immune cells in TME. Moreover, we discuss the strategies targeting CAFs and the remodeling of the ECM to improve immunotherapy and drug therapies.