In intestinal homeostasis, continuous renewal of the epithelium is crucial to withstand the plethora of stimuli which can damage the structural integrity of the intestines. Fibroblasts contribute to this renewal by facilitating epithelial cell differentiation as well as providing the structural framework in which epithelial cells can regenerate. Upon dysregulation of intestinal homeostasis, (pre-) malignant neoplasms develop, a process which is accompanied by (epi) genetic alterations in epithelial cells as well as phenotypic changes in fibroblast populations. In the context of invasive carcinomas, these fibroblast populations are termed cancer-associated fibroblasts (CAFs). CAFs are the most abundant cell type in the tumor microenvironment of colorectal cancer (CRC) and consist of various functionally heterogeneous subsets which can promote or restrain cancer progression. Although most previous research has focused on the biology of epithelial cells, accumulating evidence shows that certain fibroblast subsets can also importantly contribute to tumor initiation and progression, thereby possibly providing avenues for improvement of clinical care for CRC patients. In this review, we summarized the current literature on the emerging role of fibroblasts in various stages of CRC development, ranging from adenoma initiation to the metastatic spread of cancer cells. In addition, we highlighted translational and therapeutic perspectives of fibroblasts in the different stages of intestinal tumor progression.