The macrophage-associated molecule CD163 has been reported as a prognostic biomarker in different cancer types, but its role in colorectal cancer (CRC) is unclear. We studied CD163 in the tumor microenvironment and circulation of patients with CRC in relation to clinicopathological parameters. An enzyme-linked immunosorbent assay (ELISA) was used to measure the serum sCD163 levels and multiparameter flow cytometry was used to study the peripheral blood monocytes and their CD163 expression in CRC patients (N = 78) and healthy donors (N = 50). The distribution of tumor-associated macrophages (TAMs) was studied in primary colorectal tumors with multiplex immunofluorescence. We showed that CRC patients with above-median sCD163 level had a shorter overall survival (OS, p = 0.035) as well as disease-free survival (DFS, p = 0.005). The above-median sCD163 remained significantly associated with a shorter DFS in the multivariate analysis (p = 0.049). Moreover, a shorter OS was observed in CRC patients with an above-median total monocyte percentage (p = 0.007). The number and phenotype of the stromal and intraepithelial TAMs in colorectal tumors were not associated with clinical outcome. In conclusion, sCD163 and monocytes in the circulation may be potential prognostic biomarkers in CRC patients, whereas TAMs in the tumor showed no association with clinical outcome. Thus, our results emphasize the importance of the innate systemic immune response in CRC disease progression.