Immune cells are known to affect clinical outcome in colorectal cancer. Subsets of immune cells can both support and inhibit immunological interaction with tumor cells. We examined the clinical impact of T cells that are supposed to be responsible for the down regulation of a T cell response: regulatory T cells or Tregs. The study population (n = 76) consisted of a random population of colorectal cancer patients who did not receive any (neo-) adjuvant therapy, with a median follow-up time of 7.3 years (range 0.1–23.1 years). Expression of FoxP3 was used as an immunohistochemical marker to identify Tregs. We considered FoxP3+ cells present in tumor stroma and tumor epithelium separately, and related results to clinical outcome and to data on CD8+ immune cell infiltration that we previously obtained in the same patient cohort. All samples showed presence of Foxp3+ cells and in the majority of the patients (85.5%) these cells were also present in the tumor epithelial compartment. A relative high level of Foxp3+ cells in the tumor epithelium was significantly related to down regulation of HLA Class I expression (p-value 0.03). There was a trend, but no significant relation, towards a longer overall survival (p-value; 0.084) and disease-free survival (p-value; 0.073) when high levels of Foxp3+ cells were present in the tumor epithelium. More importantly, the ratio of CD8+/Foxp3+ cells did show a significant correlation with distant-recurrence-free survival. This was the case for both Foxp3+ cells specifically located in the tumor epithelium (p-value 0.024) as well as in the stroma compartment (p-value 0.018). Unfortunately due to the small sample size the ratios did not retain their statistical significance in multivariate analysis. These results provide further evidence that local interactions in the cancer microenvironment between tumor cells and immune cells are not only determined by tumor cell-related factors like HLA expression, but also by interactions among immune cells.