Prognosis in colorectal cancer patients is quite variable, even after adjustment for clinical parameters such as disease stage and microsatellite instability status. It is possible that the psychological distress experienced by patients, including anxiety and depression, may be correlated with poor prognosis. In the present study, we hypothesize that genetic variations within three genes biologically linked to the stress response, namely serotonin transporter (SLC6A4), brain-derived neurotrophic factor (BDNF), and arginine vasopressin receptor (AVPR1B) genes are associated with prognosis in colorectal cancer patients. We used a population-based cohort of 280 patients who were followed for up to 12.5 years after diagnosis. Our multivariate analysis showed that a tagSNP in the SLC6A4 gene (rs12150214) was a predictor of shorter overall survival (HR: 1.572, 95%CI: 1.142–2.164, p = 0.005) independent of stage, age, grade and MSI status. Additionally, a multivariate analysis using the combined genotypes of three polymorphisms in this gene demonstrated that the presence of any of the minor alleles at these polymorphic loci was an independent predictor of both shorter overall survival (HR: 1.631, 95%CI: 1.190–2.236, p = 0.002) and shorter disease specific survival (HR: 1.691, 95%CI: 1.138–2.512, p = 0.009). The 5-HTT protein coded by the SLC6A4 gene has also been implicated in inflammation. While our results remain to be replicated in other patient cohorts, we suggest that the genetic variations in the SLC6A4 gene contribute to poor survival in colorectal cancer patients.