Toll-like receptors (TLRs) are mediators of inflammation in the gut and potentially important modulators of colon and rectal cancer risk. We use data from a population-based study of incident colon cancer cases (n = 1,555) and controls (n = 1,956) and rectal cancer cases (n = 754) and controls (n = 959). We evaluate genetic variation in TLR2 (six SNPs), TLR3 (four SNPs) and TLR4 (eight SNPs) with risk of developing colon or rectal cancer and survival after diagnosis. TLR3 rs11721827 was associated with rectal cancer (odds ratio [OR] 1.27, 95% confidence interval [CI] 1.02, 1.58 for AC/CC vs. AA genotype, Wald p = 0.035; adjusted p = 0.126); TLR3 rs3775292 and TLR4 rs11536898 were associated with colon cancer (OR 0.68, 95% CI 0.49, 0.95 for GG vs. CC/CG and OR 0.50. 95% CI 0.29, 0.87 for AA vs. CA/CC, respectively; Wald p = 0.023 and 0.015; adjusted p = 0.085 and 0.101, respectively). TLR2 rs7656411 and rs3804099, respectively, interacted with nonsteroidal anti-inflammatory drug (NSAID) use and cigarette smoking to alter risk of colon cancer (adjusted p = 0.034 and 0.077); TLR3 rs11721827 interacted with NSAID use to alter risk of colon cancer (adjusted p = 0.071). TLR3 rs3775292 interacted with dietary carbohydrates to alter colon cancer risk and with dietary carbohydrates and saturated fat to alter rectal cancer risk (adjusted p = 0.064, 0.0035 and 0.025, respectively). Multiple SNPs in TLR2 and TLR4 were associated with colon cancer survival. Although few independent associations with TLR genes were observed, we observed significant interaction of TLR2 and TLR3 with hypothesized lifestyle factors. Interaction with dietary factors remained significant for rectal cancer after adjustment for multiple comparisons.