Abstract Whole grain and fiber intakes may decrease the risk of colorectal cancer. The interplay between host genetic factors, colonization of specific bacteria, production of short-chain fatty acids (SCFA), and intake of whole grains and fiber could alter the protective role of carbohydrates against colorectal cancer. Here, we assessed intakes of types and sources of carbohydrates in 114,217 UK Biobank participants with detailed dietary data (2–5 24-hour dietary assessments), and a host polygenic score (PGS) was applied to categorize participants as high or low for intraluminal microbial SCFA production, namely, butyrate and propionate. Multivariable Cox proportional hazards models were used to determine the associations of carbohydrates and SCFA with colorectal cancer incidence. During a median follow-up of 9.4 years, 1,193 participants were diagnosed with colorectal cancer. Risk was inversely associated with intakes of non-free sugar and whole grain fiber. Evidence of heterogeneity was observed by the butyrate PGS; consuming higher amounts of whole grain starch was only associated with a lower risk of colorectal cancer in those with predicted high SCFA production. Similarly, in additional analyses utilizing the larger UK Biobank cohort (N = 343,621) with less detailed dietary assessment, only individuals with a high genetically predicted butyrate production had a lower risk of colorectal cancer per 5 g/day intake of bread and cereal fiber. This study suggests that colorectal cancer risk varies by intake of carbohydrate types and sources, and the impact of whole grain intake may be modified by SCFA production. Significance: Prospective population-level analyses provide evidence supporting the importance of butyrate production in reduction of colorectal cancer risk by whole grain consumption.