We examined the association between caffeine and coffee intake and the community composition and structure of colonic microbiota. A total of 34 polyp-free adults donated 97 colonic biopsies. Microbial DNA was sequenced for the 16S rRNA gene V4 region. The amplicon sequence variant was assigned using DADA2 and SILVA. Food consumption was ascertained using a food frequency questionnaire. We compared the relative abundance of taxonomies by low (<82.9 mg) vs. high (≥82.9 mg) caffeine intake and by never or <2 cups vs. 2 cups vs. ≥3 cups coffee intake. False discovery rate-adjusted p values (q values) <0.05 indicated statistical significance. Multivariable negative binomial regression models were used to estimate the incidence rate ratio and its 95% confidence interval of having a non-zero count of certain bacteria by intake level. Higher caffeine and coffee intake was related to higher alpha diversity (Shannon index p < 0.001), higher relative abundance of Faecalibacterium and Alistipes, and lower relative abundance of Erysipelatoclostridium (q values < 0.05). After adjustment of vitamin B2 in multivariate analysis, the significant inverse association between Erysipelatoclostridium count and caffeine intake remained statistically significant. Our preliminary study could not evaluate other prebiotics in coffee.