Although short-term feeding studies demonstrated effects of grains, fiber, and gluten on gut microbiome composition, the impact of habitual intake of these dietary factors is poorly understood. We examined whether habitual intakes of whole and refined grains, fiber, and gluten are associated with gut microbiota in a cross-sectional study. This study included 779 participants from the multi-ethnic Food and Microbiome Longitudinal Investigation study. Bacterial 16SV4 rRNA gene from baseline stool was amplified and sequenced using Illumina MiSeq. Read clustering and taxonomic assignment was performed using QIIME2. Usual dietary intake was assessed by a 137-item food frequency questionnaire. Association of diet with gut microbiota was assessed with respect to overall composition and specific taxon abundances. Whole grain intake was associated with overall composition, as measured by the Jensen–Shannon divergence (multivariable-adjusted Ptrend for quartiles = 0.03). The highest intake quartile was associated with higher abundance of Bacteroides plebeius, Faecalibacterium prausnitzii, Blautia producta, and Erysipelotrichaceae and lower abundance of Bacteroides uniformis. These bacteria also varied by dietary fiber intake. Higher refined grain and gluten intake was associated with lower Shannon diversity (Ptrend < 0.05). These findings suggest that whole grain and dietary fiber are associated with overall gut microbiome structure, largely fiber-fermenting microbiota. Higher refined grain and gluten intakes may be associated with lower microbial diversity. Significance: Regular consumption of whole grains and dietary fiber was associated with greater abundance of gut bacteria that may lower risk of colorectal cancer. Further research on the association of refined grains and gluten with gut microbial composition is needed to understand their roles in health and disease.