Abstract Background Phosphorus is an essential nutrient in all living organisms and, currently, it is the focus of much attention due to its global scarcity, the environmental impact of phosphorus from excreta, and its low digestibility due to its storage in the form of phytates in plants. In poultry, phosphorus utilization is influenced by composition of the ileum microbiota and host genetics. In our study, we analyzed the impact of host genetics on composition of the ileum microbiota and the relationship of the relative abundance of ileal bacterial genera with phosphorus utilization and related quantitative traits in Japanese quail. An F2 cross of 758 quails was genotyped with 4k genome-wide single nucleotide polymorphisms (SNPs) and composition of the ileum microbiota was characterized using target amplicon sequencing. Heritabilities of the relative abundance of bacterial genera were estimated and quantitative trait locus (QTL) linkage mapping for the host was conducted for the heritable genera. Phenotypic and genetic correlations and recursive relationships between bacterial genera and quantitative traits were estimated using structural equation models. A genomic best linear unbiased prediction (GBLUP) and microbial (M)BLUP hologenomic selection approach was applied to assess the feasibility of breeding for improved phosphorus utilization based on the host genome and the heritable part of composition of the ileum microbiota. Results Among the 59 bacterial genera examined, 24 showed a significant heritability (nominal p ≤ 0.05), ranging from 0.04 to 0.17. For these genera, six genome-wide significant QTL were mapped. Significant recursive effects were found, which support the indirect host genetic effects on the host’s quantitative traits via microbiota composition in the ileum of quail. Cross-validated microbial and genomic prediction accuracies confirmed the strong impact of microbial composition and host genetics on the host’s quantitative traits, as the GBLUP accuracies based on the heritable microbiota-mediated components of the traits were similar to the accuracies of conventional GBLUP based on genome-wide SNPs. Conclusions Our results revealed a significant effect of host genetics on composition of the ileal microbiota and confirmed that host genetics and composition of the ileum microbiota have an impact on the host’s quantitative traits. This offers the possibility to breed for improved phosphorus utilization based on the host genome and the heritable part of composition of the ileum microbiota.