The infant’s gut microbiome is generally rich in the Bifidobacterium genus. The mother’s milk contains natural prebiotics, called human milk oligosaccharides (HMOs), as the third most abundant solid component after lactose and lipids, and of the different gut microbes, infant gut-associated bifidobacteria are the most efficient in assimilating HMOs. Indeed, the fecal concentration of HMOs was found to be negatively correlated with the fecal abundance of Bifidobacterium in infants. Given these results, two HMO molecules, 2′-fucosyllactose and lacto-N-neotetraose, have recently been industrialized to fortify formula milk. As of now, however, our knowledge about the HMO consumption pathways in infant gut-associated bifidobacteria is still incomplete. The recent studies indicate that HMO assimilation abilities significantly vary among different Bifidobacterium species and strains. Therefore, to truly maximize the effects of prebiotic and probiotic supplementation in commercialized formula, we need to understand HMO consumption behaviors of bifidobacteria in more detail. In this review, we summarized how different Bifidobacterium species/strains are equipped with varied gene sets required for HMO assimilation. We then examined the correlation between the abundance of the HMO-related genes and bifidobacteria-rich microbiota formation in the infant gut through data mining analysis of a deposited fecal microbiome shotgun sequencing dataset. Finally, we shortly described future perspectives on HMO-related studies.