The WHO refers to human milk as the nutritional gold standard for term infants. Human milk contains many immunomodulatory compounds, including oligosaccharides. Human-milk oligosaccharides can serve as prebiotics because the nondigestible oligosaccharides present in human milk show a clear bifidogenic effect on the gut microbiota. Dietary oligosaccharide structures that have prebiotic effects similar to human-milk oligosaccharides include galacto-oligosaccharides, fructo-oligosaccharides, and pectin-derived acidic oligosaccharides. Both animal studies and human clinical trials showed that dietary intervention with these dietary oligosaccharides in early life could lead to the prevention of atopic dermatitis, food allergy, and allergic asthma. The immune-modulating effects of these oligosaccharides are likely assisted via alteration of the intestinal microbiota or in a microbiota-independent manner by direct interaction on immune cells or both. In this review, an overview of the prebiotic role of dietary oligosaccharides on the microbiota and the microbiota-independent immune modulation by these prebiotics is provided. In addition, recent publications that report on the pathways by which the oligosaccharides might exert their direct immunomodulatory effect are summarized.