Identifying characteristics associated with fast or slow growth during early life in a pig model will help in the design of nutritional strategies or recommendations during infancy. The aim of this study was to identify if a differential growth during lactation and/or the nursery period may be associated with fecal microbiota composition and fermentation capacity, as well as to leave a print of glucocorticoid biomarkers in the hair. Seventy-five commercial male and female pigs showing extreme growth in the lactation and nursery periods were selected, creating four groups (First, lactation growth, d0–d21; second, nursery growth, d21–d62): Slow_Slow, Slow_Fast, Fast_Slow, and Fast_Fast. At d63 of life, hair and fecal samples were collected. Fast-growing pigs during nursery had higher cortisone concentrations in the hair (p < 0.05) and a tendency to have a lower cortisol-to-cortisone ratio (p = 0.061). Both lactation and nursery growth conditioned the fecal microbiota structure (p < 0.05). Additionally, fast-growing pigs during nursery had higher evenness (p < 0.05). Lactation growth influenced the relative abundance of eight bacterial genera, while nursery growth affected only two bacterial genera (p < 0.05). The fecal butyrate concentration was higher with fast growth in lactation and/or nursery (p < 0.05), suggesting it has an important role in growth, while total SCFA and acetate were related to lactation growth (p < 0.05). In conclusion, piglets’ growth during nursery and, especially, the lactation period was associated with changes in their microbiota composition and fermentation capacity, evidencing the critical role of early colonization on the establishment of the adult microbiota. Additionally, cortisol conversion to cortisone was increased in animals with fast growth, but further research is necessary to determine its implications.