Male and female infants respond differentially to environmental stimuli, with different growth and neurodevelopmental trajectories. Male infants are more likely to be disadvantaged when subjected to adversity and show a higher risk of perinatal complications. However, the underlying causes of this sex-bias are not well defined and optimising the early life nutritional care may be necessary to minimise the “male disadvantage” that may be experienced early in life. Experimental models have demonstrated that animal milk composition differs according to offspring sex, suggesting that the tailoring of early life nutrition may be one mechanism to maximise health protection and development to infants of both sexes. However, evidence for a sex-specificity in human milk composition is limited and conflicting, with studies documenting higher milk energy content for either male or female infants. These data show sex differences, however, there has been limited compositional analysis of the current data nor strategies proposed for how sex-specific compositional differences in early life nutrition may be used to improve infant health. The present narrative review highlights that an improved understanding of sex-specific human milk composition is essential for promoting optimal infant growth and development.