Introduction It is uncertain whether Assisted Reproductive Technology (ART) is associated with an increased risk of poor breastfeeding outcomes and what could be possible mechanisms. This study aimed to examine the effect of mode of conception on breastfeeding outcomes during the first two months postpartum and identify the potential mediating pathways for this relationship. Methods A retrospective cohort study was conducted in a sample of 3,565 women with live births. Participants were classified by mode of conception as follows: fertile women who conceived naturally (fertile women; n = 2,857), women with infertility who conceived naturally (sub-fertile women; n = 483), and women with infertility who conceived through ART (women with infertility; n = 310). The infant-feeding patterns were assessed with four-time points before two months postpartum. Binary and multinomial logistic regression and causal mediation analyses were performed. Results The rates of breastfeeding initiation and discontinuation across modes of conception were similar. However, infertile and sub-fertile women had 37% (95% CI 1.02, 1.83) and 56% (95% CI 1.06, 2.27) increased risks of introducing formula before the first week postpartum, respectively, and 35% (95% CI 1.01, 1.82) and 52% (95% CI 1.04, 2.24) higher risks of exclusive breastfeeding for less than one week, respectively, compared to fertile women. The relationships were mainly mediated through multiple gestation and admission to neonatal/pediatric intensive care units (NICU/PICU; proportions of mediation were over 50%). The effects of mode of conception on breastfeeding outcomes became not significant in cases of singleton birth. Conclusions Sub-fertile women and women with infertility intended to breastfeed but experienced higher perinatal risks in the early postpartum period. Multiple gestation and admission to NICU/PICU forced them to introduce formula earlier than preferred, thus leading to a shorter duration of exclusive breastfeeding. Single embryo transfer policy and breastfeeding support in NICU/PICU could help those women achieve positive early breastfeeding outcomes.