Abstract Background Prenatal exposure to persistent organic pollutants, e.g. polychlorinated biphenyls (PCBs) and polybrominated diphenyl ethers (PBDEs) has been suggested to negatively affect birth weight although epidemiological evidence is still inconclusive. We investigated if prenatal exposure to PCBs and PBDEs is related to birth weight in a Swedish population with background exposure. Methods Breast milk was sampled during the third week after delivery from first-time mothers in Uppsala county, Sweden 1996–2010 (POPUP cohort) (N = 413). Samples were analysed for di- ortho PCBs (CB-138, 153, 180) and tetra- to hexa- brominated PBDEs (BDE-47, 99, 100, 153). Simple and multiple linear regression models were used to investigate associations between lipid-adjusted, ln-transformed PCB and PBDE concentrations, and birth weight. Covariates included in the multivariate regression model were PCB and PBDE exposure, maternal age, pre-pregnancy BMI, weight gain during pregnancy, education, smoking, gender of the infant and gestational length. The effect of including fish consumption was also investigated. Results In the multivariate model, prenatal exposure to di- ortho PCBs was significantly associated with increased birth weight (β = 137; p = 0.02). The result did not change when gestational length was added to the model. An inverse association between PBDE(4) (sum of BDE-47, -99, -100 and −153) and birth weight was observed in the multivariate model including gestational length (β = −106; p = 0.04). Maternal pre-pregnancy BMI and weight gain during pregnancy were important confounders of the association between di- ortho PCBs and birth weight. The associations were not alleviated after adjustment for fish consumption, a major source of PCB and PBDE exposure. The observed associations were stronger for boys than for girls. Conclusions Our results indicate that prenatal exposure to di- ortho PCBs and PBDE(4) may influence birth weight in different directions, i.e. PCB exposure was associated with higher birth weight and PBDE exposure with lower birth weight. Maternal pre-pregnancy BMI and weight gain during pregnancy were important confounders that may hide positive association between di- ortho PCB exposure and birth weight if they are not included in the statistical model. We speculate that even small PCB- and PBDE-induced shifts in the distribution of birth weight may influence future public health in populations with background exposure.