Abstract Background Successful scale-up of antiretroviral therapy (ART) during pregnancy has minimized infant HIV acquisition, and over 1 million infants are born HIV-exposed but uninfected (HEU), with an increasing proportion also exposed in utero to maternal ART. While benefits of ART in pregnancy outweigh risks, some studies have reported associations between in utero ART exposure and impaired fetal growth, highlighting the need to identify the safest ART regimens for use in pregnancy. Methods We compared birth anthropometrics of infants who were HEU with those HIV-unexposed (HU) in Cape Town, South Africa. Pregnant women had gestational age assessed by ultrasound at enrolment. Women living with HIV were on ART (predominately tenofovir-emtricitabine-efavirenz) either prior to conception or initiated during pregnancy. Birth weights and lengths were converted to weight-for-age (WAZ) and length-for-age (LAZ) z-scores using Intergrowth-21st software. Linear regression was used to compare mean z-scores adjusting for maternal and pregnancy characteristics. Results Among 888 infants, 49% (n = 431) were HEU and 51% (n = 457) HU. Of 431 HEU infants, 62% (n = 268) were exposed to HIV and antiretrovirals (ARVs) from conception and 38% (n = 163) were exposed to ARVs during gestation but after conception (median fetal ARV exposure of 21 weeks [IQR; 17–26]). In univariable analysis, infants who were HEU had lower mean WAZ compared with HU [β = − 0.15 (95% Confidence Interval (CI): − 0.28, − 0.020)]. After adjustment for maternal age, gravidity, alcohol use, marital and employment status the effect remained [adjusted β − 0.14 (95%CI: − 0.28, − 0.01]. Similar differences were noted for mean LAZ in univariable [β − 0.20 (95%CI: − 0.42, − 0.01] but not multivariable analyses [adjusted β − 0.18 (95%CI: − 0.41, + 0.04] after adjusting for the same variables. Mean WAZ and LAZ did not vary by in utero ARV exposure duration among infants who were HEU. Conclusion In a cohort with high prevalence of ART exposure in pregnancy, infants who were HEU had lower birth WAZ compared with those HU. Studies designed to identify the mechanisms and clinical significance of these disparities, and to establish the safest ART for use in pregnancy are urgently needed.