Trace elements such as zinc, copper, and selenium are essential for reproductive health, but there is limited work examining how circulating trace elements may associate with fertility in humans. The aim of this study was to determine the association between maternal plasma concentrations of zinc, copper, and selenium, and time to pregnancy and subfertility. Australian women (n = 1060) who participated in the multi-centre prospective Screening for Pregnancy Endpoints study were included. Maternal plasma concentrations of copper, zinc and selenium were assessed at 15 ± 1 weeks’ gestation. Estimates of retrospectively reported time to pregnancy were documented as number of months to conceive; subfertility was defined as taking more than 12 months to conceive. A range of maternal and paternal adjustments were included. Women who had lower zinc (time ratio, 1.20 (0.99–1.44)) or who had lower selenium concentrations (1.19 (1.01–1.40)) had a longer time to pregnancy, equivalent to a median difference in time to pregnancy of around 0.6 months. Women with low selenium concentrations were also at a 1.46 (1.06–2.03) greater relative risk for subfertility compared to women with higher selenium concentrations. There were no associations between copper and time to pregnancy or subfertility. Lower selenium and zinc trace element concentrations, which likely reflect lower dietary intakes, associate with a longer time to pregnancy. Further research supporting our work is required, which may inform recommendations to increase maternal trace element intake in women planning a pregnancy.