We use simple energetic arguments to estimate the contribution of massive X-ray binaries and supernova remnants to the cosmic X-ray background (XRB) at energies in excess of 2 keV. Recent surveys have shown that AGN probably account for most of the hard XRB (E > 2 keV), but there have been many suggestions that star-forming galaxies could emerge at fainter fluxes and perhaps account for a significant fraction of the soft and hard X-ray energy density. Assuming that the formation rate of massive X-ray binaries (MXRBs) traces the global star-formation rate, we find that their integrated contribution to the hard XRB can be estimated and is shown to be small (at less than the 1% level). Similarly, the integrated flux of SN is also shown to be insignificant, or at most comparable to MXRBs. AGN therefore remain the most viable candidates for producing the hard XRB, unless additional processes can be shown to dominate the global hard X-ray emission in distant starburst galaxies. ; Comment: MNRAS pink pages, accepted