We review the available estimates of the masses of the compact object in Ultraluminous X-ray Sources (ULXs) and critically reconsider the stellar-mass versus intermediate-mass black hole interpretations. Black holes of several hundreds to thousands of $M_⊙$ are not required for the majority of ULXs, although they might be present in the handful of known hyper-luminous ($∼ 10^{41}$ erg s$^{-1}$) objects and/or some sources showing timing features in their power density spectra. At the same time, however, stellar mass BHs may be quite a reasonable explanation for ULXs below $∼ 10^{40}$ erg s$^{-1}$, but they need super-Eddington accretion and some suitable dependence of the beaming factor on the accretion rate in order to account for ULXs above this (isotropic) luminosity. We investigate in detail a 'third way' in which a proportion of ULXs contain $≈ 30-90 M_⊙$ black holes formed in a low metallicity environment and accreting in a slightly critical regime and find that it can consistently account for the properties of bright ULXs. Surveys of ULX locations looking for a statistically meaningful relationship between ULX position, average luminosity and local metallicity will provide a definitive test of our proposal. Comment: 11 pages, 1 figure, accepted by MNRAS