ABSTRACT We present measurements of the rest-frame ultraviolet luminosity function (LF) at redshifts z = 3, z = 4, and z = 5, using 96894, 38655, and 7571 sources, respectively, to map the transition between active galactic nuclei (AGN) and galaxy-dominated ultraviolet emission shortly after the epoch of reionization (EoR). Sources are selected using a comprehensive photometric redshift approach, using 10 $\, {\rm deg}^2$ of deep extragalactic legacy fields covered by both HSC and VISTA. The use of template fitting spanning a wavelength range of 0.3–2.4 μm achieves 80–90 per cent completeness, much higher than the classical colour–colour cut methodology. The measured LF encompasses −26 < MUV < −19.25. This is further extended to −28.5 < MUV < −16 using complementary results from other studies, allowing for the simultaneous fitting of the combined AGN and galaxy LF. We find that there are fewer UV luminous galaxies (MUV < −22) at z ∼ 3 than z ∼ 4, indicative of an onset of widespread quenching alongside dust obscuration, and that the evolution of the AGN LF is very rapid, with their number density rising by around two orders of magnitude from 3 < z < 6. It remains difficult to determine if a double power law functional form is preferred over the Schechter function to describe the galaxy UV LF. Estimating the hydrogen ionizing photon budget from our UV LFs, we find that AGN can contribute to, but cannot solely maintain, the reionization of the Universe at z = 3–5. However, the rapidly evolving AGN LF strongly disfavours a significant contribution within the EoR.