For most optoelectronic applications of graphene a thorough understanding of the processes that govern energy relaxation of photoexcited carriers is essential. The ultrafast energy relaxation in graphene occurs through two competing pathways: carrier-carrier scattering -- creating an elevated carrier temperature -- and optical phonon emission. At present, it is not clear what determines the dominating relaxation pathway. Here we reach a unifying picture of the ultrafast energy relaxation by investigating the terahertz photoconductivity, while varying the Fermi energy, photon energy, and fluence over a wide range. We find that sufficiently low fluence ($\lesssim$ 4 $μ$J/cm$^2$) in conjunction with sufficiently high Fermi energy ($\gtrsim$ 0.1 eV) gives rise to energy relaxation that is dominated by carrier-carrier scattering, which leads to efficient carrier heating. Upon increasing the fluence or decreasing the Fermi energy, the carrier heating efficiency decreases, presumably due to energy relaxation that becomes increasingly dominated by phonon emission. Carrier heating through carrier-carrier scattering accounts for the negative photoconductivity for doped graphene observed at terahertz frequencies. We present a simple model that reproduces the data for a wide range of Fermi levels and excitation energies, and allows us to qualitatively assess how the branching ratio between the two distinct relaxation pathways depends on excitation fluence and Fermi energy. ; Comment: Nano Letters 2014