Aims. The dust content of normal galaxies and the dust mass density (DMD) at high-z (z > 4) are unconstrained given the source confusion and the sensitivity limitations of previous observations. The ALMA Large Program to INvestigate [CII] at Early times (ALPINE), which targeted 118 ultra-violet (UV)-selected star-forming galaxies at 4.4 < z < 5.9, provides a new opportunity to tackle this issue for the first time with a statistically robust dataset. Methods. We exploited the rest-frame far-infrared (FIR) fluxes of 23 galaxies individually detected in their continuum emission, as well as stacked continuum images, to measure the dust content of the 118 UV-selected ALPINE galaxies. We focused on the dust scaling relations and, by comparison with predictions from chemical evolution models, we probed the evolutionary stage of UV-selected galaxies at high-z. By using the observed correlation between the UV luminosity and the dust mass, we estimated the DMD of UV-selected galaxies at z ∼ 5, weighting the galaxies by means of the UV luminosity function. The derived DMD is compared with the value we estimated from ten ALPINE galaxies blindly detected in the FIR continuum, at the redshift of the ALPINE targets. Results. Our ALMA survey allows the exploration for the first time of the dust content in normal star-forming galaxies at z > 4 in a statistically robust sample of sources. The comparison of the observed dust scaling relations with chemical evolution models suggests that ALPINE galaxies are not likely progenitors of disc galaxies, but of intermediate- and low-mass proto-spheroids, resulting in present-day bulges of spiral or elliptical galaxies. Interestingly, this conclusion is in line with the independent morphological analysis that shows that the majority (∼70%) of the dust-continuum detected galaxies have a disturbed morphology. The DMD obtained at z ∼ 5 from UV-selected sources is ∼30% of the value obtained from blind FIR-selected sources, showing that the UV selection misses the most dust-rich, UV-obscured galaxies.