Context. The classification scheme for high redshift galaxies is complex at the present time, with simple colour-selection criteria (i.e. EROs, IEROs, LBGs, DRGs, BzKs), resulting in ill-defined properties for the stellar mass and star formation rate of these distant galaxies. Aims. The goal of this work is to investigate the properties of different classes of high-$z$ galaxies, focusing in particular on the stellar masses of LBGs, DRGs, and BzKs, in order to derive their contribution to the total mass budget of the distant Universe. Methods. We used the GOODS-MUSIC catalog, containing ~3000 $K$s-selected (~10 000 $z$-selected) galaxies with multi-wavelength coverage extending from the U band to the Spitzer $8~μ$m band, with spectroscopic or accurate photometric redshifts. We selected samples of BM/BX/LBGs, DRGs, and BzK galaxies to discuss the overlap and the limitations of these criteria, which can be overridden by a selection criterion based on physical parameters. We then measured the stellar masses of these galaxies and computed the stellar mass density (SMD) for the different samples up to redshift $≃$$4$. Results. We show that the BzK-PE criterion is not optimal for selecting early type galaxies at the faint end. On the other hand, BzK-SF is highly contaminated by passively evolving galaxies at red $z-K$s colours. We find that LBGs and DRGs contribute almost equally to the global SMD at $z\ge 2$ and, in general, that star-forming galaxies form a substantial fraction of the universal SMD. Passively evolving galaxies show a strong negative density evolution from redshift 2 to 3, indicating that we are witnessing the epoch of mass assembly of such objects. Finally we have indications that by pushing the selection to deeper magnitudes, the contribution of less massive DRGs could overtake that of LBGs. Deeper surveys, like the HUDF, are required to confirm this suggestion.