The process that quenched star formation in galaxies at intermediate and high redshift is still the subject of considerable debate. One way to investigate this puzzling issue is to study the number density of quiescent galaxies at z ' 2, and its dependence on mass. Here we present the results of a new study based on very deep Ks-band imaging (with the HAWK-I instrument on the VLT) of two HST CANDELS fields (the UKIDSS Ultra-deep survey (UDS) field and GOODS-South). The new HAWK-I data (taken as part of the HUGS VLT Large Program) reach detection limits of Ks > 26 (AB mag).We select a sample of passively-evolving galaxies in the redshift range 1:4 23 a higher redshift population of z ' 3 pBzK galaxies is detected, and dominates the counts at the faintest magnitudes. Finally we compare the observed pBzK number counts with those of quiescent galaxies extracted from four different semi-analytic models. We find that only two of these models reproduce even qualitatively the observed trend in the number counts, and that none of the models provides a statistically acceptable description of the number density of quiescent galaxies at these redshifts. We conclude that the mass function of quiescent galaxies as a function of redshift continues to present a key and demanding challenge for proposed models of galaxy formation and evolution.