The relationship between orbital and spin degrees of freedom in the single-crystals of the hole-doped Pr$_{1-x}$Ca$_{1+x}$MnO$_4$, 0.3 $≤$ $x$ $≤$ 0.7, has been investigated by means of ac-magnetometry and charge transport. Even though there is no cation ordering on the $A$-site, the quenched disorder is extremely weak in this system due to the very similar ionic size of Pr$^{3+}$ and Ca$^{2+}$. A clear asymmetric response of the system to the under- (respective over-) hole doping was observed. The long-ranged charge-orbital order established for half doping ($x$=0.5) subsists in the over-doping case ($x$ $>$ 0.5), albeit rearranged to accommodate the extra holes introduced in the structure. The charge-orbital order is however destabilized by the presence of extra localized electrons (under-doping, $x$ $<$ 0.5), leading to its disappearance below $x$=0.35. We show that in an intermediate under-doped region, with 0.35 $≤$ $x$ $<$ 0.5, the ``orbital-master spin-slave'' relationship commonly observed in half-doped manganites does not take place. The long-ranged charge-orbital order is not accompanied by an antiferromagnetic transition at low temperatures, but by a frustrated short-ranged magnetic state bringing forth a spin-glass phase. We discuss in detail the nature and origin of this spin-glass state, which, as in the half-doped manganites with large quenched disorder, is not related to the macroscopic phase separation observed in crystals with minor defects or impurities.