We analyze how passive galaxies at z $∼$ 1.5 populate the mass-size plane as a function of their stellar age, to understand if the observed size growth with time can be explained with the appearance of larger quenched galaxies at lower redshift. We use a sample of 32 passive galaxies extracted from the Wide Field Camera 3 Infrared Spectroscopic Parallel (WISP) survey with spectroscopic redshift 1.3 $\lesssim$ z $\lesssim$ 2.05, specific star-formation rates lower than 0.01 Gyr$^{-1}$, and stellar masses above 4.5 $\times$ 10$^{10}$ M$_⊙$. All galaxies have spectrally determined stellar ages from fitting of their rest-frame optical spectra and photometry with stellar population models. When dividing our sample into young (age $≤$ 2.1 Gyr) and old (age $>$ 2.1 Gyr) galaxies we do not find a significant trend in the distributions of the difference between the observed radius and the one predicted by the mass-size relation. This result indicates that the relation between the galaxy age and its distance from the mass-size relation, if it exists, is rather shallow, with a slope alpha $\gtrsim$ -0.6. At face value, this finding suggests that multiple dry and/or wet minor mergers, rather than the appearance of newly quenched galaxies, are mainly responsible for the observed time evolution of the mass-size relation in passive galaxies. ; Comment: Accepted for publication in ApJ Letters; 6 pages, 3 figures, 1 table