In a sample of about 45,700 early-type galaxies extracted from SDSS, we find that the shape, normalization, and dispersion around the mean size-stellar mass relation is the same for young and old systems, provided the stellar mass is greater than 3*10^10 Msun. This is difficult to reproduce in pure passive evolution models, which generically predict older galaxies to be much more compact than younger ones of the same stellar mass. However, this aspect of our measurements is well reproduced by hierarchical models of galaxy formation. Whereas the models predict more compact galaxies at high redshifts, subsequent minor, dry mergers increase the sizes of the more massive objects, resulting in a flat size-age relation at the present time. At lower masses, the models predict that mergers are less frequent, so that the expected anti-correlation between age and size is not completely erased. This is in good agreement with our data: below 3*10^10 Msun, the effective radius R_e is a factor of ~2 lower for older galaxies. These successes of the models are offset by the fact that the predicted sizes have other serious problems, which we discuss. Comment: 13 pages, 9 Figures, 1 Table. Accepted by MNRAS