Understanding the physical principle that governs the stimuli-induced swelling and shrinking kinetics of hydrogels is indispensable for their applications. Here, we show that the shrinking and swelling kinetics of self-healing hydrogels could be intrinsically asymmetric. The structure frustration, formed by the large difference in the heat and solvent diffusions, remarkably slows down the shrinking kinetics. The plateau modulus of viscoelastic gels is found to be a key parameter governing the formation of structure frustration and, in turn, the asymmetric swelling and shrinking kinetics. This work provides fundamental understandings on the temperature-triggered transient structure formation in self-healing hydrogels. Our findings will find broad use in diverse applications of self-healing hydrogels, where cooperative diffusion of water and gel network is involved. Our findings should also give insight into the molecular diffusion in biological systems that possess macromolecular crowding environments similar to self-healing hydrogels.