Abstract Recent progress in photonics has led to a renewed interest in time-varying media that change on timescales comparable to the optical wave oscillation time. However, these studies typically overlook the role of material dispersion that will necessarily imply a delayed temporal response or, stated alternatively, a memory effect. We investigate the influence of the medium memory on a specific effect, i.e. the excitation of quantum vacuum radiation due to the temporal modulation. We construct a framework which reduces the problem to single-particle quantum mechanics, which we then use to study the quantum vacuum radiation. We find that the delayed temporal response changes the vacuum emission properties drastically: frequencies mix, something typically associated with nonlinear processes, despite the system being completely linear. Indeed, this effect is related to the parametric resonances of the light-matter system, and to the parametric driving of the system by frequencies present locally in the drive but not in its spectrum.