Although the issue has been repeatedly explored, data on the impact of the COVID-19 pandemic on children’s sleep quality are inconsistent. To clarify these discrepancies, here we investigate possible age-related differences. During the lockdown, 112 parents of toddlers (0–3 years, N = 61) and pre-schoolers (4–5 years, n = 51) completed an online survey including the Children’s Sleep Habits Questionnaire (CSHQ). Sleep-related items required an additional retrospective judgment, referring to the pre-pandemic period. During the lockdown, sleep schedules were delayed in both age groups whereas sleep quality (CSHQ total scores) improved in pre-schoolers but not in toddlers. Between-groups comparisons revealed that, prior to the lockdown, pre-schoolers showed worse sleep quality than toddlers, whereas this difference disappeared during home confinement. Also, pre-schoolers’ sleep timing was advanced before the lockdown and delayed during the lockdown relative to toddlers’. Our data highlight a significant modulation of age on the impact of the pandemic crisis on sleep, with pre-schoolers experiencing greater effects than toddlers. This profile suggests that factors affecting sleep features have different weights at different ages: sleep patterns would be mainly determined by developmental factors (i.e., biological drive) in younger children, whereas environmental factors (e.g., major lifestyle changes) would have a stronger effect on older ones.