Background: During the coronavirus disease 2019 (COVID-19) pandemic, the implementation of social distancing and home confinement measures may elevate the risk of vitamin D deficiency particularly for infants. This study aimed to quantify changes in vitamin D level among infants and toddlers in Hong Kong after the COVID-19 outbreak. Methods: We recruited 303 infants and toddlers aged 2–24 months by stratified random sampling from 1 June 2019 to November 30, 2020. Regression models were used to estimate the effect of time on infants’ serum 25-hydroxyvitamin D (25(OH)D) level overall and by age groups before and after the outbreak. Interrupted time series (ITS) analysis was performed to examine the sustained effect of COVID-19 on their serum 25(OH)D level. Results: The ITS results showed no immediate reduction in serum 25(OH)D level among infants, but a decreasing trend was observed in the subsequent months post-outbreak at a monthly decline rate of −6.32 nmol/L. When analyzed by age group, the magnitude of post-outbreak reduction in 25(OH)D was stronger among younger infants (aged 2–6 months). Conclusion: Guidelines and recommendations should be given to pregnant women and mothers to ensure sufficient vitamin D level in their infants during the COVID-19 period.