Ungulate impacts on forest understory alter tree species composition, with cascading effects on forest functions and resilience against future climate conditions. Indeed, the ungulate browsing pressure on tree seedlings is species-specific and causes contrasted growth reductions that alter tree recruitment rates. Untangling the effects of browsing from the effects of the other factors driving regeneration success is required to guide the forest and ungulate management. In particular, Fagus sylvatica L. strongly dominates temperate Quercus-Fagus forests close to their climax, and it remains unclear if controlling ungulate populations can maintain tree species diversity in naturally regenerated forests. We addressed this question by monitoring 734 pairs of fenced and unfenced 6-m2 plots across a broad gradient of Cervus elaphus L. abundance in Belgian Quercus-Fagus forests managed by continuous cover forestry. Seedling height, density, and vegetation cover were monitored from 2016 to 2021. Species diversity and ecological affinity for light, temperature, and atmospheric humidity conditions were computed from these measures. With ungulates, the mean growth of Betula pendula Roth. and Sorbus aucuparia L. was negligible, whereas, without ungulates, their growth was higher than the growth of other species. With ungulates, the growth of Fagus sylvatica L. and Picea abies (L.) H. Karst was higher than other species. Quercus (Quercus petreae (Matt.) Liebl and Quercus robur L.) growth was the lowest in all conditions. Finally, Carpinus betulus L. was heavily browsed but still grew higher than its competitors with ungulates. Ungulate browsing can then severely affect seedling growth and likely reduce the diversity of future recruited trees. In the study area, browsing unfavored the regeneration of the species that are less shade tolerant, more-drought tolerant, and more-heat tolerant. It thus accelerates the natural succession and reduces forest resilience to heat and drought events. Such an observation was found valid over a wide study area encompassing contrasting levels of Cervus elaphus L. abundance. Combining further reductions of ungulate populations with foodscape improvement is likely required to maintain species diversity in these forests.