The impact of variants of concern (VoC) on SARS-CoV-2 viral dynamics remains poorly understood and essentially relies on observational studies subject to various sorts of biases. In contrast, experimental models of infection constitute a powerful model to perform controlled comparisons of the viral dynamics observed with VoC and better quantify how VoC escape from the immune response. Here we used molecular and infectious viral load of 78 cynomolgus macaques to characterize in detail the effects of VoC on viral dynamics. We first developed a mathematical model that recapitulate the observed dynamics, and we found that the best model describing the data assumed a rapid antigen-dependent stimulation of the immune response leading to a rapid reduction of viral infectivity. When compared with the historical variant, all VoC except beta were associated with an escape from this immune response, and this effect was particularly sensitive for delta and omicron variant (p<10⁻⁶ for both). Interestingly, delta variant was associated with a 1.8-fold increased viral production rate (p = 0.046), while conversely omicron variant was associated with a 14-fold reduction in viral production rate (p<10⁻⁶). During a natural infection, our models predict that delta variant is associated with a higher peak viral RNA than omicron variant (7.6 log₁₀ copies/mL 95% CI 6.8–8 for delta; 5.6 log₁₀ copies/mL 95% CI 4.8–6.3 for omicron) while having similar peak infectious titers (3.7 log₁₀ PFU/mL 95% CI 2.4–4.6 for delta; 2.8 log₁₀ PFU/mL 95% CI 1.9–3.8 for omicron). These results provide a detailed picture of the effects of VoC on total and infectious viral load and may help understand some differences observed in the patterns of viral transmission of these viruses.