The present longitudinal study aimed to investigate the burden of disease activity change on health-related quality of life (HRQoL) of patients with inflammatory bowel disease (IBD) during the two different pandemic waves in 2020 and 2021. A sample of 221 IBD patients (recruited during March–May 2020 for T0 and March–May 2021 for T1) was included. The psychological impact of the COVID-19 pandemic (Impact of Event Scale-Revised (IES-R)) and HRQoL (Inflammatory Bowel Disease Questionnaire (IBDQ)) were assessed. Post-traumatic COVID-19-related symptoms (IES-R) were not significantly different across the disease activity-related groups. Conversely, IBDQ was consistently higher in patients with persistent, quiescent disease activity compared to the other groups, as expected. Even after controlling for baseline IES-R, repeated-measures ANCOVA showed a non-significant main effect of time (p = 0.60) but a significant time-per-group interaction effect with a moderate effect size (η2 = 0.08). During the two different phases of pandemic restrictions, IBD-specific HRQoL was modified by disease-related factors such as disease activity, rather than by the post-traumatic symptoms of COVID-19. This lends further weight to the need for developing an evidence-based, integrated, biopsychosocial model of care for patients with IBD to identify subjective and objective factors that affect the burden of disease.