(1) Background: Physical exercise has been suggested to promote disease progression in patients with arrhythmogenic right ventricular cardiomyopathy (ARVC). We aimed to investigate the exercise performance and ventricular function of ARVC patients during follow-up, while taking into account their adherence to exercise restriction recommendations. (2) Methods: This retrospective study included 49 patients (33 male, 67%) who had an exercise test at baseline and after 4.2 ± 1.6 years. Of the 49 ARVC patients, 27 (55%) were athletes, while 22 (45%) were non-athletes. Of the athletes, 12 (44%) continued intensive sports activity (non-adherent), while 15 (56%) stopped intensive physical activity upon recommendation (adherent). The maximum workload in Watts (W), percentage of the target workload (W%), and double product (DP) factor were measured for all patients. (3) Results: The non-adherent cohort had a significant decrease in physical performance (W at baseline vs. follow-up, p = 0.012; W% at baseline vs. follow-up, p = 0.025; DP-factor at baseline vs. follow-up, p = 0.012) over time. Left ventricular (LV) function (LV ejection fraction at baseline vs. follow-up, p = 0.082) showed a decreasing trend in the non-adherent cohort, while the performance of the adherent cohort remained at a similar level. (4) Conclusions: If intensive sports activities are not discontinued, exercise capacity and left ventricular function of athletes with ARVC deteriorates during follow-up. All patients with ARVC need to strictly adhere to the recommendation to cease intense sports activity in order to halt disease progression.