Abstract Background The influence of physical activity on patient-reported recovery of physical functioning after colorectal cancer (CRC) surgery is unknown. Therefore, we studied recovery of physical functioning after hospital discharge by (a) a relative increase in physical activity level and (b) absolute activity levels before and after surgery. Methods We included 327 incident CRC patients (stages I–III) from a prospective observational study. Patients completed questionnaires that assessed physical functioning and moderate-to-vigorous physical activity shortly after diagnosis and 6 months later. Cox regression models were used to calculate prevalence ratios (PRs) of no recovery of physical functioning. All PRs were adjusted for age, sex, physical functioning before surgery, stage of disease, ostomy and body mass index. Results At 6 months post-diagnosis 54% of CRC patients had not recovered to pre-operative physical functioning. Patients who increased their activity by at least 60 min/week were 43% more likely to recover physical function (adjusted PR 0.57 95%CI 0.39–0.82), compared with those with stable activity levels. Higher post-surgery levels of physical activity were also positively associated with recovery (P for trend = 0.01). In contrast, activity level before surgery was not associated with recovery (P for trend = 0.24). Conclusions At 6 month post-diagnosis, about half of CRC patients had not recovered to preoperative functioning. An increase in moderate-to-vigorous physical activity after CRC surgery was associated with enhanced recovery of physical functioning. This benefit was seen regardless of physical activity level before surgery. These associations provide evidence to further explore connections between physical activity and recovery from CRC surgery after discharge from the hospital.