Abstract Evidence suggests that physical activity (PA) is positively associated with (health-related) quality of life (QOL) in colorectal cancer survivors. However, little is known regarding long-term effects of PA on QOL and if prediagnosis PA is associated with QOL in the years after diagnosis. Our study aimed to investigate the association of prediagnosis and postdiagnosis PA with long-term QOL in colorectal cancer survivors. This study is based on a population-based cohort from Germany of 1,781 newly diagnosed colorectal cancer survivors over a 5-year period. PA was assessed at diagnosis and at 5-year follow-up (5YFU). Quality of life was assessed by the European Organisation for Research and Treatment of Cancer C Quality of Life Questionnaire QLQ-C30 at 5YFU. Multivariable linear regression was used to explore associations between prediagnosis and postdiagnosis PA and QOL at 5YFU. No evidence of a positive association between higher levels of prediagnosis PA and better long-term QOL was found. Higher levels of prediagnosis work-related PA and vigorous PA were even associated with decreased QOL in domains such as cognitive [Beta(β) = −2.52, 95% confidence interval (CI) = −3.77, −1.27; β = −1.92, CI = −3.17, −0.67) and emotional functioning (β = −2.52, CI = −3.84, −1.19; β = −2.12, CI = −3.44, −0.80). In cross-sectional analyses, higher postdiagnosis PA was strongly associated with higher QOL. Survivors physically active at both prediagnosis and postdiagnosis as well as survivors who increased their PA between prediagnosis and postdiagnosis reported significantly higher long-term QOL compared with survivors who remained inactive at prediagnosis and postdiagnosis. In this study, higher prediagnosis PA does not appear to be associated with higher QOL among long-term colorectal cancer survivors but our results support the importance of ongoing PA throughout survivorship.