Abstract Background Polygenic risk scores (PRS), which are derived from results of large genome-wide association studies (GWAS), are increasingly propagated for colorectal cancer (CRC) risk stratification. The majority of studies included in the large GWAS consortia were conducted in the United States and Germany, where colonoscopy with detection and removal of polyps has been widely practiced over the last decades. We aimed to assess if and to what extent the history of colonoscopy with polypectomy may alter metrics of the predictive ability of PRS for CRC risk. Methods A PRS based on 140 single nucleotide polymorphisms was compared between 4939 CRC cases and 3797 controls of the DACHS study, a population-based case-control study conducted in Germany. Risk discrimination was quantified according to the history of colonoscopy and polypectomy by areas under the curves (AUCs) and their 95% confidence intervals (CIs). All statistical tests were two-sided. Results AUCs (95% CIs) were higher among subjects without previous colonoscopy (0.622 [0.606–0.639]) than among those with previous colonoscopy and polypectomy (0.568 [0.536–0.601], difference [Δ AUC] = 0.054, p = .004). Such differences were consistently seen in sex-specific groups (women: Δ AUC = 0.073, p = .02; men: Δ AUC = 0.046, p = .048) and age-specific groups (<70 years: Δ AUC = 0.052, p = .07; ≥70 years: Δ AUC= 0.049, p = .045). Conclusions Predictive performance of PRS may be underestimated in populations with widespread use of colonoscopy. Future studies using PRS to develop CRC prediction models should carefully consider colonoscopy history in order to provide more accurate estimates.