Colorectal cancer risk stratification is crucial to improve screening and risk-reducing recommendations, and consequently do better than a one-size-fits-all screening regimen. Current screening guidelines in the UK, USA and Australia focus solely on family history and age for risk prediction, even though the vast majority of the population do not have any family history. We investigated adding a polygenic risk score based on 45 single-nucleotide polymorphisms to a family history model (combined model) to quantify how it improves the stratification and discriminatory performance of 10-year risk and full lifetime risk using a prospective population-based cohort within the UK Biobank. For both 10-year and full lifetime risk, the combined model had a wider risk distribution compared with family history alone, resulting in improved risk stratification of nearly 2-fold between the top and bottom risk quintiles of the full lifetime risk model. Importantly, the combined model can identify people (n = 72,019) who do not have family history of colorectal cancer but have a predicted risk that is equivalent to having at least one affected first-degree relative (n = 44,950). We also confirmed previous findings by showing that the combined full lifetime risk model significantly improves discriminatory accuracy compared with a simple family history model 0.673 (95% CI 0.664–0.682) versus 0.666 (95% CI 0.657–0.675), p = 0.0065. Therefore, a combined polygenic risk score and first-degree family history model could be used to improve risk stratified population screening programs.