Regular use of non-steroidal anti-inflammatory drugs (NSAIDs) was associated with the lower risk of colorectal cancer (CRC). However, whether regular use of NSAIDs could attenuate the effect of genetic risk and environmental risk factors on CRC is unknown. We aimed to evaluate the association of NSAID use, genetic risk, and environmental risk factors with CRC. Using data from a UK Biobank, a Cox proportional hazards model was performed to estimate the risk of CRC according to NSAID use, polygenic risk score, and environmental risk factors. Regular use of NSAIDs was associated with a 36.0% lower risk of CRC. No statistically significant interaction was observed between NSAID use and the genetic risk score (p = 0.190), and between NSAID use and the environmental risk score (p = 0.740). However, regular NSAID use was still associated with lower CRC incidence among subjects with either high environmental risk or high genetic risk. Furthermore, the genetic and environmental risk of CRC were additives. These findings appear to support the chemopreventive effect of regular NSAID use. Furthermore, controlling of modifiable environmental risk factors can reduce the CRC risk, especially among individuals with a moderate or high genetic risk of CRC.