Background: The 10-year Atherosclerotic Cardiovascular Disease risk score is the standard approach to predict risk of incident cardiovascular events, and recently, addition of coronary artery disease (CAD) polygenic scores has been evaluated. Although age and sex strongly predict the risk of CAD, their interaction with genetic risk prediction has not been systematically examined. This study performed an extensive evaluation of age and sex effects in genetic CAD risk prediction. Methods: The population-based Norwegian HUNT2 (Trøndelag Health Study 2) cohort of 51 036 individuals was used as the primary dataset. Findings were replicated in the UK Biobank (372 410 individuals). Models for 10-year CAD risk were fitted using Cox proportional hazards, and Harrell concordance index, sensitivity, and specificity were compared. Results: Inclusion of age and sex interactions of CAD polygenic score to the prediction models increased the C-index and sensitivity by accounting for nonadditive effects of CAD polygenic score and likely countering the observed survival bias in the baseline. The sensitivity for females was lower than males in all models including genetic information. We identified a total of 82.6% of incident CAD cases by using a 2-step approach: (1) Atherosclerotic Cardiovascular Disease risk score (74.1%) and (2) the CAD polygenic score interaction model for those in low clinical risk (additional 8.5%). Conclusions: These findings highlight the importance and complexity of genetic risk in predicting CAD. There is a need for modeling age- and sex-interaction terms with polygenic scores to optimize detection of individuals at high risk, those who warrant preventive interventions. Sex-specific studies are needed to understand and estimate CAD risk with genetic information.