Background: The risk of developing atherosclerotic cardiovascular disease (ASCVD) is unknown for subjects with both gallstones and renal stones, nor is it known whether there is a difference in the risk between gallstones and renal stones. This study aimed to determine the risk relationship between gallstones and renal stones and the risk of ASCVD in a male population. Methods: We recruited 6371 eligible males aged 40 to 79 years old who did not have a documented ASCVD history. The ten-year ASCVD risk was calculated using the pooled cohort equations developed by the American College of Cardiology (ACC) and the American Heart Association (AHA). The ASCVD risk score was classified as a low risk (<7.5%), an intermediate risk (7.5% to 19.9%), or a high risk (≥20%). The diagnosis of gallstones and renal stones was established based on the results of abdominal sonography. Results: Both gallstones and renal stones were associated with a high level of intermediate risk (OR = 3.21, 95% CI = 1.89–5.49, p < 0.001) and high risk (OR = 3.01, 95% CI = 1.48–6.12, p < 0.001), compared to individuals with no stones at all, after adjusting for the effects of other clinical variables. The possession of gallstones was associated with a higher level of high ASCVD risk (OR = 1.84, 95% CI = 1.31–2.59, p < 0.05) than that of renal stones. Conclusions: The ASCVD risk was higher for males with gallstones than for those with renal stones. Men with both types of stones faced a risk of ASCVD that was three times higher than that of men without stones.