Background The aim of this study was to investigate whether increased severity of coronary artery calcium (CAC), an imaging biomarker of subclinical coronary atherosclerosis, is associated with worse cognitive function independent of cardiovascular risk factors in a large population‐based Dutch cohort with broad age range. Methods and Results A cross‐sectional analysis was performed in 4988 ImaLife participants (aged 45–91 years, 58.3% women) without history of cardiovascular disease. CAC scores were obtained using nonenhanced cardiac computed tomography scanning. The CogState Brief Battery was used to assess 4 cognitive domains: processing speed, attention, working memory, and visual learning based on detection task, identification task, 1‐back task, and 1‐card‐learning task, respectively. Differences in mean scores of each cognitive domain were compared among 4 CAC categories (0, 1–99, 100–399, ≥400) using analysis of covariates to adjust for classical cardiovascular risk factors. Age‐stratified analysis (45–54, 55–64, and ≥65 years) was performed to assess whether the association of CAC severity with cognitive function differed by age. Overall, higher CAC was associated with worse performance on 1‐back task after adjusting for classical cardiovascular risk factors, but CAC was not associated with the other cognitive tasks. Age‐stratified analyses revealed that the association of CAC severity with working memory persisted in participants aged 45 to 54 years, while in the elderly this association lost significance. Conclusions In this Dutch population of ≥45 years, increased CAC severity was associated with worse performance of working memory, independent of classical cardiovascular risk factors. The inverse relationship of CAC score categories with working memory was strongest in participants aged 45 to 54 years.