Abstract Aims Prior studies have reported inconsistent results for the association between sex hormone-binding globulin (SHBG) and cardiovascular disease among men and women. Although it is suggested that SHBG levels change with aging, the exact trend of SHBG across age and cardiovascular risk and the underlying mechanisms of these changes remain to be elucidated. Methods Using data of 3264 men and women from a large population-based cohort study, we first visualized the distribution of serum SHBG levels across age. Second, we computed a cardiovascular risk factor sum score and investigated the mean SHBG levels across categories of the risk factor sum score and stratified per age-category. Next, linear regression models were used to investigate the associations between serum SHBG levels and age and potential regulators of SHBG, including body mass index (BMI), fasting insulin, sex steroids, thyroxine, and triglycerides. Results Among men, a linear increase in SHBG levels with age and among women a U-shaped pattern was observed. Participants with larger number of cardiovascular risk factors had lower SHBG levels. When stratified by age, older participants had higher SHBG levels. A multivariate model including total testosterone and triglyceride levels in men and total testosterone, triglycerides, BMI, and fasting insulin in women explained, respectively, 46.2% and 31.8% of the variance in SHBG levels. Conclusion We observed a clear sex-specific pattern for SHBG levels with age. Our findings highlight the importance of taking into account the age-related changes in SHBG levels to avoid controversial results in the assessment of the cardiovascular risk associated with SHBG.