While recent ‘big data' analyses discovered structural brain networks that alter with age and relate to cognitive decline, identifying modifiable factors that prevent these changes remains a major challenge. We therefore aimed to determine the effects of common cardiovascular risk factors on vulnerable gray matter (GM) networks in a large and well-characterized population-based cohort. In 616 healthy elderly (258 women, 60–80 years) of the LIFE-Adult-Study, we assessed the effects of obesity, smoking, blood pressure, markers of glucose and lipid metabolism as well as physical activity on major GM-networks derived using linked independent component analysis. Age, sex, hypertension, diabetes, white matter hyperintensities, education and depression were considered as confounders. Results showed that smoking, higher blood pressure, and higher glycated hemoglobin (HbA1c) were independently associated with lower GM volume and thickness in GM-networks that covered most areas of the neocortex. Higher waist-to-hip ratio was independently associated with lower GM volume in a network of multimodal regions that correlated negatively with age and memory performance. In this large cross-sectional study, we found selective negative associations of smoking, higher blood pressure, higher glucose, and visceral obesity with structural covariance networks, suggesting that reducing these factors could help to delay late-life trajectories of GM aging.