Aging is associated with cognitive decline, diminished brain function, regional brain atrophy, and disrupted structural and functional brain connectivity. Understanding brain networks in aging is essential, as brain function depends on large-scale distributed networks. Little is known of structural covariance networks to study inter-regional gray matter anatomical associations in aging. Here, we investigate anatomical brain networks based on structural covariance of gray matter volume among 370 middle-aged to older adults of 45–85 years. For each of 370 subjects, we acquired a T1-weighted anatomical MRI scan. After segmentation of structural MRI scans, nine anatomical networks were defined based on structural covariance of gray matter volume among subjects. We analyzed associations between age and gray matter volume in anatomical networks using linear regression analyses. Age was negatively associated with gray matter volume in four anatomical networks (P < 0.001, corrected): a subcortical network, sensorimotor network, posterior cingulate network, and an anterior cingulate network. Age was not significantly associated with gray matter volume in five networks: temporal network, auditory network, and three cerebellar networks. These results were independent of gender and white matter hyperintensities. Gray matter volume decreases with age in networks containing subcortical structures, sensorimotor structures, posterior, and anterior cingulate cortices. Gray matter volume in temporal, auditory, and cerebellar networks remains relatively unaffected with advancing age.