Age-related declines in cognition are associated with widespread structural and functional brain changes, including changes in resting-state functional connectivity and gray and white matter status. Recently we have shown that the elasticity of cerebral arteries also explains some of the variance in cognitive and brain health in aging. Here, we investigated how network segregation, cerebral arterial elasticity (measured with pulse-DOT—the arterial pulse based on diffuse optical tomography) and gray and white matter status jointly account for age-related differences in cognitive performance. We hypothesized that at least some of the variance in brain and cognitive aging is linked to reduced cerebrovascular elasticity, leading to increased cortical atrophy and white matter abnormalities, which, in turn, are linked to reduced network segregation and decreases in cognitive performance. Pairwise comparisons between these variables are consistent with an exploratory hierarchical model linking them, especially when focusing on association network segregation (compared with segregation in sensorimotor networks). These findings suggest that preventing or slowing age-related changes in one or more of these factors may induce a neurophysiological cascade beneficial for preserving cognition in aging.