Background Evidence in older adults suggests that higher cardiorespiratory fitness and lower cardiovascular risk are associated with greater cognition. However, given that changes in the brain that lead to cognitive decline begin decades before the onset of symptoms, understanding the mechanisms by which modifiable cardiovascular factors are associated with brain health in midlife is critical and can lead to the development of strategies to promote and maintain brain health as we age. Methods and Results In 501 middle‐aged (aged 40–65 years) adult participants of the BBHI (Barcelona Brain Health Initiative), we found differential associations among cardiorespiratory fitness, cardiovascular risk, and cognition and cortical thickness. Higher cardiorespiratory fitness was significantly associated with better visuospatial abilities and frontal loading abstract problem solving (β=3.16, P =0.049) in the older middle‐aged group (aged 55–65 years). In contrast, cardiovascular risk was negatively associated with better visuospatial reasoning and problem‐solving abilities (β=−0.046, P =0.002), flexibility (β=−0.054, P <0.001), processing speed (β=−0.115, P <0.001), and memory (β=−0.120, P <0.001). Cortical thickness in frontal regions mediated the relationship between cardiorespiratory fitness and cognition, whereas cortical thickness in a disperse network spanning multiple cortical regions across both hemispheres mediated the relationship between cardiovascular risk and cognition. Conclusions The relationships between modifiable cardiovascular factors, cardiorespiratory fitness, and cardiovascular risk, and cognition are present in healthy middle‐aged adults. These relationships are also mediated by brain structure highlighting a potential mechanistic pathway through which higher cardiorespiratory fitness and lower cardiovascular risk can positively impact cognitive function in midlife.