A positive association between cardiorespiratory fitness (CRF) and white matter integrity has been consistently reported in older adults. However, it is unknown whether this association exists in adults over 80 with a range of chronic disease conditions and low physical activity participation, which can influence both CRF and brain health. This study examined whether higher CRF was associated with greater microstructural integrity of gray and white matter in areas related to memory and information processing in adults over 80 and examined moderating effects of chronic diseases and physical activity. CRF was measured as time to walk 400 m as quickly as possible with concurrent 3Telsa diffusion tensor imaging in 164 participants (57.1%female, 40.3%black). Fractional anisotropy (FA) was computed for cingulum, uncinate and superior longitudinal fasciculi. Mean diffusivity (MD) was computed for dorsolateral prefrontal cortex, hippocampus, parahippocampus, and entorhinal cortex. Moderating effects were tested using hierarchical regression models. Higher CRF was associated with higher FA in cingulum and lower MD in hippocampus and entorhinal cortex (β, sex-adjusted p: -0.182, 0.019; 0.165, 0.035; and 0.220, 0.006, respectively). Hypertension attenuated the association with MD in entorhinal cortex. Moderating effects of chronic diseases and physical activity in walking and climbing stairs on these associations were not significant. The association of higher CRF with greater microstructural integrity in selected subcortical areas appears robust, even among very old adults with a range of chronic diseases. Intervention studies should investigate whether increasing CRF can preserve memory and information processing by improving microstructure and potential effects of hypertension management.