Abstract Background To examine the effect of frailty on cognitive decline independent of cerebral small vessel disease (cSVD) and brain atrophy, and whether associations between neuropathology and cognition differed depending on frailty status. Methods The Tasmanian Study of Cognition and Gait was a population-based longitudinal cohort study with data collected at 3 phases from 2005 to 2012. Participants aged 60–85 were randomly selected from the electoral roll. Various data were used to operationalize a 36-item frailty index (FI) at baseline. Brain MRI was undertaken to obtain baseline measures of neuropathology. A neuropsychological battery was used to assess cognition at each time point. Generalized linear mixed models were used to examine the effect of frailty and MRI measures on cognition over time. The associations between MRI measures and cognition were explored after stratifying the sample by baseline frailty status. All analyses were adjusted for age, sex, and education. Results A total of 385 participants were included at baseline. The mean age was 72.5 years (standard deviation [SD] 7.0), 44% were female (n = 171). In fully adjusted linear mixed models, frailty (FI × time β −0.001, 95% confidence interval [CI] −0.003, −0.001, p = .03) was associated with decline in global cognition, independent of brain atrophy, and cSVD. The association between cSVD and global cognition was significant only in those with low levels of frailty (p = .03). Conclusion These findings suggest that frailty is an important factor in early cognitive dysfunction, and measuring frailty may prove useful to help identify future risk of cognitive decline.