Abstract Background Sensorineural impairments and cardiovascular risk factors (CVRF) and disease (CVD) in midlife may be important predictors of future cognitive health, but longitudinal studies that include multiple sensorineural measures in middle-aged adults are lacking. Methods Hearing, vision, and olfaction, and CVRF and CVD were measured at the Beaver Dam Offspring Study baseline (2005–2008) examination. The Mini-Mental State Examination and Trail Making Tests A and B were administered at all phases and additional cognitive function measures were obtained at 5 (2010–2013) and 10 years (2015–2017). Cox proportional hazards models were used to evaluate associations between baseline sensorineural impairments, CVRF, CVD, and 10-year cumulative incidence of cognitive impairment and decline. Results There were 2,556 participants (22–84 years) without cognitive impairment at baseline and data from at least one follow-up. In a multivariable model including age, sex, education, and head injury, visual impairment (hazard ratio = 2.59, 95% confidence interval = 1.34, 5.02), olfactory impairment (hazard ratio = 3.18, 95% confidence interval = 1.53, 6.59), CVD (hazard ratio = 2.37, 95% confidence interval = 1.24, 4.52), and not consuming alcohol in the past year (hazard ratio = 2.21, 95% confidence interval = 1.16, 4.19) were associated with the 10-year cumulative incidence of cognitive impairment. Current smoking and diabetes were associated with increased risk, and exercise with decreased risk, of 10-year decline in cognitive function. Conclusions Visual and olfactory impairments, CVRF, and CVD were associated with the 10-year cumulative incidence of cognitive impairment and decline in middle-aged adults. Identifying modifiable factors associated with cognitive decline and impairment in midlife may provide opportunities for prevention or treatment and improve cognitive health later in life.