Olfactory impairment is associated with cognitive impairment in older adults but less is known about the association of olfactory impairment and cognitive function in middle-aged adults. The association between olfactory impairment and cognitive function tests of attention, processing speed and executive and psychomotor function was explored in 2837 participants (21–84 years; mean age 49 years) in the Beaver Dam Offspring Study. Among middle-aged participants (aged 35–64 years), those with impairment on an odor identification test took significantly longer to complete the Trail Making Test (TMT-A and TMT-B) and the Grooved Peg Board (GPB) test, than those without olfactory impairment in regression models adjusted for multiple factors. Similar results were found for the TMT-A and TMT-B, but not the GPB, in the whole cohort. Olfactory impairment was associated with poorer performance on cognitive function tests in a primarily middle-aged cohort.