Magnetization transfer imaging (MTI) can detect microstructural brain tissue changes and may be helpful in determining age-related cerebral damage. We investigated the association between the magnetization transfer ratio (MTR) in gray and white matter (WM) and cognitive functioning in 355 participants of the Austrian stroke prevention family study (ASPS-Fam) aged 38-86 years. MTR maps were generated for the neocortex, deep gray matter structures, WM hyperintensities, and normal appearing WM (NAWM). Adjusted mixed models determined whole brain and lobar cortical MTR to be directly and significantly related to performance on tests of memory, executive function, and motor skills. There existed an almost linear dose-effect relationship. MTR of deep gray matter structures and NAWM correlated to executive functioning. All associations were independent of demographics, vascular risk factors, focal brain lesions, and cortex volume. Further research is needed to understand the basis of this association at the tissue level, and to determine the role of MTR in predicting cognitive decline and dementia.