Age-related dysfunction in dopaminergic neuromodulation is assumed to contribute to age-associated memory impairment. However, to date there are no in vivo data on how structural parameters of the substantia nigra/ventral tegmental area (SN/VTA), the main origin of dopaminergic projections, relate to memory performance in healthy young and older adults. We investigated this relationship in a cross-sectional study including data from the hippocampus and frontal white matter (FWM) and also assessing working memory span and attention. In groups of young and older adults matched for the variance of their age distribution, gender and body mass index, we observed a robust positive correlation between Magnetization Transfer Ratio (MTR)--a measure of structural integrity--of the SN/VTA and FWM with verbal learning and memory performance among older adults, while there was a negative correlation in the young. Two additional imaging parameters, anisotropy of diffusion and diffusion coefficient, suggested that in older adults FWM changes reflected vascular pathology while SN/VTA changes pointed towards neuronal loss and loss of water content. The negative correlation in the young possibly reflected maturational changes. Multiple regression analyses indicated that in both young and older adults, SN/VTA MTR explained more variance of verbal learning and memory than FWM MTR or hippocampal MTR, and contributed less to explaining variance of working memory span. Together these findings indicate that structural integrity in the SN/VTA has a relatively selective impact on verbal learning and memory and undergoes specific changes from young adulthood to older age that qualitatively differ from changes in the FWM and hippocampus.