The present study examines the epidemiological evidence for a causal relationship between caffeine consumption and cognitive deterioration in the elderly. Using a population of 641 elderly persons, we examined cognitive functioning, caffeine consumption, magnetic resonance imaging volumetrics, and other factors known to affect cognitive performance. Our findings demonstrate the association between caffeine consumption and lower cognitive change over time to be statistically significant for women only, taking into account multiple confounders, to be dose-dependent and temporarily related (caffeine consumption precedes cognitive change). Mean log transformed white matter lesion/cranial volume ratios were found to be significantly lower in women consuming more than 3 units of caffeine per day after adjustment for age (-1.23 SD=0.06) than in women consuming 2-3 units (-1.04 SD=0.04) or one unit or less (-1.04 SD=0.07, -35% in cm3 compared to low drinkers). This observation is coherent with biological assumptions that caffeine through adenosine is linked to amyloid accumulation and subsequently white matter lesion formation. The significant relationship observed between caffeine intake in women and lower cognitive decline is highly likely to be a true causal relationship and not a spurious association.