Cognitive reserve is hypothesized to help people withstand greater brain pathology without manifesting clinical symptoms, and may be regarded as a preventive factor of dementia. It is unclear whether the effect of cognitive reserve is evident only among the older adults or after conversion to dementia, or if it can also be seen earlier in life before the prominent effects of cognitive aging become apparent. While finding a main effect of cognitive reserve on cognitive outcome may be consistent with the reserve hypothesis, in our view, it is unnecessary to invoke the idea of reserve if only a main effect is present. Rather, it is the interaction between a measure of reserve and a brain measure on cognitive outcome that is key for confirming that the effects of brain pathology affect people differently according to their cognitive reserve. We studied whether general cognitive ability at an average age of 20 years, as a direct measure of cognitive reserve, moderates the association between hippocampal volume and episodic memory performance in 494 middle-aged men ages 51 to 60. Whereas there was no statistically significant direct relationship between hippocampal volume and episodic memory performance in middle age, we found a statistically significant interaction such that there was a positive association between hippocampal volume and episodic memory only among people with lower general cognitive ability at age 20, i.e., lower levels of cognitive reserve. Our results provide support for the hypothesis that cognitive reserve moderates the relationship between brain structure and cognition in middle age, well before the onset of dementia.