We examined the long-term impact of the 1959–1961 Chinese Famine on the survivors’ cognitive performance in this study. Using data from the 2010 China Family Panel Study, our cohort comparison analysis showed that people who experienced the famine in early childhood (aged 1–3) had a lower score on a vocabulary test and that those who were exposed to the famine in utero did not differ from those born after the famine, probably due to positive selection for the in utero survivors. To deal with the problem of the lack of a comparable control group, we further applied a migrant–stayer comparison approach, with data from the 2016 China Family Panel Study and the 2017 Hong Kong Panel Study of Social Dynamics, to examine the effects of famine exposure at different life stages on adult cognition. We compared the people who stayed in Guangdong with the people who crossed the border to Hong Kong before the famine. The results showed that Guangdong stayers who experienced the famine when they were aged 1–18 had worse performance in immediate word recall. The findings suggested that exposure to malnutrition during childhood has long-term adverse effects on cognitive performance.