Background Cognitive impairment can complicate minor stroke, but there is limited information on risk factors including peak cognitive ability earlier in life. Methods We recruited patients with clinically-evident lacunar or minor non-lacunar ischaemic stroke, recorded clinical features, vascular risk factors, magnetic resonance imaging-detected stroke sub-type and small vessel disease burden. At 1–3 and 12 months after stroke, we assessed educational attainment (years of education), current cognition (Addenbrooke’s Cognitive Examination–Revised), pre-morbid intelligence (National Adult Reading Test) and dependency (modified Rankin Scale). Results We recruited 157 patients (87 lacunar, 64 non-lacunar ischaemic strokes), median age 66 (inter-quartile range 56–74) years, 36/157 (23%) patients had a Addenbrooke’s Cognitive Examination–Revised score < 82 at one to three months, 29/151 (19%) had a Addenbrooke’s Cognitive Examination–Revised < 82 at one year. Lower National Adult Reading Test score (cognitive impairment per point on National Adult Reading Test odds ratio 0.91, 95% confidence interval 0.87, 0.95) and older age (per year of age odds ratio 1.04 (95% confidence interval 1.01, 1.08) predicted one-year cognitive impairment more than stroke severity (per point on National Institute of Health Stroke Scale odds ratio 0.96 (95% confidence interval 0.0.68, 1.31)) or vascular risk factors e.g. hypertension (odds ratio for diagnosis of hypertension 0.52 (95% confidence interval 0.24, 1.15). Cognitive impairment was associated with having more white matter hyper-intensities (odds ratio per point increase in Fazekas score 1.42, 95% confidence interval 1.11, 1.83). Discussion This observational study provides evidence that pre-morbid intelligence quotient and education predict cognition after stroke, and confirms the association between cognitive impairment and small vessel disease. Conclusion Pre-morbid intelligence should be considered in future studies of post-stroke cognition.