Abstract Development of cerebral small vessel disease, a major cause of stroke and dementia, may be influenced by early life factors. It is unclear whether these relationships are independent of each other, of adult socio-economic status or of vascular risk factor exposures. We examined associations between factors from birth (ponderal index, birth weight), childhood (IQ, education, socio-economic status), adult small vessel disease, and brain volumes, using data from four prospective cohort studies: STratifying Resilience And Depression Longitudinally (STRADL) (n = 1080; mean age = 59 years); the Dutch Famine Birth Cohort (n = 118; mean age = 68 years); the Lothian Birth Cohort 1936 (LBC1936; n = 617; mean age = 73 years), and the Simpson’s cohort (n = 110; mean age = 78 years). We analysed each small vessel disease feature individually and summed to give a total small vessel disease score (range 1–4) in each cohort separately, then in meta-analysis, adjusted for vascular risk factors and adult socio-economic status. Higher birth weight was associated with fewer lacunes [odds ratio (OR) per 100 g = 0.93, 95% confidence interval (CI) = 0.88 to 0.99], fewer infarcts (OR = 0.94, 95% CI = 0.89 to 0.99), and fewer perivascular spaces (OR = 0.95, 95% CI = 0.91 to 0.99). Higher childhood IQ was associated with lower white matter hyperintensity burden (OR per IQ point = 0.99, 95% CI 0.98 to 0.998), fewer infarcts (OR = 0.98, 95% CI = 0.97 to 0.998), fewer lacunes (OR = 0.98, 95% CI = 0.97 to 0.999), and lower total small vessel disease burden (OR = 0.98, 95% CI = 0.96 to 0.999). Low education was associated with more microbleeds (OR = 1.90, 95% CI = 1.33 to 2.72) and lower total brain volume (mean difference = −178.86 cm3, 95% CI = −325.07 to −32.66). Low childhood socio-economic status was associated with fewer lacunes (OR = 0.62, 95% CI = 0.40 to 0.95). Early life factors are associated with worse small vessel disease in later life, independent of each other, vascular risk factors and adult socio-economic status. Risk for small vessel disease may originate in early life and provide a mechanistic link between early life factors and risk of stroke and dementia. Policies investing in early child development may improve lifelong brain health and contribute to the prevention of dementia and stroke in older age.