Background High blood pressure levels and higher arterial stiffness have been shown to be associated with lower cognition during adulthood, possibly by accumulative changes over time. However, vascular factors may already affect the brain during early life. Methods and Results We examined the relation between cognition and vascular factors within 5853 children from the Generation R Study (mean age 6.2 years) and 5187 adults from the Rotterdam Study (mean age 61.8 years). Diastolic and systolic blood pressure and arterial stiffness were assessed, the latter by measuring pulse‐wave velocity and pulse pressure. For cognition, the Generation R Study relied on nonverbal intelligence, whereas the Rotterdam Study relied on a cognitive test battery to calculate the g‐factor, a measure of global cognition. In the Generation R Study, standardized diastolic blood pressure showed a significant association with standardized nonverbal intelligence (β=−0.030, 95% confidence interval=[−0.054; −0.005]) after full adjustment. This association held up after excluding the top diastolic blood pressure decile (β=−0.042 [−0.075; −0.009]), suggesting that the relation holds in normotensives. Within the Rotterdam Study, standardized cognition associated linearly with standardized systolic blood pressure (β=−0.036 [−0.060; −0.012]), standardized pulse‐wave velocity (β=−0.064 [−0.095; −0.033]), and standardized pulse pressure (β=−0.044 [−0.069; −0.020], and nonlinearly with standardized diastolic blood pressure (quadratic term β=−0.032 [−0.049; −0.015]) after full adjustment. Conclusions Blood pressure and cognition may already be related in the general population during early childhood, albeit differently than during adulthood.