Evidence has linked sporting leisure time physical activity (sporting‐LTPA) to healthy cognition throughout adulthood. This may be due to the physiological effects of physical activity (PA), or to other, psychosocial facets of sport. We examined associations between sporting‐LTPA and cognition while adjusting for device‐measured PA volume devoid of context, both in midlife (N = 4041) participants from the 1970 British Cohort Study and later‐life (N = 957) participants from the British Regional Heart Study. Independent of device‐measured PA, we identified positive associations between sporting‐LTPA and cognition. Sports with team/partner elements were strongly positively associated with cognition, suggesting LTPA context may be critical to this relationship.