Background and purposePrevious studies have shown associations between atrial fibrillation (AF) and cognitive decline. We investigated this association in a prospective population study, focusing on whether stroke risk factors modulated this association in stroke‐free women and men.MethodsWe included 4983 participants (57% women) from the fifth survey of the Tromsø Study (Tromsø 5, 2001), of whom 2491 also participated in the sixth survey (Tromsø 6, 2007–2008). Information about age, education, blood pressure, body mass index, lipids, smoking, coffee consumption, physical activity, depression, coronary and valvular heart disease, heart failure and diabetes was obtained at baseline. AF status was based on hospital records. The outcome was change in cognitive score from Tromsø 5 to Tromsø 6, measured by the verbal memory test, the digit–symbol coding test and the tapping test.ResultsMean age at baseline was 65.4 years. The mean reduction in the tapping test scores was significantly larger in participants with AF (5.3 taps/10 s; 95% CI: 3.9, 6.7) compared with those without AF (3.8 taps/10 s; 95% CI: 3.5, 4.1). These estimates were unchanged when adjusted for other risk factors and were similar for both sexes. AF was not associated with change in the digit–symbol coding or the verbal memory tests.ConclusionAtrial fibrillation in stroke‐free participants was independently associated with cognitive decline as measured with the tapping test.