Atrial fibrillation (AF) is the most common cardiac arrhythmia. The prevalence increases with increasing age. In middle-aged men, endurance sport practice is associated with increased risk of AF but there are few studies among elderly people. The aim of this study was to investigate the role of long-term endurance sport practice as a risk factor for AF in elderly men. A cross-sectional study compared 509 men aged 65–90 years who participated in a long-distance cross-country ski race with 1768 men aged 65–87 years from the general population. Long-term endurance sport practice was the main exposure. Self-reported AF and covariates were assessed by questionnaires. Risk differences (RDs) for AF were estimated by using a linear regression model. After multivariable adjustment, a history of endurance sport practice gave an added risk for AF of 6.0 percent points (pp) (95% confidence interval 0.8–11.1). Light and moderate leisure-time physical activity during the last 12 months reduced the risk with 3.7 and 4.3 pp, respectively, but the RDs were not statistically significant. This study suggests that elderly men with a history of long-term endurance sport practice have an increased risk of AF compared with elderly men in the general population.