Abstract Background Falls are important causes of injury and mortality in older people, and associated medical costs can be enormous. Physical activity (PA) is a potential preventive factor for falls. However, few studies have examined the effect of different types of PA on fall prevention. This study aimed to evaluate the association between PA levels and the incidence of recurrent falls by type of PA in middle-aged and older people. Methods This cohort study targeted 7,561 community-dwelling individuals aged 40–74 years who did not experience recurrent falls in the year before baseline. Information on PA levels, demographics, body size, lifestyle, and fall/disease history was obtained using a self-administered questionnaire in the baseline survey. Levels of total PA, leisure-time PA, and non-leisure-time PA (occupation, commuting, and housework) were estimated using metabolic equivalent (MET) scores (MET-h/day; hours spent on a given activity per day multiplied by its MET intensity). PA levels were categorized into four groups. Falls were recorded as none, once, or twice or more (recurrent falls). The outcome of the study was the incidence of recurrent falls in the past year before a survey conducted 5 years after the baseline survey. Logistic regression analyses were performed to calculate odds ratios for recurrent falls. Results Higher total PA and non-leisure-time PA levels were associated with a higher risk of recurrent falls (P for trend = 0.0002 and 0.0001, respectively), with the highest total PA and non-leisure-time PA groups having a significantly higher adjusted OR (1.96 [95%CI:1.33–2.88] and 2.15 [95%CI:1.48–3.14], respectively) relative to the lowest group (reference). As for leisure-time PA, the medium group had a significantly lower adjusted OR (0.70 [95%CI:0.49–0.99]) relative to the reference group. By sex, the adjusted OR in the medium leisure-time PA group was significantly lower relative to the reference group in women (0.50 [95%CI: 0.29–0.85]) but not in men. Conclusions Medium level leisure-time PA reduces the risk of recurrent falls in middle-aged and older people, whereas higher level non-leisure-time PA is associated with a higher risk of recurrent falls.