Dairy products (DP) are part of a food group that may contribute to the prevention of physical frailty. We aimed to investigate DP exposure, including total DP, milk, fresh DP and cheese, and their cross-sectional and prospective associations with physical frailty in community-dwelling older adults. The cross-sectional analysis was carried out on 1490 participants from the Three-City Bordeaux cohort. The 10-year frailty risk was examined in 823 initially non-frail participants. A food frequency questionnaire was used to assess DP exposure. Physical frailty was defined as the presence of at least 3 out of 5 criteria of the frailty phenotype: weight loss, exhaustion, slowness, weakness, and low physical activity. Among others, diet quality and protein intake were considered as confounders. The baseline mean age of participants was 74.1 y and 61% were females. Frailty prevalence and incidence were 4.2% and 18.2%, respectively. No significant associations were observed between consumption of total DP or DP sub-types and frailty prevalence or incidence (OR = 1.40, 95%CI 0.65–3.01 and OR = 1.75, 95%CI 0.42–1.32, for a total DP consumption >4 times/d, respectively). Despite the absence of beneficial associations of higher DP consumption on frailty, older adults are encouraged to follow the national recommendations regarding DP.