Changes in physical performance (PP) occur with aging, and understanding the magnitude of these changes over time is important. This study evaluated changes in Gait Speed (GS) and Timed Up and Go (TUG) performance and their association with related factors among community-dwelling older adults over a period of five to six years. A cohort study of 476 older adults with baseline assessment conducted in 2014 and reassessment in 2019–2020 was carried out. Associations between PP changes over time and sociodemographic, behavioral, and health variables were evaluated using mixed linear models. Approximately 68% of the participants declined PP; 20% had no relevant change in GS and 9% in TUG time (sustained PP); 12% increased GS, and 23% decreased TUG time (improved PP). Being male (p = 0.023), living without a partner/separated (p = 0.035), higher education (p = 0.019), and alcohol consumption in the prior month (p = 0.045) were associated with decreased GS, while older age (p < 0.001), having lower socioeconomic status (p < 0.004), physical inactivity (p = 0.017), and being overweight (p = 0.007) were associated with increased TUG time. PP declined for most participants. Factors most strongly associated with PP decline are non-modifiable. The high prevalence of PP decline over time signals the importance of including physical tests in yearly health assessments.