Background: Ageing is a life-long process characterized by a progressive loss of physical fitness compromising strength, flexibility, and agility. The purpose of this study was to use accelerometry to examine the relationship between sedentary time, light physical activity (LPA), and moderate to vigorous physical activity (MVPA) with the elderly’s physical fitness. Additionally, we aimed to examine the association between the aforementioned variables on older adults who fulfilled global recommendations on physical activity for health and on those who did not fulfil these recommendations. Methods: Eighty-three elderly (mean ± SD: 72.14 ± 5.61 years old) of both genders volunteered to participate in this cross-sectional study, being divided into an active group (n = 53; 71.02 ± 5.27 years old) and an inactive group (n = 30; 74.13 ± 5.72 years old) according to the established guidelines. Sedentary and physical activity times were assessed using an ActiGraph® GT1M accelerometer, whereas physical fitness was evaluated with the Senior Fitness Test. Results: MVPA time was correlated with lower body mass index (BMI) ((rs = −0.218; p = 0.048; −0.3 < r ≤ −0.1 (small)) and shorter time to complete the agility test ((rs = −0.367; p = 0.001; −0.5 < r ≤ −0.3 (low)). Moreover, MVPA time was positively correlated with aerobic endurance ((rs = 0.397; p = 0.000; 0.5 < r ≤ 0.3 (low)) and strength ((rs = 0.243; p = 0.027; 0.3 < r ≤ 0.1 (small)). In the inactive group, MVPA time was positively correlated with upper limb flexibility ((rs = 0.400; p = 0.028; 0.5 < r ≤ 0.3 (low)); moreover, sedentary time was negatively correlated with upper limb flexibility ((r = −0.443; p = 0.014; −0.5 < r ≤ −0.3 (low)), and LPA time was negatively correlated with BMI ((r = −0.423; p = 0.020; −0.5 < r ≤ −0.3 (low)). In the active group, MVPA time was correlated with lower BMI ((rs = −0.320; p = 0.020; −0.5 < r ≤ −0.3 (low)), and shorter time to complete agility test ((rs = −0.296; p = 0.031; −0.3 < r ≤ −0.1 (small)). Conclusions: Our results reinforce the importance of promoting MVPA practice among the elderly, thereby allowing physical fitness maintenance or improvement.