Aging causes some unfavorable morphological and functional changes, such as the decline in bone mineral density (BMD) and physical function. Moderate-to-vigorous physical activity (MVPA) and sedentary time seem to be related with these alterations, but the impact of distinct patterns remains unclear. The aim of this study was to cross-sectionally and prospectively assess the association between objectively measured MVPA and sedentary patterns (bouts and breaks) with BMD and physical function in older adults. The study considered 151 Brazilians (aged ≥ 60 years), out of which 68 participants completed 2-year follow-up measurements. MVPA and sedentary patterns were measured by means of accelerometry, BMD—(total proximal femur and lumbar spine (L1-L4)) by means of dual-energy X-ray absorptiometry (DXA), and physical function—by means of physical tests. In older women, sedentary bouts >60 min were inversely associated with handgrip strength (β = −2.03, 95% CI: from −3.43 to −0.63). The prospective analyses showed that changes in sedentary bouts (20 to 30 min and >60 min) were inversely associated with changes in the lumbar spine’s BMD (β = −0.01, 95% CI: from −0.01 to −0.00 and β = −0.03, 95% CI: from −0.06 to −0.01) and the lumbar spine’s T-score (β = −0.06, 95% CI: from −0.10 to −0.01 and β = −0.27, 95% CI: from −0.49 to −0.04), respectively. In older women, sedentary patterns are cross-sectionally associated with handgrip strength and prospectively associated with BMD independent of MVPA.