The aim of this study was to determine the relationship between bone mass (BM) and physical activity (PA) and sedentary behavior (SB) according to frailty status and sex using compositional data analysis. We analyzed 871 older people with an adequate nutritional status. Fried criteria were used to classify by frailty status. Time spent in SB, light intensity PA (LPA) and moderate-to-vigorous intensity PA (MVPA) was assessed from accelerometry for 7 days. BM was determined by dual-energy X-ray absorptiometry (DXA). The combined effect of PA and SB was significantly associated with BM in robust men and women (p ≤ 0.05). In relation to the other behaviors, SB was negatively associated with BM in robust men while BM was positively associated with SB and negatively with LPA and MVPA in robust women. Moreover, LPA also was positively associated with arm BM (p ≤ 0.01). Finally, in pre-frail women, BM was positively associated with MVPA. In our sample, to decrease SB could be a good strategy to improve BM in robust men. In contrast, in pre-frail women, MVPA may be an important factor to consider regarding bone health.