Chewing betel nut is common in Taiwan. Although previous studies have shown that chewing betel nuts is associated with adverse health effects, findings about the impact on bone density have been inconsistent. Therefore, the aim of this study was to investigate the correlation between betel nut chewing and calcaneus ultrasound T-score in a longitudinal study of 118,856 participants from the Taiwan Biobank. Of these participants, 27,002 were followed up with for a median of 4 years. The T-score of the calcaneus was measured in the non-dominant foot using ultrasound. Multivariable analysis showed that a history of chewing betel nut (coefficient β = −0.232; p < 0.001) was significantly associated with low baseline T-score in all participants (n = 118,856). In addition, a long duration of betel nut chewing (per 1 year; coefficient β = −0.003; p = 0.022) was significantly associated with a low baseline T-score in the participants with a history of chewing betel nut (n = 7210). Further, a long duration of betel nut chewing (per 1 year; coefficient β = −0.004; p = 0.039) was significantly associated with a low ΔT-score in the participants with a history of chewing betel nut (n = 1778) after 4 years of follow-up. In conclusion, our results showed that betel nut chewing was associated with a decrease in calcaneus ultrasound T-score, and thus, it is important to stop chewing betel nut to help prevent an increased risk of osteoporosis in the Taiwanese population.