Aims The aim was to investigate the long-term association between leisure time physical activity and hip areal bone mineral density (aBMD), in addition to change in hip aBMD over time, in 32–86 years old women and men. Methods Data were retrieved from the 2001, 2007–2008, and 2015–2016 surveys of the Tromsø Study, a longitudinal population study in Norway. Leisure time physical activity was assessed by the four-level Saltin-Grimby Physical Activity Level Scale which refers to physical exertion in the past twelve months. Hip aBMD was assessed by Dual-Energy X-ray Absorptiometry. Linear Mixed Model analysis was used to examine long-term associations between physical activity and hip aBMD (n = 6324). In addition, the annual change in hip aBMD was analyzed in a subsample of 3199 participants. Results Physical activity was significantly and positively associated with total hip aBMD in the overall cohort (p<0.005). Participants who reported vigorous activity had 28.20 mg/cm² higher aBMD than those who were inactive (95% CI 14.71; 41.69, controlled for confounders), and even light physical activity was associated with higher aBMD than inactivity (8.32 mg/cm², 95% CI 4.89; 11.76). Associations between physical activity and femoral neck aBMD yielded similar results. Hip aBMD decreased with age in both sexes, although more prominently in women. From 2001 to 2007–2008, aBMD changed by –5.76 mg/cm² per year (95% CI –6.08; –5.44) in women, and –2.31 mg/cm² (95% CI –2.69; –1.93), in men. From 2007–2008 to 2015–2016, the change was –4.45 mg/cm² per year (95% CI –4.84; –4.06) in women, and –1.45 mg/cm² (95% CI –1.92; –0.98) in men. Conclusions In this cohort of adult men and women, physical activity levels were positively associated with hip aBMD in a dose-response relationship. Hip aBMD decreased with age, although more pronounced in women than men.