Abstract Overt and subclinical hyperthyroidism are associated with increased fracture risk, but whether thyroid hormones are associated with fracture risk in individuals with normal thyroid-stimulating hormone (TSH) has mostly been investigated in women. Therefore, we investigated if serum levels of free thyroxine (FT4) or TSH are associated with fracture risk in Swedish men. We followed (median 12.2 years) elderly men (n = 1825; mean age 75, range 69–81 years) participating in the Gothenburg and Malmö subcohorts of the prospective, population-based MrOS-Sweden study. The statistical analyses included Cox proportional hazards regression. Men receiving levothyroxine treatment were excluded. In our total cohort, serum FT4 (per SD increase) was associated with increased risk of major osteoporotic fractures [MOF, n = 479; fully adjusted hazard ratio (HR) 1.14, 95% confidence interval (CI): 1.05–1.24] and hip fractures (n = 207; HR 1.18, 95% CI: 1.04–1.33). Also, in men with normal TSH (n = 1658), FT4 (per SD increase) was significantly associated with increased risk of MOF and hip fractures. Furthermore, men in the highest FT4 quartile had a 1.5-fold increase in hip fracture risk compared with men in the three lower FT4 quartiles, both in the total population and in men with normal TSH (fully adjusted: HR 1.45, 95% CI: 1.04–2.02 and HR 1.51, 95% CI: 1.07–2.12, respectively). In contrast, the risk of MOF was not statistically different in the highest FT4 quartile compared with the three lower FT4 quartiles. Finally, serum TSH was not associated with fracture risk after full adjustment for covariates. In conclusion, serum FT4, but not serum TSH, is a predictor of hip fracture risk in elderly Swedish men. Additionally, there was an association between FT4 (per SD increase) and the risk of MOF.