Objectives: Bone mineral density (BMD) is a complex trait resulting from the interplay of genetic and acquired factors. The objective of this study was to explore the influence of several anthropometric, lifestyle, genetic, and hormonal factors on BMD and analyze the possible differences in men and women. Methods: We studied 572 individuals over 50 years of age (381 postmenopausal women and 191 men). Lumbar spine and femoral neck BMD were measured by dual energy x-ray absorptiometry. The free estrogen index (FEI) was calculated as the ratio of serum estradiol to sex hormone binding globulin in 241 individuals. Three polymorphisms in the genes coding for 17-hydroxylase/liase, sulfotransferase, and 5α-reductase were studied in DNA isolated from blood cells. Results: Body mass index was strongly correlated to spine and femoral BMD both in women and in men (r = 0.32–0.49; P < 0.001). FEI was also independently correlated with spine BMD in both sexes (r = 0.23 and 0.34, P < 0.01), and with femoral neck in women (r = 0.30). Women with G alleles of the sulfotransferase gene tended to have higher spine BMD than those with C alleles (P = 0.025). No other genotype-related differences in BMD were found. Conclusions: In conclusion, the results of this study point toward body weight and estradiol levels as major factors determining BMD both in women and in men. A common polymorphism of the sulfotransferase gene also appears to be associated to spine BMD in women.