Aim: Both tricyclic antidepressants (TCAs) and selective serotonin reuptake inhibitors (SSRIs) have been associated with an increased risk of fractures. The serotonin transporter (5-HTT) has been located in the bone and may play a role in bone physiology. We assessed the association between antidepressant drug use, categorized in a therapeutical-based way and on basis of their affinity for the 5-HTT, and the risk of both osteoporotic and non-osteoporotic fractures. Methods: A case–control study was conducted using the PHARMO RLS. Cases were patients with a first hospital admission for a fracture during the study period. Up to four controls were matched to each case on gender, age, geographical area, and index date. Results: We identified 16,717 cases, of whom 59.5% had an osteoporotic fracture, and 61,517 controls. Compared to no use, current use of SSRIs was associated with a statistically significant increased risk of osteoporotic fractures (OR 1.95, 95% CI 1.69–2.26), as was current use of TCAs and non-SSRI/non-TCA antidepressant drugs (ORs 1.37, 95% CI 1.16–1.63 and 1.40, 95% CI 1.06–1.85, respectively). The risk of an osteoporotic fracture was statistically significantly higher for antidepressants with a high affinity for the 5-HTT (OR 1.86, 95% CI 1.63–2.13) compared to antidepressants with a medium or low affinity (OR 1.43, 95% CI 1.19–1.72 (medium) and OR 1.32 95% CI 0.98–1.79 (low) (p