Progressive deterioration of male reproductive function is occurring in Western countries. Environmental factors and unhealthy lifestyles have been implicated in the decline of testosterone levels and sperm production observed in the last fifty years. Among unhealthy lifestyles, substance and drug abuse is a recognized cause of possible alterations of steroidogenesis and spermatogenesis. Alcohol, opioids and anabolic-androgenic steroids are capable to reduce testosterone production in male interfering with testicular and/or hypothalamic-pituitary function. Other substances such as nicotine, cannabis, and amphetamines alter spermatogenesis inducing oxidative stress and subsequent apoptosis in testicular tissue. Substance and drug abuse is a potentially reversible cause of hypogonadism, defined as the failure of the testis to produce physiological concentrations of testosterone and/or a normal number of spermatozoa. The identification of the abuse is important because the withdrawal of substance intake can reverse the clinical syndrome. This review summarizes the most important clinical and experimental evidence on the effect of substance abuse on testosterone and sperm production.