Recreational drug intake may be associated with a range of medical untoward consequences, including male infertility. However, as the related evidence is still limited, the main outcome of this review is to provide a better understanding of the existence of any association between recreational drug misuse and male fertility levels’ decline. Whilst searching the MEDLINE/PubMed, a comprehensive overview of the literature regarding male infertility and substances of abuse (e.g., phytocannabinoids, opiates/opioids, stimulants, ‘herbal highs’, psychedelics, and ‘novel psychoactive substances) was here undertaken. Due to the paucity of robust, high-quality, empirical, human studies, a narrative strategy was here preferred over a systematic approach. Relevant data are qualitatively analyzed and presented in a table. Although most available evidence is in support of a detrimental role of cannabis on human spermatogenesis, a few remaining studies failed to document any effect of this drug on seminal quality, and it is not clear to which extent this drug impacts fertility rates/time to pregnancy. The current understanding of the impact of opiate-, cocaine- and amphetamine/stimulant-misuse on human reproduction is widely unknown, and most studies dealing with this matter represent only an extrapolation of data derived from specific clinical circumstances. Although the message of ‘no smoking, no alcohol and no street drugs’ should always be offered as good health advice to all patients seeking medical help for fertility issues, robust scientific clinical evidence in support of a direct detrimental impact of recreational drugs on spermatogenesis is scant to date.