Diet has a key role in the reproductive axis both in males and females. This review aims to analyze the impacts of different dietary patterns on fertility. It appears that the Mediterranean diet has a predominantly protective role against infertility, while the Western diet seems to be a risk factor for infertility. Moreover, we focus attention also on dietary patterns in different countries of the World (Middle Eastern diet, Asian diet). In particular, when analyzing single nutrients, a diet rich in saturated fatty acids, cholesterol, animal proteins, and carbohydrates with high glycemic index is highly associated with male and female infertility. Finally, we evaluate the effects of vegetarian, vegan, and ketogenic diets on fertility, which seem to be still unclear. We believe that comprehension of the molecular mechanisms involved in infertility will lead to more effective and targeted treatments for infertile couples.