The decline in sperm count rates over the last 50 years appears to parallel the rising prevalence of obesity. As lipids levels are strongly associated with obesity, high lipids levels or hyperlipidemia may thus play an important role in the decline in fertility in addition to other environmental or lifestyle factors. The objective of this population based cohort study was to evaluate the association between men’s serum lipid concentrations and semen quality parameters among 501 male partners of couples desiring pregnancy and discontinuing contraception. Each participant provided prospectively up to two semen samples (94% of men provided one or more semen samples, and 77% of men provided a second sample approximately one month later). Linear mixed effects models were used to estimate the associations between baseline lipid concentrations and semen quality parameters, adjusted for age, body mass index, and race. We found that higher levels of serum total cholesterol, free cholesterol and phospholipids were associated with a significantly lower percentage of sperm with intact acrosome and smaller sperm head area and perimeter. Our results suggest that lipid concentrations may affect semen parameters, specifically sperm head morphology, highlighting the importance of cholesterol and lipid homeostasis for male fecundity.