There is inconsistent evidence on a possible association of vitamin D and androgen levels in men. We therefore aim to investigate the association of 25-hydroxyvitamin D (25(OH)D) with androgen levels in a cohort of middle-aged men. This cross-sectional study included 225 men with a median (interquartile range) age of 35 (30–41) years. We measured 25(OH)D, total testosterone (TT) and SHBG concentrations. Hypogonadism was defined as TT <10.4 nmol/L. We found no significant correlation of 25(OH)D and androgen levels. Furthermore, androgen levels were not significantly different across 25(OH)D quintiles. The overall prevalence of hypogonadism was 21.5% and lowest in men within 25(OH)D quintile 4 (82–102 nmol/L). We found a significantly increased risk of hypogonadism in men within the highest 25(OH)D quintile (>102 nmol/L) compared to men in quintile 4 (reference) in crude (OR 5.10, 1.51–17.24, p = 0.009) as well as in multivariate adjusted analysis (OR 9.21, 2.27–37.35, p = 0.002). We found a trend towards increased risk of hypogonadism in men within the lowest 25(OH)D quintile (≤43.9 nmol/L). In conclusion, our data suggest that men with very high 25(OH)D levels (>102 nmol/L) might be at an increased risk of hypogonadism. Furthermore, we observed a trend towards increased risk of hypogonadism in men with very low vitamin D levels indicating a U-shaped association of vitamin D levels and hypogonadism. With respect to risk of male hypogonadism, our results suggest optimal serum 25(OH)D concentrations of 82–102 nmol/L.