The present study investigated whether an unhealthy diet and other lifestyle behaviors may modify the genetic susceptibility to impulsivity. A total of 33,047 participants (mean age = 42.1 years, 59.8% females) from the Dutch Lifelines cohort were included. Each diet index and other lifestyle behaviors were tested for their interactions on the effect on the attention-deficit/hyperactivity disorder (ADHD) polygenic risk score (PRS) on impulsivity using a linear regression model with adjustment for covariates. The ADHD PRS was significantly associated with impulsivity (B = 0.03 (95% CI: 0.02, 0.04); p = 2.61 × 10−9). A poorer diet, a higher intake of energy, and a higher intake of fat were all associated with higher impulsivity, and a high intake of energy amplified the effect of ADHD PRS on impulsivity (e.g., for the interaction term of ADHD PRS and highest tertile on intake of energy, B = 0.038 (95% CI: 0.014, 0.062); p = 0.002. The other lifestyle factors, namely short and long sleep duration, current and past smoking, higher alcohol intake, and more time spent on moderate-to-vigorous physical activity were associated with higher impulsivity, but no interaction effect was observed. In conclusion, we found that a high intake of energy exacerbated the genetic susceptibility to impulsivity. Our study helps to improve our understanding of the role of diet and genetic factors on impulsivity.