Background: Dietary habits were investigated as environmental risk factors for Attention-Deficit/Hyperactivity Disorder (ADHD). However, no previous studies explored the effects of dietary factors on modifying the role of genetic factors on ADHD. Methods: Based on a Swedish population-based twin study with 1518 twin pairs aged 20–47 years, we tested whether the importance of genetic and environmental effects on ADHD varied as a function of dietary habits. Self-reported dietary habits and ADHD symptoms were collected. Twin methods were used to test the degree to which high-sugar and unhealthy food intake moderated the genetic and environmental influences on ADHD symptoms. Results: In middle-aged adults, genetic influences on inattention symptoms were statistically significantly higher among individuals with higher levels of high-sugar (45%, 95%CI: 25–54%) and unhealthy food intake (51%, 95%CI: 31–60%), compared with those with lower levels of consumption of high-sugar (36%, 95%CI: 25–47%) and unhealthy foods (30%, 95%CI: 20–41%). Similar patterns were also found for the associations between hyperactivity/impulsivity and high-sugar/unhealthy food intake, even though the moderation effects were not statistically significant. Conclusion The present study suggests that genetic factors play a more prominent role in individual differences of ADHD symptoms in the presence of the high consumption of sugar and unhealthy foods. Future longitudinal studies with multiple assessments of ADHD and dietary habits are needed to replicate our findings.