Nutrition is one of the modifiable risk factors for cognitive decline and Alzheimer’s disease (AD) dementia, and is therefore highly relevant in the context of prevention. However, knowledge of dietary quality in clinical populations on the spectrum of AD dementia is lacking, therefore we studied the association between dietary quality and cognitive impairment in Alzheimer’s disease (AD) dementia, mild cognitive impairment (MCI) and controls. We included 357 participants from the NUDAD project (134 AD dementia, 90 MCI, 133 controls). We assessed adherence to dietary guidelines (components: vegetables, fruit, fibers, fish, saturated fat, trans-fat, salt, and alcohol), and cognitive performance (domains: memory, language, visuospatial functioning, attention, and executive functioning). In the total population, linear regression analyses showed a lower vegetable intake is associated with poorer global cognition, visuospatial functioning, attention and executive functioning. In AD dementia, lower total adherence to dietary guidelines and higher alcohol intake were associated with poorer memory, a lower vegetable intake with poorer global cognition and executive functioning, and a higher trans-fat intake with poorer executive functioning. In conclusion, a suboptimal diet is associated with more severely impaired cognition—this association is mostly attributable to a lower vegetable intake and is most pronounced in AD dementia.