Abstract Aims Alcohol consumption is a modifiable and plausible risk factor for age-related cognitive decline but more longitudinal studies investigating the association are needed. Our aims were to estimate associations of adult-life alcohol consumption and consumption patterns with age-related cognitive decline. Methods We investigated the associations of self-reported adult-life weekly alcohol consumption and weekly extreme binge drinking (≥10 units on the same occasion) with changes in test scores on an identical validated test of intelligence completed in early adulthood and late midlife in 2498 Danish men from the Lifestyle and Cognition Follow-up study 2015. Analyses were adjusted for year of birth, retest interval, baseline IQ, education and smoking. Results Men with adult-life alcohol consumption of more than 28 units/week had a larger decline in IQ scores from early adulthood to late midlife than men consuming 1–14 units/week (B29–35units/week = −3.6; P < 0.001). Likewise, a 1-year increase in weekly extreme binge drinking was associated with a 0.12-point decline in IQ scores (P < 0.001). Weekly extreme binge drinking explained more variance in IQ changes than average weekly consumption. In analyses including mutual adjustment of weekly extreme binge drinking and average weekly alcohol consumption, the estimated IQ decline associated with extreme binge drinking was largely unaffected, whereas the association with weekly alcohol consumption became non-significant. Conclusions Adult-life heavy alcohol consumption and extreme binge drinking appear to be associated with larger cognitive decline in men. Moreover, extreme binge drinking may be more important than weekly alcohol consumption in relation to cognitive decline.