Background Substance use (SU) has been linked with impaired cognitive functioning. Evidence comes mainly from clinical studies or studies examining heavy users. Though, the majority of users are not involved in heavy use. This study investigates the association between moderate use and cognition in a population-based sample. Methods A total of 284 young adults with ecstasy, cannabis or alcohol use and a control group were sampled from the EDSP database for participation in the Munich Assessment of Young Adults (MAYA) study. Subjects completed a comprehensive battery of neuropsychological tests (executive functions, working memory and impulsivity). Multiple linear regression models were conducted to examine the relationship between use and cognitive performance. Results Increased ecstasy consumption was associated with increased error-proneness (Stroop task, CANTAB ID/ED-shift, spatial working memory). More frequent cannabis use and more extensive alcohol consumption were associated with a higher degree of impulsiveness. Conclusions Based on mild to moderate SU, little indication of differences in executive functioning was found. For ecstasy use, an increased error-proneness was revealed. The subtle differences in relatively young individuals warrant further investigation in prospective long-term studies to identify subjects at risk, and to examine effects of prolonged patterns of use on executive functioning.