During conversation, speakers constantly make choices about how specific they wish to be in their use of referring expressions. In the present study we investigate whether speakers take the listener into account or whether they base their referential choices solely on their own representation of the discourse. We do this by examining the cognitive mechanisms that underlie the choice of referring expression at different discourse moments. Furthermore, we provide insights into how children with Autism Spectrum Disorder (ASD) and Attention Deficit Hyperactivity Disorder (ADHD) use referring expressions and whether their use differs from that of typically developing (TD) children. Children between 6 and 12 years old (ASD: n=46; ADHD: n=37; TD: n=38) were tested on their production of referring expressions and on Theory of Mind, response inhibition and working memory. We found support for the view that speakers take the listener into account when choosing a referring expression: Theory of Mind was related to referential choice only at those moments when speakers could not solely base their choice on their own discourse representation to be understood. Working memory appeared to be involved in keeping track of the different referents in the discourse. Furthermore, we found that TD children as well as children with ASD and children with ADHD took the listener into account in their choice of referring expression. In addition, children with ADHD were less specific than TD children in contexts with more than one referent. The previously observed problems with referential choice in children with ASD may lie in difficulties in keeping track of longer and more complex discourses, rather than in problems with taking into account the listener.