Previous studies have documented the primarily genetic aetiology for the stronger phenotypic covariance between reading disability and ADHD inattention symptoms, compared to hyperactivity-impulsivity symptoms. In this study, we examined to what extent this covariation could be attributed to “generalist genes” shared with general cognitive ability or to “specialist” genes which may specifically underlie processes linking inattention symptoms and reading difficulties. We used multivariate structural equation modeling on IQ, parent and teacher ADHD ratings and parent ratings on reading difficulties from a general population sample of 1312 twins aged 7.9–10.9 years. The covariance between reading difficulties and ADHD inattention symptoms was largely driven by genetic (45%) and child-specific environment (21%) factors not shared with IQ and hyperactivity-impulsivity; only 11% of the covariance was due to genetic effects common with IQ. Aetiological influences shared among all phenotypes explained 47% of the variance in reading difficulties. The current study, using a general population sample, extends previous findings by showing, first, that the shared genetic variability between reading difficulties and ADHD inattention symptoms is largely independent from genes contributing to general cognitive ability and, second, that child-specific environment factors, independent from IQ, also contribute to the covariation between reading difficulties and inattention symptoms.