Neuroticism, a ‘Big Five’ personality trait, has been associated with sub-clinical traits of both autism spectrum disorder (ASD) and attention deficit hyperactivity disorder (ADHD). The objective of the current study was to examine whether causal overlap between ASD and ADHD traits can be accounted for by genetic and environmental risk factors that are shared with neuroticism. We performed twin-based structural equation modeling using self-report data from 12 items of the Neo Five-Factor Inventory Neuroticism domain, 11 Social Responsiveness Scale items, and 12 Adult ADHD Self-Report Scale items obtained from 3,170 young adult Australian individual twins (1,081 complete pairs). Univariate analysis for neuroticism, ASD, and ADHD traits suggested that the most parsimonious models were those with additive genetic and unique environmental components, without sex limitation effects. Heritability of neuroticism, ASD, and ADHD traits, as measured by these methods, was moderate (between 40% and 45% for each respective trait). In a trivariate model, we observed moderate phenotypic (between 0.45 and 0.62), genetic (between 0.56 and 0.71), and unique environmental correlations (between 0.37and 0.55) among neuroticism, ASD, and ADHD traits, with the highest value for the shared genetic influence between neuroticism and self-reported ASD traits (r_g = 0.71). Together, our results suggest that in young adults, genetic, and unique environmental risk factors indexed by neuroticism overlap with those that are shared by ASD and ADHD.