Abstract Background Autism and psychosis co-occur at elevated rates, with implications for clinical outcomes, functioning, and suicidality. The PANSS-Autism-Severity-Score (PAUSS) is a measure of autism trait severity which has not yet been validated externally or longitudinally. Study Design Participants were derived from the GROUP and SCOPE datasets. Participants included 1448 adults with schizophrenia spectrum disorder (SSD), 800 SSD-siblings, 103 adults diagnosed with an autistic spectrum condition (ASC), and 409 typically-developing controls (TC). Analyses from the original validation study were conducted with SSD participants, and extended into ASC, SSD-sibling, and TC participants. Test–retest reliability of the PAUSS at 2-weeks and long-term stability 3 and 6-years was also examined. Study Results Results differed in important ways from the original validation. SSD participants reported higher PAUSS scores than other groups, with only a fraction of ASC participants scoring as “PAUSS-Autistic.” Cronbach’s alpha was acceptable for the SSD cohort only. Two-week stability of the PAUSS was fair to good for all PAUSS scores. Long-term stability was poor for most PAUSS items but fair for total PAUSS score. Conclusions Results suggest that the PAUSS does not appear appropriate for assessing autism, with the low rate of PAUSS-Autistic in the ASC population suggesting the PAUSS may not accurately reflect characteristics of autism. The relative lack of long-term stability is cause for concern and suggestive that the PAUSS is capturing features of psychosis rather than autism traits.