AbstractObjectiveHistorically, assessment of the psychometric properties of the Positive and Negative Syndrome Scale (PANSS) has had several foci: (1) calculation of reliability indexes, (2) extraction of subdimensions from the scale, and (3) assessment of the validity of the total score. In this study, we aimed to examine the scalability and to assess the clinical performance of the 30‐item PANSS total score as well as the scalability of a shorter version (PANSS‐6) of the scale.MethodsA composite data set of 1073 patients with first‐episode schizophrenia or schizophrenia spectrum disorder was subjected to Rasch analysis of PANSS data from baseline and 4–6 weeks follow‐up.ResultsThe central tests of fit of the Rasch model failed to satisfy the statistical requirements behind item homogeneity for the PANSS‐30 as well as the PANSS‐6 total score. For the PANSS‐30, Differential Item Functioning was pronounced both for the 7‐point Likert scale rating categories and when dichotomizing the rating categories. Subsequently, the Rasch structure analysis in the context of dichotomized items was used to isolate and estimate a systematic error because of item inhomogeneity, as well as a random error. The size of the combined sources of error for the PANSS‐30 total score approximated 20% which is often regarded as clinical cut‐off between response versus no‐response.ConclusionThe results demonstrate the operational consequences of a lack of statistical fit of the Rasch model and suggest that the calculated measure of uncertainty needs to be considered when using the PANSS‐30 total score.