Previous studies on the Highly Sensitive Person Scale (HSPS) were primarily conducted in highly selective, mostly student populations. The objective of the present study was to examine the factor structure of the Dutch HSPS in the general population ( n = 998), prolonged fatigue ( n = 340), and chronic pain samples ( n = 283), and investigate its psychometric properties. The factorial structure of the HSPS was assessed by an exploratory factor analysis, followed by a confirmatory factor analysis. To this end, the overall sample consisting of 1,621 subjects was randomly split into two subsamples. The factor structure was subsequently confirmed in the general population, fatigue, and chronic pain samples separately. A bi-factor model, consisting of a general factor and three separate factors, provided the best fit to the data in each sample. The three separate factors, capturing different dimensions of sensory processing sensitivity, were labeled “Ease of Excitation” (11 items), “Sensory and Aesthetic Sensitivity” (5 items), and “Low Sensory Threshold” (5 items). Internal consistency was satisfactory. Distinct patterns of associations were found between these factors and (un)related personality constructs.