Chronic rhinosinusitis (CRS) is a common disease that substantially impairs the quality of life (QoL). Here, we aimed to assess patients’ QoL in different subtypes of CRS and correlated this with nasal polyp size to improve the clinical understanding of the burden of disease. In this retrospective single-center study, 107 patients with the following diagnoses were analyzed: CRS without nasal polyps (CRSsNP), CRS with nasal polyps (CRSwNP), or aspirin-exacerbated respiratory disease (AERD). Sino-Nasal Outcome Test-20 German Adapted Version (SNOT-20 GAV) scores and their correlation with endoscopic Total Polyp Scores (TPS) were evaluated. The mean SNOT-20 GAV scores were highest in patients with AERD (AERD = 43.4, CRSwNP = 36.3, CRSsNP = 30.9). A statistically significant correlation of total SNOT-20 GAV score with TPS was observed in CRSwNP patients (r = 0.3398, p = 0.0195), but not in AERD patients (r = 0.2341, p = 0.1407). When analyzing single SNOT-20 parameters, a strong correlation with TPS was observed for blockage/congestion of the nose, particularly in AERD patients (r = 0.65, p < 0.0001). The impact of nasal polyp size on the QoL differs amongst the subgroups of CRS. Nasal symptoms have the greatest impact on QoL in patients suffering from AERD. CRSwNP and AERD patients should be separately analyzed in clinical investigations and interpretations due to significant differences in QoL.