(1) Background: Evaluation of impact of adjuvant radiation therapy (RT) in patients with oral squamous cell carcinoma of the oral cavity/oropharynx (OSCC) of up to 4 cm (pT1/pT2) and solitary ipsilateral lymph node metastasis (pN1). A non-irradiated group with clinical follow-up was chosen for control, and survival and quality of life (QL) were compared; (2) Methods: This prospective multicentric comprehensive cohort study included patients with resected OSCC (pT1/pT2, pN1, and cM0) who were allocated into adjuvant radiation therapy (RT) or observation. The primary endpoint was overall survival. Secondary endpoints were progression-free survival and QL after surgery; (3) Results: Out of 27 centers, 209 patients were enrolled with a median follow-up of 3.4 years. An amount of 137 patients were in the observation arm, and 72 received adjuvant irradiation. Overall survival did not differ between groups (hazard ratio (HR) 0.98 [0.55–1.73], p = 0.94). There were fewer neck metastases (HR 0.34 [0.15–0.77]; p = 0.01), as well as fewer local recurrences (HR 0.41 [0.19–0.89]; p = 0.02) under adjuvant RT. For QL, irradiated patients showed higher values for the symptom scale pain after 0.5, two, and three years (all p < 0.05). After six months and three years, irradiated patients reported higher symptom burdens (impaired swallowing, speech, as well as teeth-related problems (all p < 0.05)). Patients in the RT group had significantly more problems with mouth opening after six months, one, and two years (p < 0.05); (4) Conclusions: Adjuvant RT in patients with early SCC of the oral cavity and oropharynx does not seem to influence overall survival, but it positively affects progression-free survival. However, irradiated patients report a significantly decreased QL up to three years after therapy compared to the observation group.