(1) Background: Individualization of treatment is a major challenge in oncology and requires a variety of predictive and prognostic parameters. In addition to tumor biology analyses, baseline health-related quality of life might be a valid tool to predict overall survival. This study was conducted to evaluate the prognostic relevance of baseline quality of life data in patients with rectal cancer. In this context, differences between patients with and without distant metastases were of particular interest. (2) Methods: Our cohort included 258 patients with rectal cancer treated in the radiotherapy department of the University Hospital Erlangen. Patients completed the European Organisation for Research and Treatment of Cancer (EORTC) core quality of life questionnaire (QLQ C30) and colorectal cancer questionnaire (CR38). Clinical and survival data were provided by the Gießener Tumor Documentation System (GTDS) of the Comprehensive Cancer Center Erlangen-EMN (CCC, Friedrich-Alexander University Erlangen-Nuremberg, Erlangen, Germany). Statistical analyses were performed using Kaplan–Meier analyses and univariate and multivariate Cox regression. (3) Results: A cohort of 258 patients with rectal adenocarcinoma was analyzed including 50 patients (19.4%) with metastatic disease. No differences were observed between patients with and without distant metastases in most areas of quality of life studied, with the exception of physical function, loss of appetite, chemotherapy side effects and weight loss. Gender, baseline physical function, sexual function, diarrhea, and weight loss over time had a prognostic value in the entire cohort. Appetite loss was an additional prognostic parameter in patients with distant metastases. (4) Conclusions: The quality of life of patients with metastatic disease differed only slightly from non-metastatic patients. Health-related quality of life data provide prognostic information for patients with rectal cancer.