Radiotherapy (RT) or chemoradiotherapy (CRT) are frequently used in rectal cancer, sometimes resulting in complete tumor remission (CR). The predictive capacity of all clinical factors, laboratory values and magnetic resonance imaging parameters performed in routine staging was evaluated to understand what determines an excellent response to RT/CRT. A population-based cohort of 383 patients treated with short-course RT (5 × 5 Gy in one week, scRT), CRT, or scRT with chemotherapy (scRT+CT) and having either had a delay to surgery or been entered into a watch-and-wait program were included. Complete staging according to guidelines was performed and associations between investigated variables and CR rates were analyzed in univariate and multivariate analyses. In total, 17% achieved pathological or clinical CR, more often after scRT+CT and CRT than after scRT (27%, 18% and 8%, respectively, p < 0.001). Factors independently associated with CR included clinical tumor stage, small tumor size (<3 cm), tumor level, and low CEA-value (<3.8 μg/L). Size or stage of the rectal tumor were associated with excellent response in all therapy groups, with small or early stage tumors being significantly more likely to reach CR (p = 0.01 (scRT), p = 0.01 (CRT) and p = 0.02 (scRT+CT). Elevated level of carcinoembryonic antigen (CEA) halved the chance of response. Extramural vascular invasion (EMVI) and mucinous character may indicate less response to RT alone.