Purpose Improvements in magnetic resonance imaging (MRI), total mesorectal excision (TME) surgery, and the use of (chemo)radiotherapy ([C]RT) have improved local control of rectal cancer; however, we have been unable to eradicate local recurrence (LR). Even in the face of TME and negative resection margins (R0), a significant proportion of patients with enlarged lateral lymph nodes (LLNs) suffer from lateral LR (LLR). Japanese studies suggest that the addition of an LLN dissection (LLND) could reduce LLR. This multicenter pooled analysis aims to ascertain whether LLNs actually pose a problem and whether LLND results in fewer LLRs. Patients and Methods Data from 1,216 consecutive patients with cT3/T4 rectal cancers up to 8 cm from the anal verge who underwent surgery in a 5-year period were collected. LLND was performed in 142 patients (12%). MRIs were re-evaluated with a standardized protocol to assess LLN features. Results On pretreatment MRI, 703 patients (58%) had visible LLN, and 192 (16%) had a short axis of at least 7 mm. One hundred eight patients developed LR (5-year LR rate, 10.0%), of which 59 (54%) were LLRs (5-year LLR rate, 5.5%). After multivariable analyses, LLNs with a short axis of at least 7 mm resulted in a significantly higher risk of LLR (hazard ratio, 2.060; P = .045) compared with LLNs of less than 7 mm. In patients with LLNs at least 7 mm, (C)RT plus TME plus LLND resulted in a 5-year LLR of 5.7%, which was significantly lower than that in patients who underwent (C)RT plus TME (5-year LLR, 19.5%; P = .042). Conclusion LLR is still a significant problem after (C)RT plus TME in LLNs with a short axis at least 7 mm on pretreatment MRI. The addition of LLND results in a significantly lower LLR rate.