Abstract Background Lateral nodal disease in rectal cancer remains a subject of debate and is treated differently in the East and the West. The predictive value of lateral lymph node and MRI-detected extramural vascular invasion (mrEMVI) features on oncological outcomes was assessed in this study. Methods In this retrospective cohort study, data on patients with cT3–4 rectal cancer within 8 cm from the anal verge were considered over a 5-year period (2009–2013). Lateral lymph node size, malignant features and mrEMVI features were evaluated and related to oncological outcomes. Results In total, 192 patients were studied, of whom 30 (15·6 per cent) underwent short-course radiotherapy and 145 (75·5 per cent) received chemoradiotherapy. A lateral lymph node short-axis size of 10 mm or more was associated with a significantly higher 5-year lateral/presacral local recurrence rate of 37 per cent, compared with 7·7 per cent in nodes smaller than 10 mm (P = 0·041). Enlarged nodes did not result in a higher 5-year rate of distant metastasis (23 per cent versus 27·7 per cent in nodes smaller than 10 mm; P = 0·563). However, mrEMVI positivity was related to more metastatic disease (5-year rate 43 versus 26·3 per cent in the mrEMVI-negative group; P = 0·014), but not with increased lateral/presacral recurrence. mrEMVI occurred in 46·6 per cent of patients with nodes smaller than 10 mm, compared with 29 per cent in patients with nodes of 10 mm or larger (P = 0·267). Conclusion Although lateral nodal disease is more a local problem, mrEMVI mainly predicts distant recurrence. The results of this study showed an unacceptably high local recurrence rate in patients with a short axis of 10 mm or more, despite neoadjuvant (chemo)radiotherapy.