Introduction: Both intensity-modulated radiotherapy (IMRT) and image-guided radiotherapy (IGRT) have been shown to independently reduce late rectal toxicity for men with prostate cancer (PC) treated with radiotherapy. We explore whether IMRT offers further reductions in late rectal toxicity for men already being managed with IGRT and compare this with more gradual changes over time. Methods: Between 2007 and 2009, 103 patients with PC were treated with three-dimensional conformal radiotherapy (3D-CRT n = 52) or IMRT (n = 51) with doses of 74–78 Gy at 2 Gy per fraction. All men had daily IGRT using intra-prostatic gold fiducials. The primary endpoint was incidence of grade ≥2 late rectal toxicity as graded by the Radiation Therapy Oncology Group scale. Results: The relative risk of late grade ≥2 rectal toxicity in patients treated with IMRT was 68% less than seen with image-guided 3D-CRT at 36 months post-treatment (7% versus 22%; hazard ratio = 0.32, P = 0.03). IMRT remained a significant protective factor in a multivariate analysis. A discriminant analysis showed that the relative volume of rectal wall exposed to doses over 50 Gy was most strongly associated with late rectal toxicity. Controlling for duration of follow-up, a later year of treatment was the strongest clinical predictor of late rectal toxicity in multivariate modelling (P = 0.03). Conclusion: For men with PC managed to doses of 74–78 Gy with IGRT, IMRT leads to reduced rectal toxicity compared with 3D-CRT. Incremental improvements in treatment delivery over time also appear to have an independently strong beneficial effect.