Background While surgery remains the cornerstone of rectal cancer treatment, organ-preservation is upcoming. Therefore, neo-adjuvant treatment should be optimized. By escalating doses, response can be increased. To limit toxicity of boost, accurate gross tumor volume (GTV) definition is required. MRI, especially undeformed fast spin echo diffusion-weighted MRI (DWI), looks promising for delineation. However, inconsistencies between observers should be quantified before clinical implementation. We aim to find which MRI sequence (T2w, DWI or combination) is optimal and clinically useful for GTV definition by evaluating inter-observer agreement. Methods Locally advanced rectal cancer patients (tumors 2). Three independent observers delineated T2w, DWI and combination (Combi) after training-set. Volumes, conformity index (CI), and maximum Hausdorff distance (HD) were calculated between any observer-pair per patient per modality. Results Twenty-four consecutive patients were included. One patient had cT2 (4.2%), 19 cT3 (79.1%) and 4 cT4 (16.7%), with 2 clinical node negative (8.3%), 4 cN1 (16.7%), and 18 cN2 (75.0%) on MRI. From 24 patients, 70 sequences were available (24x T2, 23x DWI, and 23x Combi). Between observers, no significant volume differences were observed per modality. T2 showed significantly largest volumes compared to DWI (mean difference 19.85 ml, SD 17.42, p 0.61). Average HD was largest on T2 (18.60 mm, max 31.40 mm, min 9.20 mm). Discussion Delineation on DWI resulted in delineation of the smallest volumes with similar consistency and mean distances, but with slightly lower Hausdorff distances compared to T2 and Combi. However, with lack of a gold standard it remains difficult to establish if delineations also represent true tumor. Study strengths were DWI adaptation to exclude geometrical distortions and training-set. DWI shows great potential for delineation purposes as long as sufficient experience exists and geometrical distortions are eliminated.