This work presents a deep-learning-based denoising technique to accelerate the acquisition of high b-value diffusion-weighted MRI for rectal cancer. A denoising convolutional neural network (DCNN) with a combined L1–L2 loss function was developed to denoise high b-value diffusion-weighted MRI data acquired with fewer repetitions (NEX: number of excitations) using the low b-value image as an anatomical guide. DCNN was trained using 85 datasets acquired on patients with rectal cancer and tested on 20 different datasets with NEX = 1, 2, and 4, corresponding to acceleration factors of 16, 8, and 4, respectively. Image quality was assessed qualitatively by expert body radiologists. Reader 1 scored similar overall image quality between denoised images with NEX = 1 and NEX = 2, which were slightly lower than the reference. Reader 2 scored similar quality between NEX = 1 and the reference, while better quality for NEX = 2. Denoised images with fourfold acceleration (NEX = 4) received even higher scores than the reference, which is due in part to the effect of gas-related motion in the rectum, which affects longer acquisitions. The proposed deep learning denoising technique can enable eightfold acceleration with similar image quality (average image quality = 2.8 ± 0.5) and fourfold acceleration with higher image quality (3.0 ± 0.6) than the clinical standard (2.5 ± 0.8) for improved diagnosis of rectal cancer.