Abstract Background Due to the role expansion of radiotherapists in dosimetric aspect, radiotherapists have taken up organs at risk (OARs) contouring work in many clinical settings. However, training of newly qualified radiotherapists in OARs contouring can be time consuming, it may also cause extra burden to experienced radiotherapists. As web-based open-source radiotherapy delineation software (WORDS) has become more readily available, it has provided a free and interactive alternative to conventional one-to-one coaching approach during OARs contouring training. The present study aims to evaluate the effectiveness of WORDS in training OARs contouring skills of newly qualified radiotherapists, compared to those trained by conventional one-to-one coaching approach. Methods Nine newly qualified radiotherapists (licensed in 2017 – 2018) were enrolled to the conventional one-to-one coaching group (control group), while 11 newly qualified radiotherapists (licensed in 2019 – 2021) were assigned to WORDS training group (measured group). Ten OARs were selected to be contoured in this 3-phases quantitative study. Participants were required to undergo phase 1 OARs contouring in the beginning of the training session. Afterwards, conventional one-to-one training or WORDS training session was provided to participants according to their assigned group. Then the participants did phase 2 and 3 OARs contouring which were separated 1 week apart. Phase 1 – 3 OARs contouring aimed to demonstrate participants’ pre-training OARs contouring ability, post-training OARs contouring ability and knowledge retention after one-week interval respectively using either training approach. To prevent bias, the computed tomography dataset for OARs contouring in each phase were different. Variations in the contouring scores for the selected OARs were evaluated between 3 phases using Kruskal-Wallis tests with Dunn tests for pairwise comparisons. Variations in the contouring scores between control and measured group in phase 1 – 3 contouring were analyzed using Wilcoxon signed-rank test. A p-value < 0.05 was considered to be statistically significant. Results In both control group and measured group, significant improvement (p < 0.05) in phase 2 and 3 contouring scores have been observed comparing to phase 1 contouring scores. In comparison of contouring scores between control group and measured group, no significant differences (p > 0.05) were observed in all OARs between both groups. Conclusions The results in this study have demonstrated that the outcome of OARs contouring training using WORDS is comparable to the conventional training approach. In addition, WORDS can offer flexibility to newly qualified radiotherapists to practice OARs contouring at will, as well as reduce staff training burden of experienced radiotherapists.