Abstract Background This study aimed to evaluate a new computational histology prediction system based on colorectal polyp textural surface patterns using high definition white light images. Methods Textural elements (textons) were characterized according to their contrast with respect to the surface, shape, and number of bifurcations, assuming that dysplastic polyps are associated with highly contrasted, large tubular patterns with some degree of bifurcation. Computer-aided diagnosis (CAD) was compared with pathological diagnosis and the diagnosis made by endoscopists using Kudo and Narrow-Band Imaging International Colorectal Endoscopic classifications. Results Images of 225 polyps were evaluated (142 dysplastic and 83 nondysplastic). The CAD system correctly classified 205 polyps (91.1 %): 131/142 dysplastic (92.3 %) and 74/83 (89.2 %) nondysplastic. For the subgroup of 100 diminutive polyps (≤ 5 mm), CAD correctly classified 87 polyps (87.0 %): 43/50 (86.0 %) dysplastic and 44/50 (88.0 %) nondysplastic. There were no statistically significant differences in polyp histology prediction between the CAD system and endoscopist assessment. Conclusion A computer vision system based on the characterization of the polyp surface in white light accurately predicted colorectal polyp histology.