Brown blight, a destructive foliar disease of tea, has become a highly limiting factor for tea cultivation in Taiwan. To understand the population composition of the causal agents (Colletotrichum spp.), the fungal diversity in the main tea-growing regions all over Taiwan was surveyed from 2017 to 2019. A collection of 139 Colletotrichum isolates was obtained from 14 tea cultivars in 86 tea plantations. Phylogenic analysis using the ribosomal internal transcribed spacer, glutamine synthetase gene, Apn2-Mat1-2 intergenic spacer, β-tubulin, actin, calmodulin, and glyceraldehyde-3-phosphate dehydrogenase genes together with morphological characterization revealed three species associated with brown blight of tea; namely, Colletotrichum camelliae (95.6% of all isolates), C. fructicola (3.7%), and C. aenigma (0.7%). This is the first report of C. aenigma in Taiwan. The optimal growth temperatures were 25°C for C. camelliae and 25 and 30°C for C. fructicola and C. aenigma. Although C. fructicola and C. aenigma were more adapted to high temperature, C. camelliae was the most pathogenic across different temperatures. Regardless of whether spore suspensions or mycelial discs were used, significantly larger lesions and higher disease incidences were observed for wounded than for nonwounded inoculation and for the third and fourth leaves than for the fifth leaves. Wounded inoculation of detached third and fourth tea leaves with mycelial discs was found to be a reliable and efficient method for assessing the pathogenicity of Colletotrichum spp. within 4 days. Preventive application of fungicides or biocontrol agents immediately after tea pruning and at a young leaf stage would help control the disease.