The Dendrobium species (Orchidaceae) has been cultivated as an ornamental plant as well as used in traditional medicines. In this study, the chemical profiles of Dendrobii Herba, used as herbal medicine, Dendrobium in two different species, their hybrid, and the gamma-irradiated mutant lines of the hybrid, were systematically investigated via ultra-performance liquid chromatography coupled with quadrupole time-of-flight mass spectrometry (UPLC-QToF MS). Among the numerous peaks detected, 17 peaks were unambiguously identified. Gigantol (1), (1R,2R)-1,7-hydroxy-2,8-methoxy-2,3-dihydrophenanthrene-4(1H)-one (2), tristin (3), (−)-syringaresinol (4), lusianthridin (5), 2,7-dihydroxy-phenanthrene-1,4-dione (6), densiflorol B (7), denthyrsinin (8), moscatilin (9), lusianthridin dimer (10), batatasin III (11), ephemeranthol A (12), thunalbene (13), dehydroorchinol (14), dendrobine (15), shihunine (16), and 1,5,7-trimethoxy-2-phenanthrenol (17), were detected in Dendrobii Herba, while 1, 2, and 16 were detected in D. candidum, 1, 11, and 16 in D. nobile, and 1, 2, and 16 in the hybrid, D. nobile × candidum. The methanol extract taken of them was also examined for cytotoxicity against FaDu human hypopharynx squamous carcinoma cells, where Dendrobii Herba showed the greatest cytotoxicity. In the untargeted metabolite analysis of 436 mutant lines of the hybrid, using UPLC-QToF MS and cytotoxicity measurements combined with multivariate analysis, two tentative flavonoids (M1 and M2) were evaluated as key markers among the analyzed metabolites, contributing to the distinction between active and inactive mutant lines.