The global burden of cancer continues to increase largely with the aging and growth of the world population. The purpose of the present work was to find new anticancer molecules from a natural source. We utilized chromatographic methods to isolate compounds from medicinal plant Dryopteris fragrans (L.) Schott. The structure of the new compounds was determined by spectroscopic and spectrometric data (1D NMR, 2D NMR, and EMI-MS). Their anti-proliferation effects against five human cancer cell lines including A549, MCF7, HepG2, HeLa, and PC-3 were evaluated by CCK-8 andlactate dehydrogenase (LDH) assay. A new sesquiterpene, (7S, 10S)-2,3-dihydroxy-calamenene-15-carboxylic acid methyl ester (1), and two known compounds (2 and 3) were isolated. The new sesquiterpene was named dryofraterpene A and significantly inhibited cancer cell proliferation without any obvious necrosis below a 10 μM concentration. In conclusion, a novel anticancer sesquiterpene together with two known compounds was isolated, which might be a promising lead compound for the treatment of cancer.