The antitumor effects of Coix lacryma-jobi L. var. ma-yuen Stapf. (adlay seed) ethanolic extract have been increasingly shown. This study aimed to investigate the beneficial effects of both the fractions and subfractions of adlay seed ethanolic extract on the human breast (MCF-7) and cervical (HeLa) cancer cell lines, as well as exploring their possible mechanisms of action. The ethanolic extracts were obtained from different parts of adlay seed, including AHE (adlay hull extract), ATE (adlay testa extract), ABE (adlay bran extract) and PAE (polished adlay extract). The results of a 3-(4,5-dimethyl thiazol-2-yl)-2,5-diphenyl- tetrazolium bromide (MTT) assay showed that AHE-Ea and ATE-Ea showed significant growth inhibitory effects in a dose-dependent manner. The results also showed that the AHE-Ea-K, AHE-Ea-L, ATE-Ea-E and ATE-Ea-F subfractions inhibited cell proliferation, induced cell cycle arrest in the G0/G1 phase and decreased CDK4/Cyclin D1 protein expression. Finally, the extract activated caspase-3 activity and PARP protein expression, which induced MCF-7 and HeLa cell apoptosis. We then used liquid chromatography–mass spectrometry (LC/MS) to identify the potential active components., Quercetin showed an anticancer capacity. In conclusion, the AHE-Ea-K, AHE-Ea-L, ATE-Ea-E and ATE-Ea-F subfractions showed antitumor effects through the inhibition of MCF-7 and HeLa cell line viability, as well as inducing apoptosis and cell cycle arrest.