Phenolic and triterpenoid compounds of the olive tree are recognized as having a key role in health promotion, thanks to their multiple protective actions in humans. To expand the source of these bioactive compounds, the phenolic and triterpenoid profiles of leaf, branch, destoned fruit, destoned pomace, shell, seed, and extra virgin olive oil from the Frantoio, Leccino, and Moraiolo olive cultivars were simultaneously characterized by HPLC-DAD-MS. Overall, 43 molecules were quantitated and expressed on the obtained dry extracts. Oleuropein was mainly concentrated in branches (82.72 g/kg), fruits (55.79 g/kg), leaves (36.71 g/kg), and shells (1.26 g/kg), verbascoside (4.88 g/kg) in pomace, and nüzhenide 11-methyl oleoside (90.91 g/kg) in seeds. Among triterpenoids, which were absent in shells, the highest amount of oleanolic acid was found in olive leaves (11.88 g/kg). HCT-116 colorectal cells were chosen to assess the cytotoxicity of the dry extract, using the phytocomplex from Frantoio, which was the richest in phenols and triterpenoids. The IC50 was also determined for 13 pure molecules (phenols and terpenoids) detected in the extracts. The greatest inhibition on the cell’s proliferation was induced by the branch dry extract (IC50 88.25 μg/mL) and by ursolic acid (IC50 24 μM). A dose-dependent relationship was observed for the tested extracts.