In recent years, a remarkable increase in olive oil consumption has occurred worldwide, favoured by its organoleptic properties and the growing awareness of its health benefits. Currently, olive oil production represents an important economic income for Mediterranean countries, where roughly 98% of the world production is located. Both the cultivation of olive trees and the production of industrial and table olive oil generate huge amounts of solid wastes and dark liquid effluents, including olive leaves and pomace and olive oil mill wastewaters. Besides representing an economic problem for producers, these by-products also pose serious environmental concerns, thus their partial reuse, like that of all agronomical production residues, represents a goal to pursue. This aspect is particularly important since the cited by-products are rich in bioactive compounds, which, once extracted, may represent ingredients with remarkable added value for food, cosmetic and nutraceutical industries. Indeed, they contain considerable amounts of valuable organic acids, carbohydrates, proteins, fibers, and above all, phenolic compounds, that are variably distributed among the different wastes, depending on the employed production process of olive oils and table olives and agronomical practices. Yet, extraction and recovery of bioactive components from selected by-products constitute a critical issue for their rational valorization and detailed identification and quantification are mandatory. The most used analytical methods adopted to identify and quantify bioactive compounds in olive oil by-products are based on the coupling between gas- (GC) or liquid chromatography (LC) and mass spectrometry (MS), with MS being the most useful and successful detection tool for providing structural information. Without derivatization, LC-MS with electrospray (ESI) or atmospheric pressure chemical (APCI) ionization sources has become one of the most relevant and versatile instrumental platforms for identifying phenolic bioactive compounds. In this review, the major LC-MS accomplishments reported in the literature over the last two decades to investigate olive oil processing by-products, specifically olive leaves and pomace and olive oil mill wastewaters, are described, focusing on phenolics and related compounds.