In order to understand the distribution of the main secondary metabolites found in Olea europaea L., eight different samples (olive leaf, stem, seed, fruit skin and pulp, as well as virgin olive oil, olive oil obtained from stoned and dehydrated fruits and olive seed oil) coming from a Picudo cv. olive tree were analyzed. All the experimental conditions were selected so as to assure the maximum coverage of the metabolome of the samples under study within a single run. The use of LC and GC with high resolution MS (through different ionization sources, ESI and APCI) and the annotation strategies within MetaboScape 3.0 software allowed the identification of around 150 compounds in the profiles, showing great complementarity between the evaluated methodologies. The identified metabolites belonged to different chemical classes: triterpenic acids and dialcohols, tocopherols, sterols, free fatty acids, and several sub-types of phenolic compounds. The suitability of each platform and polarity (negative and positive) to determine each family of metabolites was evaluated in-depth, finding, for instance, that LC-ESI-MS (+) was the most efficient choice to ionize phenolic acids, secoiridoids, flavonoids and lignans and LC-APCI-MS was very appropriate for pentacyclic triterpenic acids (MS (−)) and sterols and tocopherols (MS (+)). Afterwards, a semi-quantitative comparison of the selected matrices was carried out, establishing their typical features (e.g., fruit skin was pointed out as the matrix with the highest relative amounts of phenolic acids, triterpenic compounds and hydroxylated fatty acids, and seed oil was distinctive for its high relative levels of acetoxypinoresinol and tocopherols).