The biosynthesis of the phenolic fraction of olive fruits during ripening and the transformations occurring in this moiety during virgin olive oil (VOO) extraction are discussed in this paper. The influence of agronomical factors that can significantly affect the phenolic profile of VOO is also discussed. Particularly, it is worth emphasizing the role of genetic factors, cultivation and climatic conditions such as water availability, atmospheric temperature, altitude, health status of the fruits, alternate bearing in the olive, and some processing factors such as crushing, malaxation time and temperature or volume of water added during milling. Among these parameters, special attention has been paid to genetic factors due to the high variability observed among Olea europaea genotypes for all recorded traits. In this context, interesting experimental results have been obtained with cultivated and wild olive trees, and also with segregating populations resulting from olive breeding programs. To the authors' knowledge, reviews evaluating the influence of the main factors that contribute to the profile of hydrophilic phenols have not been previously published. The discussion concerning olive breeding programs is a major and novel aspect to be emphasized considering recent trends to obtain new olive cultivars that confer better organoleptic properties and better quality to VOO.