Virgin olive oil (VOO) is the main source of lipids in the Mediterranean diet and one of the main contributors to its proven protection against diseases associated with chronic inflammation states. This oil is rich in antioxidant compounds such as tocopherols, which together constitute the vitamin E stock of the oil. The purpose of the present work was to conduct a study on the diversity of the contents of vitamin E in the olive species (Olea europaea L.), and to know how the season climatic conditions and the degree of fruit ripening stage influences the final content of this vitamin in VOO. Data showed that the content of vitamin E in VOO is highly dependent on the olive cultivar, displaying a wide variability (89–1410 mg tocopherol/kg oil) in the olive species, and that is also dependent, to a lesser extent, on the crop year climate and the stage of fruit ripening. In addition, the suitability of cultivar crosses for breeding programs to obtain new cultivars with improved vitamin E content in VOO has been assessed. Our findings demonstrated that a single cross of olive cultivars may provide sufficient variability to be used in the selection of new cultivars.