The olive family, Oleaceae, is a group of woody plants comprising 28 genera and ca. 700 species, distributed on all continents (except Antarctica) in both temperate and tropical environments. It includes several genera of major economic and ecological importance such as olives, ash trees, jasmines, forsythias, osmanthuses, privets and lilacs. The natural history of the group is not completely understood yet, but its diversification seems to be associated with polyploidisation events and the evolution of various reproductive and dispersal strategies. In addition, some taxonomical issues still need to be resolved, particularly in the paleopolyploid tribe Oleeae. Reconstructing a robust phylogenetic hypothesis is thus an important step toward a better comprehension of Oleaceae’s diversity. Here, we reconstructed phylogenies of the olive family using 80 plastid coding sequences, 37 mitochondrial genes, the complete nuclear ribosomal cluster and a small multigene family encoding phytochromes (phyB and phyE) of 61 representative species. Tribes and subtribes were strongly supported by all phylogenetic reconstructions, while a few Oleeae genera are still polyphyletic (Chionanthus, Olea, Osmanthus, Nestegis) or paraphyletic (Schrebera, Syringa). Some phylogenetic relationships among tribes remain poorly resolved with conflicts between topologies reconstructed from different genomic regions. The use of nuclear data remains an important challenge especially in a group with ploidy changes (both paleo- and neo-polyploids). This work provides new genomic datasets that will assist the study of the biogeography and taxonomy of the whole Oleaceae.