The objective of the current work was to fabricate, optimize and assess olive oil/phytosomal nanocarriers to improve quercetin skin delivery. Olive oil/phytosomal nanocarriers, prepared by a solvent evaporation/anti-solvent precipitation technique, were optimized using a Box–Behnken design, and the optimized formulation was appraised for in vitro physicochemical characteristics and stability. The optimized formulation was assessed for skin permeation and histological alterations. The optimized formulation (with an olive oil/PC ratio of 0.166, a QC/PC ratio of 1.95 and a surfactant concentration of 1.6%), and with a particle diameter of 206.7 nm, a zeta potential of −26.3 and an encapsulation efficiency of 85.3%, was selected using a Box–Behnken design. The optimized formulation showed better stability at ambient temperature when compared to refrigerating temperature (4 °C). The optimized formulation showed significantly higher skin permeation of quercetin when compared to an olive-oil/surfactant-free formulation and the control (~1.3-fold and 1.9-fold, respectively). It also showed alteration to skin barriers without remarkable toxicity aspects. Conclusively, this study demonstrated the use of olive oil/phytosomal nanocarriers as potential carriers for quercetin—a natural bioactive agent—to improve its skin delivery.