This paper develops a fractional-order sliding mode based extremum seeking controller (FO SM-ESC) for the optimization of nonlinear systems. The proposed FO SM-ESC, involving an FO derivative function is shown to achieve faster tracking and higher control accuracy than the integer-order (IO) SM-ESC. The tradeoff between control performance and parameters selection is analyzed and visualized. The comparison between the FO/IO SM-ESC is given to reveal the potential advantages of the FO controller over the IO controller. Simulation and experimental results show that the FO SM-ESC can have a faster convergence speed and a smaller neighborhood around the optimal operational point.