We introduce a new X-ray speckle-vector tracking method for phase imaging, which is based on the wavelet transform. Theoretical and experimental results show that this method, which is called wavelet-transform-based speckle-vector tracking (WSVT), has stronger noise robustness and higher efficiency compared with the cross-correlation-based method. In addition, the WSVT method has the controllable noise reduction and can be applied with fewer scan steps. These unique features make the WSVT method suitable for measurements of large image sizes and phase shifts, possibly under low-flux conditions, and has the potential to broaden the applications of speckle tracking to new areas requiring faster phase imaging and real-time wavefront sensing, diagnostics, and characterization.