We demonstrate the potential of birefringence-based, all-optical, ultrafast conversion between the transverse modes in integrated optical waveguides by modelling the conversion process by numerically solving the multi-mode coupled nonlinear Schroedinger equations. The observed conversion is induced by a control beam and due to the Kerr effect, resulting in a transient index grating which coherently scatters probe light from one transverse waveguide mode into another. We introduce birefringent phase matching to enable efficient all-optically induced mode conversion at different wavelengths of the control and probe beam. It is shown that tailoring the waveguide geometry can be exploited to explicitly minimize intermodal group delay as well as to maximize the nonlinear coefficient, under the constraint of a phase matching condition. The waveguide geometries investigated here, allow for mode conversion with over two orders of magnitude reduced control pulse energy compared to previous schemes and thereby promise nonlinear mode switching exceeding efficiencies of 90% at switching energies below 1 nJ.