We study the dynamics of a driven spherical colloidal particle moving in a fluid with a broken rotational symmetry. Using a nematic liquid crystal as a model, we demonstrate that when the applied force is not aligned along or perpendicular to the orientational order, the colloidal velocity does not align with the force, but forms an angle with respect to the pulling direction. This leads to blue an anisotropic hydrodynamic drag tensor which depends on the material parameters. In the case of nematic liquid crystal, we give an analytical expression and discuss the resulting implications for active microrheology experiments on fluids with broken rotational symmetry.