We developed a phase-field model to study the stress-driven grain boundary migration in elastically inhomogeneous polycrystalline materials with arbitrary elastic inhomogeneity and anisotropy. The dependence of elastic stiffness tensor on grain orientation is taken into account, and the elastic equilibrium equation is solved using the Fourier spectral iterative-perturbation method. We studied the migration of planar and curved grain boundaries under an applied stress. The relation between grain boundary migration velocity and driving force is found to be linear in the steady-state regime. Our study shows that the stress distribution depends on the relative misorientation between the grains and the nature of the applied load. As a consequence, the mechanism of grain boundary migration is different when the load is applied parallel or perpendicular to a grain boundary. The bulk mechanical driving force for grain boundary migration is provided by the difference in the level of stress in the adjoining grains which arise due to difference in elastic moduli. We further show that under certain conditions an applied stress may act as a precursor to abnormal grain growth.