We present a theoretical and experimental analysis of the polarization dependence of a two-mode-fiber acoustooptic device, taking into account the residual stress in the fiber and the ellipticity of the core. We successfully demonstrate the suppression of the polarization dependence by relieving the residual stress, using thermal annealing.