The dispersion of the stress-optic response of a series of barium and lead phosphate glasses is measured using an ellipsometric method. It is found that the barium glasses display consistently positive dispersion with decreasing wavelength, while the lead glasses display consistently negative dispersion. It is found that the dispersion is roughly additive so that in mixed barium/lead phosphates, a composition may be found for which the dispersion is essentially zero across the optical range. The dispersion itself is shown to correlate with glass structure, giving an empirical relationship between stress-optic dispersion and composition. These results thus suggest a design rule for controlling stress-optic response in glass as a function of wavelength. ; NSERC Discovery Grant