We show that MgO:HxLi1−xNbO3 single crystalline solid solutions prepared by proton exchange exhibit very complex structural chemistry which is different from that for undoped congruent material. A correlation between the crystal structure and the refractive index profiles has been experimentally determined that allows us to explain specific peculiarities of the optical properties observed, as well as to predict the characteristics of a variety of waveguides. We have established that the well-known annealing proton exchange process leads to crystal structure disorder in MgO:LiNbO3 within the surface region of waveguides fabricated this way, but the soft proton exchange process preserves the crystal structure of magnesium-doped material. © 2003 American Institute of Physics.