Magneto photonic crystals exhibit unique combination of magneto-optical nonreciprocity and resonant behavior originating from periodic structure. Modeling of magneto photonic structures requires inclusion of magnetic induced anisotropy and description of medium using a permittivity tensor. On the other hand, differential theory of periodic gratings based on Fourier expansion of permittivity tensor and electromagnetic field is recently frequently applied. In this paper we propose an approach of effective propagation constant calculation from scattering matrix of the waveguide consisting of the anisotropic magneto-optical lamellar grating. The structure is modeled using Rigorous Coupled Wave Analysis (RCWA) extended by Fourier factorization method. The approach is applied to optimize parameters of integrated magneto-optic waveguide isolator with lamellar grating from magneto-optic medium at transverse geometry.