Abstract The comprehensive influence of the amplitude and frequency of the modulated magnetic field and the magnitude of the bias magnetic field on the performance of an atomic magnetometer have been investigated. Under different magnetic fields, the combined action of the spin precession signal caused by a high-amplitude magnetic field and the influence of magnetic field on relaxation makes the time domain output signal and the amplitude of the first to fourth harmonics show different characteristics, which cannot be explained by the classical analytical calculation solution. By considering the influence of the magnetic field on the transverse relaxation, a more complete model is constructed to explain the phenomenon with a numerical solution, and the overall fit is 93.26%. Based on the single beam and magnetic field modulation scheme, a compact magnetometer is constructed for verification, with a volume of 56.7 cm³ and a sensitivity of 30 fT/Hz^1/2.