Abstract Magnetite (Fe₃O₄) nanoparticles are widely used in microwave components, such as microwave absorbers and anti-reflection coatings. Their magnetic properties are important for these applications. In this work, the weak magnetic field created by the Fe₃O₄ nanoparticles was estimated using a coherent-spectroscopy all-optical method for measurement of weak magnetic fields. Both the magnitude and the direction of the magnetic field were evaluated through the position and amplitude of a magneto-optical resonance obtained in a paraffin-coated sensor cell containing Rb vapors. A pump-probe scheme was used to prepare a high-contrast magneto-optical resonance. Different possible geometries related to this concept are discussed, including sensor miniaturization.