Progress in the emerging field of engineered quantum systems requires the development of devices that can act as quantum memories. The realisation of such devices by doping solid state cavities with paramagnetic ions imposes a trade-off between ion concentration and cavity coherence time. Here, we investigate an alternative approach involving interactions between photons and naturally occurring impurity ions in ultra-pure crystalline microwave cavities exhibiting exceptionally high quality factors. We implement a hybrid Whispering Gallery/Electron Spin Resonance method to perform rigorous spectroscopy of an undoped single-crystal sapphire resonator over the frequency range 8--19 GHz, and at external applied DC magnetic fields up to 0.9 T. Measurements of a high purity sapphire cooled close to 100 mK reveal the presence of Fe$^{3+}$, Cr$^{3+}$, and V$^{2+}$ impurities. A host of electron transitions are measured and identified, including the two-photon classically forbidden quadrupole transition ($Δ m_s =2$) for Fe$^{3+}$, as well as hyperfine transitions of V$^{2+}$.