Spinel Li$_x$Mn$_2$O$_4$ is a key cathode material that is used extensively in commercial Li-ion batteries. A challenge with this material has been that the capacity of the battery fades with cycling, an effect that can be traced to the presence of an anti-ferromagnetic insulator phase in the fully lithiated LiMn$_2$O$_4$ (LMO) and the associated charge disproportionation that drives distortions of the MnO$_6$ octahedra. Here, by combining x-ray magnetic Compton scattering experiments with parallel first-principles computations, we show that the anti-ferromagnetic phase of LMO is surrounded by a robust ferrimagnetic metallic phase, which becomes stable when even a small amount of Li is removed from or added to the charge-ordered LMO. In this surprising ferrimagnetic state, charge-ordering and octahedral distortions are found to be strongly suppressed. We identify the nature of the ferrimagnetic orbitals involved through theoretical and experimental analyses of the magnetic Compton scattering spectra.