We investigate the magnetic properties of three Mn-6 single-molecule magnets by means of inelastic neutron scattering and frequency domain magnetic resonance spectroscopy. The experimental data reveal that small structural distortions of the molecular geometry produce a significant effect on the energy-level diagram and therefore on the magnetic properties of the molecule. We show that the giant spin model completely fails to describe the spin-level structure of the ground spin multiplets. We analyze theoretically the spin Hamiltonian for the low-spin Mn-6 molecule (S=4) and we show that the excited S multiplets play a key role in determining the effective energy barrier for the magnetization reversal, in analogy to what was previously found for the two high spin Mn-6 (S=12) molecules [S. Carretta et al., Phys. Rev. Lett. 100, 157203 (2008)].