We investigate the electronic structure and properties of Rydberg atoms exposed to a magnetic quadrupole field. It is shown that the spatial as well as generalized time reversal symmetries lead to a two-fold degeneracy of the electronic states in the presence of the external field. A delicate interplay between the Coulomb and magnetic interactions in the inhomogeneous field leads to an unusual weak field splitting of the energy levels as well as complex spatial patterns of the corresponding spin polarization density of individual Rydberg states. Remarkably the magnetic quadrupole field induces a permanent electric dipole moment of the atom. ; Comment: 8 pages, 5 figures, accepted for publication in EPL