Exchange bias, referred to the interaction between a ferromagnet FM and an antiferromagnet AFM, is a fundamental interfacial magnetic phenomenon, which is key to current and future applications. The effect was discovered half a century ago, and it is well established that the spin structures at the FM/AFM interface play an essential role. However, currently, ad hoc phenomenological anisotropies are often postulated without microscopic justification or sufficient experimental evidence to address magnetization-reversal behavior in exchange-bias systems. We advance toward a detailed microscopic understanding of the magnetic anisotropies in exchange-bias FM/AFM systems by showing that symmetry-breaking anisotropies leave a distinct finger-print in the asymmetry of the magnetization reversal and we demonstrate how these emerging anisotropies are correlated with the intrinsic anisotropy. Angular and vectorial resolved Kerr hysteresis loops from FM/AFM bilayers with varying degree of ferromagnetic anisotropy reveal a noncollinear anisotropy, which becomes important for ferromagnets with vanishing intrinsic anisotropy. Numerical simulations show that this aniso-tropy naturally arises from the inevitable spin frustration at an atomically rough FM/AFM interface. As a consequence, we show in detail how the differences observed for different materials during magnetization reversal can be understood in general terms as originating from the interplay between interfacial frustration and intrinsic anisotropies. This understanding will certainly open additional avenues to tailor future advanced magnetic materials.