In this Letter, we report on a crucial experiment showing that magnetic impurities reduce the ferromagnetic order temperature in thiol-capped Au glyconanoparticles (GNPs). The spontaneous magnetization of AuFe GNPs exhibits a fast decrease with temperature that contrasts with the almost constant value of the magnetization observed in Au NPs. Moreover, hysteresis disappears below 300 K. Both features indicate that Fe impurities reduce the high local anisotropy field responsible for the ferromagnetic behavior in Au GNPs. As a consequence, the amazing ferromagnetism in Au NPs should not be associated with the presence of magnetic impurities.