We observe a magnetic-field-induced enhancement of the nonlocal spin signal in Ni80Fe20/Ag lateral spin valves. The enhancement depends on the bias current polarity but not on the field direction. We present a theoretical model that explains our experimental results, taking into account the electron-spin relaxation of magnetic impurities. We find that the relaxation is about an order of magnitude weaker than Elliott-Yafet relaxation.