The geometric phase has been proposed as a candidate for noise resilient coherent manipulation of fragile quantum systems. Since it is determined only by the path of the quantum state, the presence of noise fluctuations affects the geometric phase in a different way than the dynamical phase. We have experimentally tested the robustness of Berry's geometric phase for spin-1/2 particles in a cyclically varying magnetic field. Using trapped polarized ultra-cold neutrons it is demonstrated that the geometric phase contributions to dephasing due to adiabatic field fluctuations vanish for long evolution times. ; Comment: 4 pages, 4 figures