According to Bell's theorem, no theory based on the joint assumption of realism and locality can reproduce certain predictions of quantum mechanics. Another class of realistic models, proposed by Leggett, that demands realism but abandons reliance on locality, is predicted to be in conflict with quantum mechanics. In this paper, we report on an experimental test of a contextual realistic model analogous to the model of Leggett performed with matter waves, more precisely with neutrons. Correlation measurements of the spin-energy entangled single-particle system show violation of a Leggett-type inequality by more than 7.6 standard deviations. Our experimental data falsify the contextual realistic model and are fully in favor of quantum mechanics.