Context: Nearly all members of the Vesta family cross the orbits of (4) Vesta, one of the most massive asteroids in the main belt, and some of them approach it closely. When mutual velocities during such close encounters are low, the trajectory of the small body can be gravitationally deflected, consequently changing its heliocentric orbital elements. While the effect of a single close encounter may be small, repeated close encounters may significantly change the proper element distribution of members of asteroid families. Aims: We develop a model of the long-term effect of close encounters with massive asteroids, so as to be able to predict how far former members of the Vesta family could have drifted away from the family. Methods: We first developed a new symplectic integrator that simulates both the effects of close encounters and the Yarkovsky effect. We analyzed the results of a simulation involving a fictitious Vesta family, and propagated the asteroid proper element distribution using the probability density function (pdf hereafter), i.e. the function that describes the probability of having an encounter that modifies a proper element x by Deltax, for all the possible values of Deltax. Given any asteroids' proper element distribution at time t, the distribution at time t+T may be predicted if the pdf is known (Bachelier 1900, Théorie de la spéculation; Hughes 1995, Random Walks and Random Environments, Vol. I). Results: We applied our new method to the problem of V-type asteroids outside the Vesta family (i.e., the 31 currently known asteroids in the inner asteroid belt that have the same spectral type of members as the Vesta family, but that are outside the limits of the dynamical family) and determined that at least ten objects have a significant diffusion probability over the minimum estimated age of the Vesta family of 1.2 Gyr (Carruba et al. 2005, A&A, 441, 819). These objects can therefore be explained in the framework of diffusion via repeated close encounters with (4) Vesta of asteroids originally closer to the parent body. Conclusions: We computed diffusion probabilities at the location of four of these asteroids for various initial conditions, parametrized by values of initial ejection velocity V_ej. Based on our results, we believe the Vesta family age is (1200 ± 700) Myr old, with an initial ejection velocity of (240 ± 60) m/s. Appendices are only available in electronic form at http://www.aanda.org