Statistical methods are widely used in environmental studies to evaluate natural hazards. Within groundwater vulnerability in particular, statistical methods are used to support decisions about environmental planning and management. The production of vulnerability maps obtained by statistical methods can greatly help decision making. One of the key points in all of these studies is the validation of the model outputs, which is performed through the application of various techniques to analyze the quality and reliability of the final results and to evaluate the model having the best performance. In this study, a groundwater vulnerability assessment to nitrate contamination was performed for the shallow aquifer located in the Province of Milan (Italy). The Weights of Evidence modeling technique was used to generate six model outputs, each one with a different number of input predictive factors. Considering that a vulnerability map is meaningful and useful only if it represents the study area through a limited number of classes with different degrees of vulnerability, the spatial agreement of different reclassified maps has been evaluated through the kappa statistics and a series of validation procedures has been proposed and applied to evaluate the reliability of the reclassified maps. Results show that performance is not directly related to the number of input predictor factors and that is possible to identify, among apparently similar maps, those best representing groundwater vulnerability in the study area. Thus, vulnerability maps generated using statistical modeling techniques have to be carefully handled before they are disseminated. Indeed, the results may appear to be excellent and final maps may perform quite well when, in fact, the depicted spatial distribution of vulnerability is greatly different from the actual one. For this reason, it is necessary to carefully evaluate the obtained results using multiple statistical techniques that are capable of providing quantitative insight into the analysis of the results. This evaluation should be done at least to reduce the questionability of the results and so to limit the number of potential choices.