Cross-Validation for Binary Classification by Real-Valued Functions: Theoretical Analysis

Journal article by Martin Anthony, Sean B. Holden

Full text: Unavailable

Publisher: Association for Computing Machinery (ACM)

Preprint: archiving allowed. Upload

Postprint: archiving allowed. Upload

Published version: archiving forbidden. Upload

Policy details (opens in a new window). Data provided by SHERPA/RoMEO
This paper concerns the use of real-valued functions for binary classification problems. Previous work in this area has concentrated on using as an error estimate the `resubstitution' error (that is, the empirical error of a classifier on the training sample) or its derivatives. However, in practice, cross-validation and related techniques are more popular. Here, we analyse theoretically the accuracy of the holdout and cross-validation estimators for the case where real-valued functions are used as classifiers. We then introduce two new error estimation techniques, which we call the adaptive holdout estimate and the adaptive cross-validation estimate, and we perform a similar analysis for these. Finally, we show how our results can be applied to certain types of neural network.