A new nonparametric method of estimating the Bayes risk using an unclassified test sample set as well as a classified design sample set is introduced. The classified design set is used to obtain nonparametric estimates of the conditional Bayes risk of classification at each point of the unclassified test set. The average of these risk estimates is the error estimate. For large numbers of design samples the new error estimate has less variance than does an error-count estimate for classified test samples using the optimum Bayes classifier. The first application of the nonparametric method usesk-nearest neighbor (k-NN) estimates of the posterior probabilities to form the risk estimate. A large-sample analysis is made of this estimate. The expected value of this estimate is shown to be a lower bound on the Bayes error. A simple modification provides unbiased estimates of thek-NN classification error, thus providing an upper bound on the Bayes error. The second application of the method uses Parzen approximation of the density functions to obtain estimates of the risk and subsequently the Bayes error. Results of experiments on simulated data illustrate the small-sample behavior.

In an optimum pattern-recognition system the error rate is determined by the reject function. This correspondence describes how this property may be exploited to provide quantitative tests of model validity using unclassified test samples. These tests are basically goodness-of-fit tests for a function of the observations. One of these tests is shown to provide an improved estimate of error in Monte Carlo studies of complex systems. Results are given for normal distributions when parameters are estimated. In this case error estimates obtained from the empirical reject rate underestimate the actual error and performance depends on the ratio of design samples to dimension.

This paper discusses methods of estimating the probability of error for the Bayes' classifier which must be designed and tested with a finite number of classified samples. The expected difference between estimates is discussed. A simplifled algorithm to compute the leaving-one-out method is proposed for multivariate normal distributions wtih unequal co-variance matrices. The discussion is extended to nonparametric classifiers by using the Parzen approximation for the density functions. Experimental results are shown for both parametric and nonparametric cases.