The Parzen density estimate is known to be an effective tool for estimating the Bayes error, given a set of training samples from the class distributions. An algorithm is developed to select a given number of representative samples whose Parzen density estimate closely matches that of the entire sample set. Using this reduced representative set, a piecewise quadratic classifier which provides nearly optimal performance is designed.>

The effect of finite sample-size on parameter estimates and their subsequent use in a family of functions are discussed. General and parameter-specific expressions for the expected bias and variance of the functions are derived. These expressions are then applied to the Bhattacharyya distance and the analysis of the linear and quadratic classifiers, providing insight into the relationship between the number of features and the number of training samples. Because of the functional form of the expressions, an empirical approach is presented to enable asymptotic performance to be accurately estimated using a very small number of samples. Results were experimentally verified using artificial data in controlled cases and using real, high-dimensional data.>

An expression for expected classifier performance previously derived by the authors (ibid., vol.11, no.8, p.873-855, Aug. 1989) is applied to a variety of error estimation methods and a unified and comprehensive approach to the analysis of classifier performance is presented. After the error expression is introduced, it is applied to three cases: (1) a given classifier and a finite test set; (2) given test distributions a finite design set; and (3) finite and independent design and test sets. For all cases, the expected values and variances of the classifier errors are presented. Although the study of Case 1 does not produce any new results, it is important to confirm that the proposed approach produces the known results, and also to show how these results are modified when the design set becomes finite, as in Cases 2 and 3. The error expression is used to compute the bias between the leave-one-out and resubstitution errors for quadratic classifiers. The effect of outliers in design samples on the classification error is discussed. Finally, the theoretical analysis of the bootstrap method is presented for quadratic classifiers.>