This paper considers binary classification. We assess a classifier in terms of the Area Under the ROC Curve (AUC). We estimate three important parameters, the conditional AUC (conditional on a particular training set) and the mean and variance of this AUC. We derive, as well, a closed form expression of the variance of the estimator of the AUC. This expression exhibits several components of variance that facilitate an understanding for the sources of uncertainty of that estimate. In addition, we estimate this variance, i.e., the variance of the conditional AUC estimator. Our approach is nonparametric and based on general methods from U-statistics; it addresses the case where the data distribution is neither known nor modeled and where there are only two available data sets, the training and testing sets. Finally, we illustrate some simulation results for these estimators.

The conventional wisdom in the field of statistical pattern recognition (SPR) is that the size of the finite test sample dominates the variance in the assessment of the performance of a classical or neural classifier. The present work shows that this result has only narrow applicability. In particular, when competing algorithms are compared, the finite training sample more commonly dominates this uncertainty. This general problem in SPR is analyzed using a formal structure recently developed for multivariate random-effects receiver operating characteristic (ROC) analysis. Monte Carlo trials within the general model are used to explore the detailed statistical structure of several representative problems in the subfield of computer-aided diagnosis in medicine. The scaling laws between variance of accuracy measures and number of training samples and number of test samples are investigated and found to be comparable to those discussed in the classic text of Fukunaga, but important interaction terms have been neglected by previous authors. Finally, the importance of the contribution of finite trainers to the uncertainties argues for some form of bootstrap analysis to sample that uncertainty. The leading contemporary candidate is an extension of the 0.632 bootstrap and associated error analysis, as opposed to the more commonly used cross-validation.

The covariance matrices associated with each state of health or disease from a previous study are used as the basis of an image staining display technique for aid in quantitative differential diagnosis. A state of health or disease is chosen by the clinician: this selects the covariance matrix from the data base. A region of interest (ROI) is then scrolled through an abdominal B-scan. For each position of the ROI a point in the four-dimensional feature space is calculated. A natural measure of the distance of this point from the center of mass (multivariate mean) of the disease class is calculated in terms of the covariance matrix of this class; this measure is the Mahalanobis distance. The confidence level for acceptance or rejection of the hypothesized disease class is obtained from the probability distribution of this distance, the T(2) probability law. This confidence level is color coded and used as a color stain that overlays the original scan at that position. The variability of the calculated features is studied as a function of ROI size, or the spatial resolution of the color coded image, and it is found that for an ROI in the neighborhood of 4 cm(2) most of the variability due to the finite number of independent samples (speckles) is averaged out, leaving the "noise floor" associated with inter- and intra-patient variability. ROIs on the order of 1 cm(2) may result with technical advances in B-scan resolution. A small number of points on organ boundaries are entered by the user, to fit with arcs of ellipses to be used to switch between organ (liver and kidney) data bases as the ROI encounters the boundary. By selecting in turn various state-of-health or state-of-disease databases, such images of confidence levels may be used for quantitative differential diagnosis. The method is not limited to ultrasound, being applicable in principle to features obtained from any modality or multimodality combination.

An autoregressive (AR) method for spectral estimation was applied toward the task of estimating ultrasonic backscatter coefficients from small volumes of tissue. High spatial resolution is desirable for generating images of backscatter coefficient. Data was acquired from a homogeneous tissue-mimicking phantom and from a normal human liver in vivo. The AR method was much more resistant to gating artifacts than the traditional DFT (discrete Fourier transform) approach. The DFT method consistently underestimated backscatter coefficients at small gate lengths. Therefore backscatter coefficient image formation will be quantitatively more meaningful if based on AR spectral estimation rather than the DFT. The autoregressive method offers promise for enhanced spatial resolution and accuracy in ultrasonic tissue characterization and nondestructive evaluation of materials.

An approach for the application of multivariate pattern recognition techniques to detection of diffuse and focal disease using acoustic data is reviewed. Supervised and unsupervised techniques are implemented to design the best ultrasonic tissue signature for a given task from a set of measurements. The performances of both techniques are evaluated and compared using several methods. However, it is desirable to utilize a technique that quantitatively detects and displays the heterogeneity of an ultrasound image. It is shown that, for a particular task, choosing features with physical significance will make the classification of the data more robust. It is also shown that the success of combining supervised and unsupervised techniques using such features extends beyond discrimination of one class of data from the other and that the approach can be used to grade and the variations in the same tissue type.>