The speaker recognition evaluation is conducted in a framework in which three score distributions and two decision thresholds are employed, and the statistic of interest is an average of the two weighted sums of the probabilities of type I and type II errors at the two thresholds correspondingly. And data dependence caused by multiple use of the same subjects exists ubiquitously in order to generate more samples because of limited resources. Under such circumstances, statistical analysis is carried out. First, the standard error (SE) of measure is estimated using the nonparametric three-sample two-layer bootstrap algorithm on a two-layer data structure constructed after dataset optimization due to data dependence, based upon our prior rigorous statistical research in ROC analysis on large datasets with data dependence. Second, only based on such SEs, can the one-classifier and two-classifier significance testing in statistics be carried out to provide quantitative information in terms of the significance level, i.e.,p-value, while dealing with evaluation and comparison of classifiers. In comparison, the positive correlation coefficient must be taken into account, which is computed using a synchronized resampling algorithm; otherwise, the likelihood of detecting the statistical significance of difference between the performance levels of two classifiers can be wrongly reduced.

BACKGROUND: Cell image segmentation (CIS) is an essential part of quantitative imaging of biological cells. Designing a performance measure and conducting significance testing are critical for evaluating and comparing the CIS algorithms for image-based cell assays in cytometry. Many measures and methods have been proposed and implemented to evaluate segmentation methods. However, computing the standard errors (SE) of the measures and their correlation coefficient is not described, and thus the statistical significance of performance differences between CIS algorithms cannot be assessed. RESULTS: We propose the total error rate (TER), a novel performance measure for segmenting all cells in the supervised evaluation. The TER statistically aggregates all misclassification error rates (MER) by taking cell sizes as weights. The MERs are for segmenting each single cell in the population. The TER is fully supported by the pairwise comparisons of MERs using 106 manually segmented ground-truth cells with different sizes and seven CIS algorithms taken from ImageJ. Further, the SE and 95% confidence interval (CI) of TER are computed based on the SE of MER that is calculated using the bootstrap method. An algorithm for computing the correlation coefficient of TERs between two CIS algorithms is also provided. Hence, the 95% CI error bars can be used to classify CIS algorithms. The SEs of TERs and their correlation coefficient can be employed to conduct the hypothesis testing, while the CIs overlap, to determine the statistical significance of the performance differences between CIS algorithms. CONCLUSIONS: A novel measure TER of CIS is proposed. The TER's SEs and correlation coefficient are computed. Thereafter, CIS algorithms can be evaluated and compared statistically by conducting the significance testing.

The data dependency due to multiple use of the same subjects has impact on the standard error (SE) of the detection cost function (DCF) in speaker recognition evaluation. The DCF is defined as a weighted sum of the probabilities of type I and type II errors at a given threshold. A two-layer data structure is constructed: target scores are grouped into target sets based on the dependency, and likewise for non-target scores. On account of the needed equal probabilities for scores being selected when resampling, target sets must contain the same number of target scores, and so must non-target sets. In addition to the bootstrap method with i.i.d. assumption, the nonparametric two-sample one-layer and two-layer bootstrap methods are carried out based on whether the resampling takes place only on sets, or subsequently on scores within the sets. Due to the stochastic nature of the bootstrap, the distributions of the SEs of the DCF estimated using the three different bootstrap methods are created and compared. After performing hypothesis testing, it is found that data dependency increases not only the SE but also the variation of the SE, and the two-layer bootstrap is more conservative than the one-layer bootstrap. The rationale regarding the different impacts of the three bootstrap methods on the estimated SEs is investigated.

Abstract A massively parallel Genetic Algorithm (GA) has been applied to RNA sequence folding on three different computer architectures. The GA, an evolution-like algorithm that is applied to a large population of RNA structures based on a pool of helical stems derived from an RNA sequence, evolves this population in parallel. The algorithm was originally designed and developed for a 16384 processor SIMD (Single Instruction Multiple Data) MasPar MP-2. More recently it has been adapted to a 64 processor MIMD (Multiple Instruction Multiple Data) SGI ORIGIN 2000, and a 512 processor MIMD CRAY T3E. The MIMD version of the algorithm raises issues concerning RNA structure data-layout and processor communication. In addition, the effects of population variation on the predicted results are discussed. Also presented are the scaling properties of the algorithm from the perspective of the number of physical processors utilized and the number of virtual processors (RNA structures) operated upon. Contact: [email protected]; [email protected]; [email protected]; [email protected] * To whom correspondence should be addressed. 4 Current address: HPC Applications Inc., 10080 Old Frederick Road, Ellicott City, MD 21042, USA.

MOTIVATION: Using the genetic algorithm (GA) for RNA folding on a massively parallel supercomputer, MasPar MP-2 with 16,384 processors, we successfully predicted the existence of H-type pseudoknots in several sequences. RESULTS: The GA is applied to folding the tRNA-like 3' end of turnip yellow mosaic virus (TYMV) RNA sequence with 82 nucleotides, the 3' UTRs of satellite tobacco necrosis virus (STNV)-2 RNA sequence with 619 nucleotides and STNV-I RNA sequence with 622 nucleotides, and the bacteriophage T2, T4 and T6 gene 32 mRNA sequences with 946, 1340 and 946 nucleotides, respectively. The GA's results match the phylogenetically supported tertiary structures of these sequences.

An annealing mutation operator in the genetic algorithms (GA) for RNA folding on a MasPar MP-2 has been designed. The mutation probability descends along a hyperbola with respect to the size of the secondary structure, hence the total number of mutations at each generation drops linearly. Especially for long sequences with thousands of nucleotides as opposed to hundreds of nucleotides, the new mutation operator can make the distribution of free energies over all processors on MasPar MP-2 converge only after hundreds of generations. Based upon this new mutation operator, a technique to terminate the GA is also developed. The new mutation operator runs very efficiently. Some variations of the annealing mutation operator are also discussed.