Online Active Learning (OAL) is a powerful tool for classifying evolving data streams using limited annotations from a human operator who is a domain expert. The objective of the OAL learning paradigm is to minimize jointly the classification error rate and the annotation cost across the data stream by posing periodic Active Learning (AL) queries. In this paper, this objective is extended to include identification of classifier errors by the expert during the typical workflow. To this end, Corrective Feedback (CF) is introduced as a second channel of interaction between the expert and the learning algorithm, complementary to the AL channel, that allows the algorithm to obtain additional training labels without disrupting the expert's workflow. Online Active Learning with Corrective Feedback (OAL-CF) is formally defined as a paradigm, and its efficacy is proven through experimental application to two binary classification tasks, Spoken Language Verification and Voice-Type Discrimination. Finally, the effects of adding CF to the OAL paradigm are analyzed in terms of classification performance, annotation cost, trends over time, and class balance of the collected training data. Overall, the addition of CF results in a 53% relative reduction in cost compared to OAL without CF.

Online Active Learning with Corrective Feedback (OALCF) is a new machine learning paradigm that learns to detect events of interest in streaming audio by adapting to feedback from an operational user, who is a domain expert. The machine learns from each batch of the data stream by posing active learning queries to the expert and updating its model before making predictions. The expert reviews the predictions and indicates which of them are incorrect. This feedback is used to update the model for the next batch. In this paper, we introduce iterative feedback, where active learning and corrective feedback are performed multiple times per batch. We validate this approach on a large Spoken Language Verification task with 35 low-prevalence languages. The evaluation metric used (IMLM) accounts for the total cost of the method (i.e., error and feedback cost combined). The iterative algorithm achieves a 47.7% relative reduction in total cost compared to the original OAL-CF algorithm.

Pattern classification systems have traditionally been trained using a set of labeled training data and subsequently evaluated using different testing data. The cost of labeling the training data is typically substantial. Online Human-In-The-Loop (HITL) algorithms present an alternate approach that enables useful classification for many real-world applications using much less labeled data. These classifiers begin with a very small amount of training data and iteratively improve their performance by labeling a selected small number of utterances manually. Unfortunately, there is no unified evaluation metric that considers both classifier performance and annotation cost, which makes it difficult to evaluate these algorithms objectively. Furthermore, the lack of such a metric restricts the evaluation of online learning algorithms to prequential evaluation (before the classifier is adapted to the newly-labeled evaluation data), which does not realistically reflect the algorithm’s ability to adapt to the data stream in real time. This paper introduces the Interactive Machine Learning Metric (IMLM), a new unified evaluation metric that makes the combination of performance and annotation cost for binary classification tasks far less arbitrary. This metric is well suited for the evaluation of online HITL algorithms and also allows for fair comparison of different algorithms after adapting to the evaluation data. The value and appropriateness of IMLM is demonstrated by evaluating a series of Online Active Learning algorithms on a Spoken Language Verification task.

Active learning is a powerful method for reducing the amount of labeled training data needed for a machine learning model to learn a task without degrading performance. This is accomplished by iteratively selecting the most informative samples from an unlabeled dataset to be labeled by an oracle (i.e., a human annotator) using an active learning sampling strategy. Pretrained models have been used in recent years as frontends for active learning neural networks to increase efficiency. This work applies active learning with pretrained models to the spoof detection task with the following two goals: 1) the identification of which pretrained speech models and active learning strategies are most effective for the spoof detection task, and 2) the development of an active learning method that selects the optimal sampling strategy from a list of available strategies at each step of the active learning process. This mixed strategy is shown to outperform all individual strategies for the task.

The paper describes a set of new algorithms that perform speaker clustering in an online fashion. Unlike typical clustering approaches, the proposed method does not require the presence of all the data before performing clustering. The clustering decision is made as soon as an audio segment is received. Being causal, this method enables low-latency incremental speaker adaptation in online speech-to-text systems. It also gives a speaker tracking and indexing system the ability to label speakers with cluster ID on the fly. We show that the new online speaker clustering method yields better performance compared to the traditional hierarchical speaker clustering. Evaluation metrics for speaker clustering are also discussed.

This paper describes a set of new algorithms that perform speaker clustering in an online fashion. Unlike typical clustering approaches, the proposed method does not require the presence of all the data before performing clustering. The clustering decision is made as soon as an audio segment is received. Being causal, this method enables low-latency incremental speaker adaptation in online speech-to-text systems. It also gives a speaker tracking and indexing system the ability to label speakers with cluster ID on the fly. We show that the new online speaker clustering method yields better performance compared to the traditional hierarchical speaker clustering. Evaluation metrics for speaker clustering are also discussed.

This paper describes the development of the BBN Audio Indexing System for broadcast news in Arabic. Key issues addressed in this work revolve around the three major components of the audio indexing system: automatic speech recognition, speaker identification, and named entity identification. The system deals with several challenges introduced by the Arabic language, including the absence of short vowels in written text and the presence of compound words that are formed by the concatenation of certain conjunctions, prepositions, articles, and pronouns, as prefixes and suffixes to the word stem. The lack of short vowels in the transcripts prompted a novel solution that further demonstrated the power of hidden Markov models to deal with ambiguity. Another challenge was the acquisition of appropriate language modeling data, given the absence of broadcast news data for that purpose. We present performance results for all three components of the Audio Indexing System, which we believe represent the state of the art for Arabic broadcast news.

With the advent of essentially unlimited data storage capabilities and with the proliferation of the use of the Internet, it becomes reasonable to imagine a world in which it would be possible to access any of the stored information at will with a few keystrokes or voice commands. Since much of this data will be in the form of speech from various sources, it becomes important to develop the technologies necessary for indexing and browsing such audio data. This paper describes some of the requisite speech and language technologies that would be required and introduces an effort aimed at integrating these technologies into a system, called Rough 'n' Ready, which indexes speech data, creates a structural summarization, and provides tools for browsing the stored data. The technologies highlighted in the paper include speaker-independent continuous speech recognition, speaker segmentation and identification, name spotting, topic classification, story segmentation, and information retrieval. The system automatically segments the continuous audio input stream by speaker, clusters audio segments from the same speaker, identifies speakers known to the system, and transcribes the spoken words. It also segments the input stream into stories, based on their topic content, and locates the names of persons, places, and organizations. These structural features are stored in a database and are used to construct highly selective search queries for retrieving specific content from large audio archives.

We present a new method of speaker adapted training (SAT) that is more robust, faster, and results in lower error rate than the previous methods. The method, called inverse transform SAT (IT-SAT) is based on removing the differences between speakers before training, rather than modeling the differences during training. We develop several methods to avoid the problems associated with inverting the transformation. In one method, we interpolate the transformation matrix with an identity or diagonal transformation. We also apply constraints to the matrix to avoid estimation problems. Finally, we show that the resulting method is much faster, requires much less disk space, and results in higher accuracy than the original SAT method.

This paper presents a new method of topic spotting that attempts to retrieve detailed multiple simultaneous topics from broadcast news stories, each of which has about four different topics out of several thousand different topics. A new topic model uses a simple HMM where each state of the HMM represents one topic and the topic state emits topic-dependent keywords probabilistically. The model allows (unobserved) transitions among topics, word by word. These characteristics improve the discriminative ability between keywords and general words in a topic model and decrease the probabilistic overlap among the topic models more than the conventional topic models (such as a simple multinomial probability model). In addition, the model is not confused by words from multiple topics within one story. We applied the new method to topic spotting from manually transcribed texts of news shows. The new method showed better results in precision and recall rates than the conventional method.

We describe our work on automatic transcription of radio and television news broadcasts. This problem is very challenging for large vocabulary speech recognition because of the frequent and unpredictable changes that occur in speaker, speaking style, topic, channel, and background conditions. Faced with such a problem, there is a strong tendency to try to carve the input into separable classes and deal with each one independently. In our early work on this problem, however, we are finding that the rewards for condition-specific techniques are disappointingly small. This is forcing us to look for general, robust, and adaptive algorithms for dealing with extremely variable data. We describe the BBN BYB-LOS recognition system configured to handle off-line transcription and we characterize the speech contained in the 1996 DARPA Hub-4 testbed. On the partitioned development test set, we achieved a 29.4% overall word error rate.

We describe a new algorithm for topic classification that allows discrimination among thousands of topics. A mixture of topics explicitly models the fact that each story has multiple topics, that different words are related to different topics, and that most of the words are not related to any topic. The resulting model, trained by EM, has sharper distributions of words that result in more accurate topic classification. We tested the algorithm on transcribed broadcast news texts. When trained on one year of stories containing over 5,000 different topics and tested on new (later) stories the first choice topic was among the manually annotated choices 76% of the time.

In this paper, we present several approaches designed to increase the robustness of BYBLOS, the BBN continuous speech, hidden Markov model (HMM) recognition system. We address the problem of increased degradation in performance when there is mismatch in the characteristics of the training and the test microphones. First we compare RASTA processing and cepstrum mean subtraction as preprocessing methods, to compensate for unknown channel transfer function effects, when we have no information about the new microphone. Then we introduce a new algorithm that computes a probabilistic transformation from the training microphone codebook to that of a new microphone, given some information about the new microphone. We test this algorithm in supervised mode and, combined with a microphone selection method, in unsupervised mode. We present experimental results which show that the proposed algorithm combined with cepstrum mean subtraction, improves the recognition accuracy when the system is tested on a microphone with different characteristics than the one on which it was trained.>

We describe recent changes to the BYBLOS system's training and recognition algorithms and report on numerous experiments in large vocabulary speech recognition. In earlier work, we performed five key experiments that were designed to answer questions related to different training scenarios. We investigated (1) the effect of varying the number of training speakers if the total amount of training data remains constant, (2) data pooling versus model averaging for generating speaker-independent (SI) HMMs, (3) the benefit of doubling the acoustic training data, (4) SI versus SD performance when the SI training data is twelve times greater, (5) the effect of cross-domain training for both the acoustic and language models. Our recent work was focused on four specific problem areas sharing the common thread that the test condition exposes the recognizer to phenomena not observed in the training data. Here we investigated (1) words outside the vocabulary, (2) spoken language effects due to subject variability and spontaneous dictation, (3) non-native dialects of the language, and (4) new microphones not used in training.>

The authors describe four different ways in which they used the N-Best paradigm within the BYBLOS system. The most obvious use is for the efficient integration of speech recognition with a linguistic natural language understanding module. However, the authors have extended this principle to several other acoustic knowledge sources. They also describe a simple and efficient means for investigating and incorporating arbitrary knowledge sources. The N-Best hypotheses are used to provide close alternatives for discriminative training. Finally, the authors have developed a simple technique that allows them to optimize several weights and parameters within a system in a way that directly minimizes word error rate. Examples of each of these uses within the BYBLOS system are described.>

A paradigm for speaker-independent (SI) training of hidden Markov models (HMMs) is presented for continuous speech recognition. The method uses a large amount of speech from a few speakers instead of the traditional practice of using a little speech from many speakers. Furthermore, it avoids the common practice of pooling speech data from many speakers prior to training a SI model. The approach has been tested using the BBN Byblos system on the DARPA Resource Management corpus under standard test conditions. With only 12 speakers for training the SI models, recognition performance. comparable to that reported for other systems using 109 training speakers has been achieved. Besides surprisingly good recognition performance, this method offers many practical advantages that have implications for the way speech corpora, are designed and used for training SI models.>

A technique for using the speech of multiple reference speakers as a basis for speaker adaptation in large-vocabulary continuous-speech recognition is introduced. In contrast to other methods that use a pooled reference model, this technique normalizes the training speech from multiple reference speakers to a single common feature space before pooling it. The normalized and pooled speech is then treated as if it came from a single reference speaker for training the reference hidden Markov model (HMM). The usual probabilistic spectrum transformation can be applied to the reference HMM to model a new speaker. Preliminary experimental results are reported from applying this approach to over 100 reference speakers from the speaker-independent portion of the DARPA 1000-Word Resource Management Database.>

We introduce a new technique for using the speech of multiple reference speakers as a basis for speaker adaptation in large vocabulary continuous speech recognition. In contrast to other methods that use a pooled reference model, this technique normalizes the training speech from multiple reference speakers to a single common feature space before pooling it. The normalized and pooled speech can then be treated as if it came from a single reference speaker for training the reference hidden Markov model (HMM). Our usual prohabilistic spectrum transformation can be applied to the reference HMM to model a new (target) speaker. In this paper, we describe our baseline (single reference speaker) speakeradaptation system and give current performance results from a recent formal evaluation of the system. We also describe our proposal for adapting from multiple reference speakers and report on recent preliminary experimental results in support of the proposed technique. 1

The authors present several techniques to improve an algorithm presented last year for speaker-adaptive training in continuous speech recognition. The previous method uses a transformation matrix to modify the hidden Markov model (HMM) parameters of a prechosen prototype speaker to model a target speaker. To estimate the transformation matrix, it aligns a set of target speech with the same set of speech uttered by the prototype speaker using dynamic time warping. The authors focus on improving the previous method in the modeling of the spectral differences between two speakers, and the accuracy of the alignment. To improve the modeling of the spectral differences, they implemented a phoneme-dependent mapping procedure which transforms the prototype HMMs to the estimated target HMMs using a set of phoneme-dependent matrices. To improve the alignment, the authors developed a modeling of the silence, a linear duration normalization, and an iterative normalization procedure. They tested the new methods in the standard DARPA database with a grammar of perplexity 60. The performance shows a 30% word-error reduction compared to the previous algorithm.>

Three methods for smoothing discrete probability functions in discrete hidden Markov models for large-vocabulary continuous-speech recognition are presented. The smoothing is based on deriving a probabilistic co-occurrence matrix between the different vector-quantized spectra. Each estimated probability density is then multiplied by this matrix, ensuring that none of the probabilities are severely underestimated due to lack of training data. Experimental results show a 20-30% reduction in error rate when this smoothing is used. A word error rate of 3.0% is achieved with the DARPA 1000-word continuous speech recognition database and a word-pair grammar with a perplexity of 60.>

A novel text-dependent probabilistic spectral mapping method is presented for rapid speaker adaptation. The algorithm has been tested on the DARPA 1000-word resource management database with a grammar perplexity of 60. It results in significant better performance than the previous algorithms, and also provides recognition performance which is less than two times the word error rate for speaker-dependent training, using two minutes of adaptation speech.>

The system was trained in a speaker dependent mode on 28 minutes of speech from each of 8 speakers, and was tested on independent test material for each speaker. The system was tested with three artificial grammars spanning a broad perplexity range. The average performance of the system measured in percent word error was: 1.4% for a pattern grammar of perplexity 9, 7.5% for a word-pair grammar of perplexity 62, and 32.4% for a null grammar of perplexity 1000.>

In this paper, we describe BYBLOS, the BBN continuous speech recognition system. The system, designed for large vocabulary applications, integrates acoustic, phonetic, lexical, and linguistic knowledge sources to achieve high recognition performance. The basic approach, as described in previous papers [1, 2], makes extensive use of robust context-dependent models of phonetic coarticulation using Hidden Markov Models (HMM). We describe the components of the BYBLOS system, including: signal processing frontend, dictionary, phonetic model training system, word model generator, grammar and decoder. In recognition experiments, we demonstrate consistently high word recognition performance on continuous speech across: speakers, task domains, and grammars of varying complexity. In speaker-dependent mode, where 15 minutes of speech is required for training to a speaker, 98.5% word accuracy has been achieved in continuous speech for a 350-word task, using grammars with perplexity ranging from 30 to 60. With only 15 seconds of training speech we demonstrate performance of 97% using a grammar.

This paper deals with rapid speaker adaptation for speech recognition. We introduce a new algorithm that transforms hidden Markov models of speech derived from one "prototype" speaker so that they model the speech of a new speaker. The Speaker normalization is accomplished by a probabilistic spectral mapping from one speaker to another. For a 350 word task with a grammar and using only 15 seconds of speech for normalization, the recognition accuracy is 97% averaged over 6 speakers. This accuracy would normally require over 5 minutes of speaker dependent training. We derive the probabilistic spectral transformation of HMMs, describe an algorithm to estimate the transformation, and present recognition results.

This paper describes the results of our work in designing a system for large-vocabulary word recognition of continuous speech. We generalize the use of context-dependent Hidden Markov Models (HMM) of phonemes to take into account word-dependent coarticulatory effects, Robustness is assured by smoothing the detailed word-dependent models with less detailed but more robust models. We describe training and recognition algorithms for HMMs of phonemes-in-context. On a task with a 334-word vocabulary and no grammar (i.e., a branching factor of 334), in speaker-dependent mode, we show an average reduction in word error rate from 24% using context-independent phoneme models, to 10% when using robust context-dependent phoneme models.