We propose a neural network model to estimate word translation probabilities for Cross-Lingual Information Retrieval (CLIR). The model estimates better probabilities for word translations than automatic word alignments alone, and generalizes to unseen source-target word pairs. We further improve the lexical neural translation model (and subsequently CLIR), by incorporating source word context, and by encoding the character sequences of input source words to generate translations of out-of-vocabulary words. To be effective, neural network models typically need training on large amounts of data labeled directly on the final task, in this case relevance to queries. In contrast, our approach only requires parallel data to train the translation model, and uses an unsupervised model to compute CLIR relevance scores.

Individual ship detection from underwater audio is the task of deciding whether a specific ship is present, using sound captured by an underwater hydrophone. It is a task analogous to speaker identification (SID), in the sense that it is an open-class detection task; the ships present could be other irrelevant (“impostor”) ships, never encountered in the training data. We present two methodologies for tackling this problem, both motivated by our work in speech-related technologies: (i) one based on neural networks, which follows, to a large extent, the approach of [1], and (ii) one based on i-vectors and PLDA [2]. To the best of our knowledge, this is the first time that the topic of individual ship detection is approached as an open-class detection problem.

In this paper we describe the 2016 BBN conversational telephone speech keyword spotting system; the culmination of four years of research and development under the IARPA Babel program. The system was constructed in response to the NIST Open Keyword Search (OpenKWS) evaluation of 2016. We present our technological breakthroughs in building top-performing keyword spotting processing systems for new languages, in the face of limited transcribed speech, noisy conditions, and limited system build time of one week.

With the completion of the IARPA Babel program, it is possible to systematically analyze the performance of speech recognition systems across a wide variety of languages. We select 16 languages from the dataset and compare performance using a deep neural network-based acoustic model. The focus is on keyword spotting using the actual term-weighted value (ATWV) metric. We demonstrate that ATWV is keyword dependent, and that this must be accounted for in any cross-language analysis. Further, we show that while performance across languages does not track with any particular feature of the language, it is correlated with inter-annotator agreement.

Spoken term detection, especially of out-of-vocabulary (OOV) keywords, benefits from the use of sub-word systems. We experiment with different language-independent approaches to sub-word unit generation, generating both syllable-like and morpheme-like units, and demonstrate how the performance of syllable-like units can be improved by artificially increasing the number of unique units. The effect of unit choice is empirically evaluated using the eight languages from the 2016 IARPA BABEL evaluation.

We describe our submission to the NIST OpenSAD evaluation of speech activity detection of noisy audio generated by the DARPA RATS program. With frequent transmission degradation, channel interference and other noises added, simple energy thresholds do a poor job at SAD for this audio. The evaluation measured performance on both in-training and novel channels. Our approach used a system combination of feed-forward neural networks and bidirectional LSTM recurrent neural networks. System combination and unsupervised adaptation provided further gains on novel channels that lack training data. These improvements lead to a 26% relative improvement for novel channels over simple decoding. Our system resulted in the lowest error rate on the in-training channels and second on the out-of-training channels.

We present a three-pronged approach to improving Statistical Machine Translation (SMT), building on recent success in the application of neural networks to SMT. First, we propose new features based on neural networks to model various non-local translation phenomena. Second, we augment the architecture of the neural network with tensor layers that capture important higher-order interaction among the network units. Third, we apply multitask learning to estimate the neural network parameters jointly. Each of our proposed methods results in significant improvements that are complementary. The overall improvement is +2.7 and +1.8 BLEU points for Arabic-English and Chinese-English translation over a state-of-the-art system that already includes neural network features.

In this paper, we describe our work on the ASpIRE (Automatic Speech recognition In Reverberant Environments) challenge, which aims to assess the robustness of automatic speech recognition (ASR) systems. The main characteristic of the challenge is developing a high-performance system without access to matched training and development data. While the evaluation data are recorded with far-field microphones in noisy and reverberant rooms, the training data are telephone speech and close talking. Our approach to this challenge includes speech enhancement, neural network methods and acoustic model adaptation, We show that these techniques can successfully alleviate the performance degradation due to noisy audio and data mismatch.

The most common pipelines in keyword spotting involve some kind of speech recognition, which leads to the generation of sets of plausible hypotheses (e.g., word lattices), which are subsequently explored. The case of out-of-vocabulary (OOV) keywords is of special interest, because it requires representing keywords and/or lattices in an alternative format, so that the two can match. A number of techniques for dealing with OOV keywords have appeared in the literature; here, we focus on (i) fuzzy-phonetic search using phonetic confusion networks [1], and (ii) proxy-keyword search [2]. As we demonstrate in this paper, the combination of these two diverse techniques improves the ATWV of OOV keywords by at least 3% on average over the five development languages used in the second year of the IARPA Babel program.

Pronunciation lexicons can range from fully graphemic (modeling \neach word using the orthography directly) to fully phonemic \n(first mapping each word to a phoneme string). Between these \ntwo options lies a continuum of modeling options. We analyze \ntechniques that can improve the accuracy of a graphemic system \nwithout requiring significant effort to design or implement. \nThe analysis is performed in the context of the IARPA Babel \nproject, which aims to develop spoken term detection systems \nfor previously unseen languages rapidly, and with minimal human \neffort. We consider techniques related to letter-to-sound \nmapping and language-independent syllabification of primarily \ngraphemic systems, and discuss results obtained for six languages: \nCebuano, Kazakh, Kurmanji Kurdish, Lithuanian, Telugu \nand Tok Pisin.

Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, John Makhoul. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2014.

We explore the impact of morphological segmentation on keyword spotting (KWS).Despite potential benefits, stateof-the-art KWS systems do not use morphological information.In this paper, we augment a state-of-the-art KWS system with sub-word units derived from supervised and unsupervised morphological segmentations, and compare with phonetic and syllabic segmentations.Our experiments demonstrate that morphemes improve overall performance of KWS systems.Syllabic units, however, rival the performance of morphological units when used in KWS.By combining morphological, phonetic and syllabic segmentations, we demonstrate substantial performance gains.

As shown in [1, 2], score normalization is of crucial importance for improving the Average Term-Weighted Value (ATWV) measure that is commonly used for evaluating keyword spotting systems. In this paper, we compare three different methods for score normalization within a keyword spotting system that employs phonetic search. We show that a new unsupervised linear fit method results in better-estimated posterior scores, that, when fed into the keyword-specific normalization of [1], result in ATWV gains of 3% on average. Furthermore, when these scores are used as features within a supervised machine learning framework, they result in additional gains of 3.8% on average over the five languages used in the first year of the IARPA-funded project Babel.

In this paper we describe the Vietnamese conversational telephone speech keyword spotting system under the IARPA Babel program for the 2013 evaluation conducted by NIST. The system contains several, recently developed, novel methods that significantly improve speech-to-text and keyword spotting performance such as stacked bottleneck neural network features, white listing, score normalization, and improvements on semi-supervised training methods. These methods resulted in the highest performance for the official IARPA Babel surprise language evaluation of 2013.

This paper presents a set of techniques that we used to improve our keyword search system for the third phase of the DARPA RATS (Robust Automatic Transcription of Speech) program, which seeks to advance state of the art detection capabilities on audio from highly degraded radio communication channels. The results for both Levantine and Farsi, which are the two target languages for the keyword search (KWS) task, are reported. About 13% absolute reduction in word error rate (from 70.2% to 57.6%) is achieved by using acoustic features derived from stacked Multi-Layer Perceptrons (MLP) and Deep Neural Network (DNN) acoustic models. In addition to score normalization and score/system combination for keyword search, we showed that the false alarm rate at the target false reject rate (15%) was reduced by about 1% (from 5.39% to 4.45%) by reducing the deletion errors of the speech-to-text system. Index Terms: speech recognition, KWS, MLP, DNN

In this paper, we investigate semi-supervised training for low resource languages where the initial systems may have high error rate (≥ 70.0% word eror rate). To handle the lack of data, we study semi-supervised techniques including data selection, data weighting, discriminative training and multilayer perceptron learning to improve system performance. The entire suite of semi-supervised methods presented in this paper was evaluated under the IARPA Babel program for the keyword spotting tasks. Our semi-supervised system had the best performance in the OpenKWS13 surprise language evaluation for the limited condition. In this paper, we describe our work on the Turkish and Vietnamese systems.

We present two techniques that are shown to yield improved Keyword Spotting (KWS) performance when using the ATWV/MTWV performance measures: (i) score normalization, where the scores of different keywords become commensurate with each other and they more closely correspond to the probability of being correct than raw posteriors; and (ii) system combination, where the detections of multiple systems are merged together, and their scores are interpolated with weights which are optimized using MTWV as the maximization criterion. Both score normalization and system combination approaches show that significant gains in ATWV/MTWV can be obtained, sometimes on the order of 8-10 points (absolute), in five different languages. A variant of these methods resulted in the highest performance for the official surprise language evaluation for the IARPA-funded Babel project in April 2013.

This paper describes the speaker identification (SID) system developed by the Patrol team for the first phase of the DARPA RATS (Robust Automatic Transcription of Speech) program, which seeks to advance state of the art detection capabilities on audio from highly degraded communication channels. We present results using multiple SID systems differing mainly in the algorithm used for voice activity detection (VAD) and feature extraction. We show that (a) unsupervised VAD performs as well supervised methods in terms of downstream SID performance, (b) noise-robust feature extraction methods such as CFCCs out-perform MFCC front-ends on noisy audio, and (c) fusion of multiple systems provides 24% relative improvement in EER compared to the single best system when using a novel SVM-based fusion algorithm that uses side information such as gender, language, and channel id.

We present a semi-supervised language modeling technique to improve search performance on words without training data. Probabilities estimated from automatic transcripts of a large corpus of in-domain audio are added to an existing LM. Requiring only pronunciations and indomain audio, our method achieves 70% of the possible gain without any supervised data or development data. This method is contrasted with techniques using additional resources. If human effort is available, we also present a transcription regime to efficiently close the gap.

Useful training data for automatic speech recognition systems of colloquial speech is usually limited to expensive in-domain transcription. Broadcast news is an appealing source of easily available data to bootstrap into a new dialect. However, some languages, like Arabic, have deep linguistic differences resulting in poor cross domain performance. If no in-domain transcripts are available, but a large amount of indomain audio is, self-training may be a suitable technique to bootstrap into the domain. In this work, we attempt to adapt Modern Standard Arabic (MSA) models to Levantine Arabic without any in-domain manual transcription. We contrast with varying amounts of in-domain transcription and show that 1) Self-training is effective with only one hour of indomain transcripts. 2) Self-training is not a suitable solution to improve strong MSA models on Levantine. 3) Two metrics that quantify model bias predict self-training success. 4) Model bias explains the failure of self-training to adapt across strong domain mismatch. Index Terms: Arabic ASR, domain adaptation, self-training 1.

We measure the effects of a weak language model, estimated from as little as 100k words of text, on unsupervised acoustic model training and then explore the best method of using word confidences to estimate n-gram counts for unsupervised language model training. Even with 100k words of text and 10 hours of training data, unsupervised acoustic modeling is robust, with 50% of the gain recovered when compared to supervised training. For language model training, multiplying the word confidences together to get a weighted count produces the best reduction in WER by 2% over the baseline language model and 0.5% absolute over using unweighted transcripts. Oracle experiments show that a larger gain is possible, but better confidence estimation techniques are needed to identify correct n-grams.

Previous work on self-training of acoustic models using unlabeled data reported significant reductions in WER assuming a large phonetic dictionary was available. We now assume only those words from ten hours of speech are initially available. Subsequently, we are then given a large vocabulary and then quantify the value of repeating selftraining with this larger dictionary. This experiment is used to analyze the effects of self-training on categories of words. We report the following findings: (i) Although the small 5k vocabulary raises WER by 2% absolute, self-training is equally effective as using a large 75k vocabulary. (ii) Adding all 75k words to the decoding vocabulary after self-training reduces the WER degradation to only 0.8% absolute. (iii) Self-training most benefits those words in the unlabeled audio but not transcribed by a wide margin.

This paper investigates the use of several language model adaptation techniques applied to the task of machine translation from Arabic broadcast speech. Unsupervised and discriminative approaches slightly outperform the traditional perplexity-based optimization technique. Language model adaptation, when used for n-best rescoring, improves machine translation performance by 0.3-0.4 BLEU and reduces translation edit rate (TER) by 0.2-0.5% compared to an unadapted LM.

This paper presents a set of experiments that we conducted in order to optimize the performance of an Arabic/English machine translation system on broadcast news and conversational speech data. Proper integration of speech-to-text (STT) and machine translation (MT) requires special attention to issues such as sentence boundary detection, punctuation, STT accuracy, tokenization, conversion of spoken numbers and dates to written form, optimization of MT decoding weights, and scoring. We discuss these issues, and show that a carefully tuned STT/MT integration can lead to significant translation accuracy improvements compared to simply feeding the regular STT output to a text MT system.

In many applications of topic spotting technology, especially those that require a human review of in-topic documents, a low false alarm rate is a key requirement. Topic spotting techniques typically include a rejection scheme to filter out off-topic documents. In this paper we present a robust methodology for rejecting off-topic messages that, in addition to modeling the topics of interest, uses a so-called alternate model for topics that are not included in the set of topics of interest. Specifically, we introduce two novel techniques for estimating topic-specific rejection thresholds - a parametric technique that can be viewed as transformation of topic-independent thresholds, and a nonparametric technique based on constrained optimization of false rejections subject to a pre-specified number of false acceptances. Our experiments on newsgroup messages demonstrate that when adequate training data is available topic-specific threshold estimation techniques can outperform topic-independent thresholds in terms of the ROC curve.

Discriminatively trained feature transforms such as MPE-HLDA, fMPE and MMI-SPLICE have been shown to be effective in reducing recognition errors in today's state-of-the-art speech recognition systems. This paper introduces the concept of region dependent linear transform (RDLT), which unifies the above three types of feature transforms and provides a framework for the estimation of piece-wise linear feature projections, based on the minimum phoneme error (MPE) criterion. Recognition results on English conversational telephone speech data show that RDLT offers consistent gains over the baseline systems, which are trained using the LDA+MLLT projection

This paper presents our recent effort that aims at improving our Arabic broadcast news (BN) recognition system by using thousands of hours of un-transcribed Arabic audio in the way of unsupervised training. Unsupervised training is first carried out on the 1,900-hour English topic detection and tracking (TDT) data and is compared with the lightly-supervised training method that we have used for the DARPA EARS evaluations. The comparison shows that unsupervised training produces a 21.7% relative reduction in word error rate (WER), which is comparable to the gain obtained with light supervision methods. The same unsupervised training strategy carried out on a similar amount of Arabic BN data produces an 11.6% relative gain. The gain, though considerable, is substantially smaller than what is observed on the English data. Our initial work towards understanding the reasons for this difference is also described

In this paper, we present a novel approach for morphological de-composition in large vocabulary Arabic speech recognition. It achieved low out-of-vocabulary (OOV) rate as well as high recognition accuracy in a state-of-the-art Arabic broadcast news transcription system. In this approach, the compound words are decomposed into stems and affixes in both language training and acoustic training data. The decomposed words in the recognition output are re-joined before scoring. Four algorithms are experimented and compared in this work. The best system achieved 1.9% absolute reduction (9.8% relative) in word error rate (WER) when compared to the 64K-word baseline. The recognition performance of this system is also comparable to a 300K-word recognition system trained on the normal words. In the meantime, the decomposed system is much faster in terms of speed and also needs less memory than the systems with larger than 64K vocabularies.

The region-dependent transform (RDT) is a feature extrac-tion method for speech recognition that employs the Minimum Phoneme Error (MPE) criterion to optimize a set of feature trans-forms, each concentrating on a region of the acoustic space. Pre-vious results have shown that RDT gives significant recognition-error reduction in a large vocabulary speaker-independent (SI) sys-tem. As a follow-up investigation, this paper presents the re-cent progress of applying RDT in speaker-adaptive training (SAT). Similar to previous SI results, the integration of RDT with SAT yields 7 % relative improvement in word error rate (WER). Also, theoretical comparisons are made between RDT and other discrim-inative feature extraction methods, including the improved version of the feature-space MPE (fMPE) that uses the “mean-offsets ” as additional input features. Index Terms: speech recognition, discriminative training, feature extraction, region-dependent transform.

This paper aims to quantify the main error types the 2004 BBN speech recognition system made in the broadcast news (BN) and conversational telephone speech (CTS) DARPA EARS evaluations. We show that many of the remaining errors occur in clusters rather than isolated, have specific causes, and differ to some extent between the BN and CTS domains. The correctly recognized words are also clustered and are highly correlated with regions where the system produces a single hypothesized choice per word. A statistical analysis of some well-known error causes (out-of-vocabulary words, word fragments, hesitations, and unlikely language constructs) was performed in order to assess their contribution to the overall word error rate (WER). We conclude with a discussion of the lower bound on the WER introduced by the human annotator disagreement

This paper describes the progress made in the transcription of broadcast news (BN) and conversational telephone speech (CTS) within the combined BBN/LIMSI system from May 2002 to September 2004. During that period, BBN and LIMSI collaborated in an effort to produce significant reductions in the word error rate (WER), as directed by the aggressive goals of the Effective, Affordable, Reusable, Speech-to-text [Defense Advanced Research Projects Agency (DARPA) EARS] program. The paper focuses on general modeling techniques that led to recognition accuracy improvements, as well as engineering approaches that enabled efficient use of large amounts of training data and fast decoding architectures. Special attention is given on efforts to integrate components of the BBN and LIMSI systems, discussing the tradeoff between speed and accuracy for various system combination strategies. Results on the EARS progress test sets show that the combined BBN/LIMSI system achieved relative reductions of 47% and 51% on the BN and CTS domains, respectively.

In this paper, we present cluster-dependent acoustic modeling for large-vocabulary speech recognition. With large amounts of acoustic training data, we build multiple cluster-dependent models (CDM), each focusing on a group of speakers in order to represent speaker-dependent characteristics. It is motivated by the fact that a sufficiently trained speaker-dependent (SD) model is better than the speaker-independent (SI) model. During decoding, we decode the data of each test speaker using CDMs selected under certain criteria to achieve high recognition accuracy. Various speaker clustering and model selection techniques are proposed and compared in the task of broadcast news (BN) transcription. The CDM provided more than 1% absolute gain in unadapted decoding and 0.5% gain in adapted decoding when compared to our baseline system on the EARS BN 2003 development test set.

This paper describes the BBN real-time recognition systems used in the 2004 Rich Transcription (RT) benchmark test for the English Conversational Telephone Speech (CTS) and Broadcast News (BN) tasks. We describe the system architecture, along withthe algorithms weused inorder to reduce computation with minimal impact on recognition accuracy. Particular choices in the design of thefinal system are analyzed toshow the trade-offs between speed and accuracy. We also present recently developed new architecture for the real-time systems, which outperforms the systems we submitted for the RT04 benchmark tests for both domains.

This paper describes the BBN English Broadcast News transcription system developed for the EARS Rich Transcription 2004 (RT04) evaluation. In comparison to the BBN RT03 system, we achieved around 22% relative reduction in word error rate for all EARS BN development test sets. The use of additional acoustic training data acquired through Light Supervision based on thousands of hours of found data made the biggest contribution to the improvement. Better audio segmentation, through the use of an online speaker clustering algorithm and chopping speaker turns into moderately long utterances, also contributed substantially to the improvement. Other contributions, even of modest size but adding up nicely, include using discriminative training for all acoustic models, using word duration as an additional knowledge source during N-best rescoring, and using updated lexicon and language models.

In this paper we describe the English Conversational Telephone Speech (CTS) recognition system jointly developed by BBN and LIMSI under the DARPA EARS program for the 2004 evalua-tion conducted by NIST. The 2004 BBN/LIMSI system achieved a word error rate (WER) of 13.5 % at 18.3xRT (real-time as mea-sured on Pentium 4 Xeon 3.4 GHz Processor) on the EARS progress test set. This translates into a 22.8 % relative improvement in WER over the 2003 BBN/LIMSI EARS evaluation system, which was run without any time constraints. In addition to reporting on the system architecture and the evaluation results, we also highlight the significant improvements made at both sites. 1.

We report on the results of the first evaluations for the BBN/LIMSI system under the new DARPA EARS program. The evaluations were carried out for conversational telephone speech (CTS) and broadcast news (BN) for three languages: English, Mandarin, and Arabic. In addition to providing system descriptions and evaluation results, the paper highlights methods that worked well across the two domains and those few that worked well on one domain but not the other. For the BN evaluations, which had to be run under 10 times real-time, we demonstrated that a joint BBN/LIMSI system with a time constraint achieved better results than either system alone.

Although Arabic is currently one of the most widely spoken languages in the world, there has been relatively little speech recognition research on Arabic compared to other languages. Moreover, most previous work has concentrated on the recognition of formal rather than dialectal Arabic. This paper reports on our project at the 2002 Johns Hopkins Summer Workshop, which focused on the recognition of dialectal Arabic. Three problems were addressed: (a) the lack of short vowels and other pronunciation information in Arabic texts; (b) the morphological complexity of Arabic; and (c) the discrepancies between dialectal and formal Arabic. We present novel approaches to automatic vowel restoration, morphology-based language modeling and the integration of out-of-corpus language model data, and report significant word error rate improvements on the LDC Arabic CallHome task.

This paper presents new approaches to headline generation for English newspaper texts, with an eye toward the production of document surrogates for document selection in cross-language information retrieval. This task is difficult because the user must make decisions about relevance based on (often poor) translations of retrieved documents. To facilitate the decision-making process we need translations that can be assessed rapidly and accurately; our approach is to provide an English headline for the non-English document. We describe two approaches to headline generation and their application to the recent DARPA TIDES-2003 Surprise Language Exercise for Hindi. For comparison, we also implemented an alternative method for surrogate generation: a system that produces topic lists for (Hindi) articles. We present the results of a series of experiments comparing each of these approaches. We demonstrate in both automatic and human evaluations that our linguistically motivated approach outperforms two other surrogate-generation methods: a statistical system and a topic discovery system.

We present a novel architecture for providing automated telephone Directory Assistance (DA). The architecture couples a large-vocabulary, statistical n-gram, speech recognition engine with a statistical retrieval system. The use of a statistical n-gram allows for the recognition of unconstrained spoken queries while the statistical retrieval engine allows for an inexact match between a particular spoken query and the training data. Allowing for unconstrained recognition and an inexact match provides the framework for high levels of automation. Once the retrieval engine returns a ranked set of frequently requested telephone numbers (FRN), the rejection module uses a classifier to compute a confidence-like score that is used to make the automation decision. With actual customer calls into an operational, automated DA call center and an FRN set size of 25000 numbers, the new architecture is capable of delivering more than 17% correct automation at a false accept rate of 0.76%.

We present a method for performing optical character recognition (OCR) of text in video images. Recognition of videotext is a challenging problem due to various factors such as the presence of rich, dynamic backgrounds, low resolution, color, etc. Our strategy is to process the video images to produce high-resolution binarized text images that resemble printed text. We describe a novel clustering and relaxation procedure that combines stroke and color information to separate the text from the background. The binarized text image is then recognized with our Byblos OCR engine (Natarajan et al., 2001; Schwartz et al., 1996) using hidden Markov models trained on similar data. We present experimental results on a video-data corpus collected from broadcast news programs. Currently the system delivers a character error rate of 8.3% on independent multi-font test data from this corpus.

This paper presents a script-independent methodology for optical character recognition (OCR) based on the use of hidden Markov models (HMM). The feature extraction, training and recognition components of the system are all designed to be script independent. The training and recognition components were taken without modification from a continuous speech recognition system; the only component that is specific to OCR is the feature extraction component. To port the system to a new language, all that is needed is text image training data from the new language, along with ground truth which gives the identity of the sequences of characters along each line of each text image, without specifying the location of the characters on the image. The parameters of the character HMMs are estimated automatically from the training data, without the need for laborious handwritten rules. The system does not require presegmentation of the data, neither at the word level nor at the character level. Thus, the system is able to handle languages with connected characters in a straightforward manner. The script independence of the system is demonstrated in three languages with different types of script: Arabic, English, and Chinese. The robustness of the system is further demonstrated by testing the system on fax data. An unsupervised adaptation method is then described to improve performance under degraded conditions.

We present a new, script-independent, HMM-based technique to locate text lines on images containing one or more paragraphs of simple-column text. The parameters of the HMMs are trained online on each image using an unsupervised training procedure. We present results of line finding experiments in Arabic, Chinese and English to demonstrate the performance as well as the script-independent nature of the technique. A comparison of the HMM-based line finding with the manual line finding shows that the use of HMM-based technique does not lead to a significant increase in the recognition error rate.

With the advent of essentially unlimited data storage capabilities and with the proliferation of the use of the Internet, it becomes reasonable to imagine a world in which it would be possible to access any of the stored information at will with a few keystrokes or voice commands. Since much of this data will be in the form of speech from various sources, it becomes important to develop the technologies necessary for indexing and browsing such audio data. This paper describes some of the requisite speech and language technologies that would be required and introduces an effort aimed at integrating these technologies into a system, called Rough 'n' Ready, which indexes speech data, creates a structural summarization, and provides tools for browsing the stored data. The technologies highlighted in the paper include speaker-independent continuous speech recognition, speaker segmentation and identification, name spotting, topic classification, story segmentation, and information retrieval. The system automatically segments the continuous audio input stream by speaker, clusters audio segments from the same speaker, identifies speakers known to the system, and transcribes the spoken words. It also segments the input stream into stories, based on their topic content, and locates the names of persons, places, and organizations. These structural features are stored in a database and are used to construct highly selective search queries for retrieving specific content from large audio archives.

We present several algorithms that speed up our BBN BYBLOS decoder. We briefly describe the techniques that we have used before this year. Then we present new techniques that speed up the recognition search by a factor of 10 with little effect on accuracy using a combination of fast Gaussian computation, grammar spreading, and grammar caching, within the 2-pass n-best paradigm. We also describe our decoder metering strategy, which allows us to conveniently test for search errors. Finally, we describe a grammar compression technique that decreases the storage needed for each additional ngram to only 10 bits.

We present a very fast and accurate fast-match algorithm which, when followed by a regular beam search restricted within only the subset of words selected by the fast-match, can speed up the recognition process by at least two orders of magnitude in comparison to a typical single-pass speech recognizer utilizing the Viterbi (or beam) search algorithm. In this search strategy, the recognition vocabulary is structured as a single phonetic tree in the fast-match pass. The search on this phonetic tree is a variation of the Viterbi algorithm. Especially, we are able to use a word bigram language model without making copies of the tree during the search. This is a novel fast-match algorithm that has two important properties: high-accuracy recognition and run-time proportional to only the cube root of the vocabulary size.

We present probabilistic models for use in detecting and tracking topics in broadcast news stories. Our information retrieval (IR) models are formally explained. The topic detection and tracking (TDT) initiative is discussed. The application of probabilistic models to the topic detection and tracking tasks is developed, and enhancements are discussed. We discuss four variations of these models, and we report our preliminary test results from the current TDT corpus.

We present some recent advances in the BBN BYBLOS OCR system. This OCR system can be used to recognize Arabic, Chinese, and English with high accuracy. A major change in the system is the use of continuous-density HMMs, which allow us to take advantage of a large amount of training data and to use unsupervised adaptation methods to improve accuracy in many cases, e.g., on degraded data. Another advance is the substantial increase in recognition speed. With this increased speed, the system is fast enough for practical use on Arabic and English data. The extension of the system to Chinese further demonstrated the language independence of this system and showed that this system can be used on languages with large character sets and complicated character structures. The Chinese OCR system yielded high accuracy on newspaper data.

This paper is concerned with techniques for performing robust OCR of degraded documents, such us faxed text, using a hidden Markov model (HMM) based OCR system. We present two strategies for dealing with degraded documents. The first strategy is to train the system on degraded documents that have been subjected to the same, or similar, degradation process as the documents to be recognized. The second, more sophisticated, strategy is to use adaptation to adjust the parameters of the trained model in order to improve recognition accuracy on a specific document. This adjustment of model parameters is typically posed as a constrained optimization problem wherein a certain prespecified objective function is to be optimized. We present a comparative study of two objective functions. The likelihood function and the posterior probability. A variation of the basic posterior probability method is also discussed. Using adaptation with a model trained on fax-degraded data we have reduced, by a factor of three, the character error rate on fax-degraded text images generated from the University of Washington English Image Database I.

In this paper, we describe our recent work in fast automatic transcription of broadcast news programming from radio and television. Given our state-of-the-art BBN BYBLOS primary system [1] running at 230 times real time (230xRT) we show that eliminating and approximating many computationally expensive components speeds up the system by a factor of more than 20 without significant loss in recognition accuracy. This is accomplished without retraining or changing the base-line system structure. The components of the primary system which are refined include segmentation, adaptation, decoding, cross-word rescoring with adaptation, and system combination.

ABSTRACT Through the present paper, the evaluation results of aSpeaker Independent Robust-to-Noise Isolated WordRecognizer are presented. The system, which is in partthe results achieved by the project IVORY (ESPRITproject No. 20277) [6], is intended for working in highlynon-stationary environments. The system comprises twomain modules: the Noise Cancellator and the SpeechRecognizer itself. System robustness is achieved in thenoise cancellation module. This module incorporates anAdaptive Filter [7] [13], operating in the time domain,and a subsequent Spectral Subtraction step [2] operatingin the frequency domain with the enhanced signalprovided by the previous stage. Recognition results fordifferent noise cancellation configurations and forseveral Parameter Extraction Front-Ends, includingLPC [7], FFT Cepstrum [3] and PLP based methods [8]are presented.Keywords: robust speech recognition, non-stationaryenvironments, recognition results. 1. INTRODUCTION Environmental noise is one of the major problems to befaced by speech recognition systems working under realconditions, as most systems performance degradesubstantially with low S/N ratios [1]. This is particularlyimportant in non-stationary environments as in cars,plane cockpits, workshops with heavy machinery,crowded public places, etc. These scenarios arecharacterized by high noise levels (of about 95 dB withSNR´s about 0- 10 dB) produced by different sourcesand where noise contributions are especially harsh, asmany different noise sources may be activesimultaneously, varying dramatically with time. To facewith the treatment of noise, Robust Speech Recognitionhas become an objective in itself [4].Along the years, several independent techniques havebeen proposed. Most of them are based in two basicapproaches: Adaptive Time-Domain Filtering (ATDF)[14], and Frequency-Domain Spectral Subtraction(FDSS) [2]. Both methods have been used independentlywith different degree of success, depending on theproblem being solved. Generally, ATDF has been usedwith array microphones, ranging from two-microphoneto multiple-microphone systems. On the other hand,FDSS has been preferred for systems where a singlemicrophone is a requirement (e.g. Telephone speechrecognition). This technique has found its niche inapplications where noise is more or less a quasi-stationary and evenly distributed process. ATDF has theadvantage of being highly suitable to deal with non-stationary signals as speech and most kinds of noise,although at a higher cost (both in instrumentation and incomputational power). Although both methods can beintegrated in the same system, most of the times theyhave been used separately [5]. However, the co-operation of both procedures allows important gains inthe S/N ratio to be achieved, improving the overallrecognition results, as well.

We p r e s e n t a new method for information retrieval using hidden Markov models (HMMs).We d e v elop a general framework for incorporating multiple word generation mechanisms within the same model.We then demonstrate that an extremely simple realization of this model substantially outperforms standard tf :idf ranking on both the TREC-6 and TREC-7 ad hoc retrieval tasks.We go on to present a novel method for performing blind feedback in the HMM framework, a more complex HMM that models bigram production, and several other algorithmic re nements.Together, these methods form a state-of-the-art retrieval system that ranked among the best on the TREC-7 ad hoc retrieval task.

We present an omnifont, unlimited-vocabulary OCR system for English and Arabic. The system is based on hidden Markov models (HMM), an approach that has proven to be very successful in the area of automatic speech recognition. We focus on two aspects of the OCR system. First, we address the issue of how to perform OCR on omnifont and multi-style data, such as plain and italic, without the need to have a separate model for each style. The amount of training data from each style, which is used to train a single model, becomes an important issue in the face of the conditional independence assumption inherent in the use of HMMs. We demonstrate mathematically and empirically how to allocate training data among the different styles to alleviate this problem. Second, we show how to use a word-based HMM system to perform character recognition with unlimited vocabulary. The method includes the use of a trigram language model on character sequences. Using all these techniques, we have achieved character error rates of 1.1 percent on data from the University of Washington English Document Image Database and 3.3 percent on data from the DARPA Arabic OCR Corpus.

We present a new method of speaker adapted training (SAT) that is more robust, faster, and results in lower error rate than the previous methods. The method, called inverse transform SAT (IT-SAT) is based on removing the differences between speakers before training, rather than modeling the differences during training. We develop several methods to avoid the problems associated with inverting the transformation. In one method, we interpolate the transformation matrix with an identity or diagonal transformation. We also apply constraints to the matrix to avoid estimation problems. Finally, we show that the resulting method is much faster, requires much less disk space, and results in higher accuracy than the original SAT method.

A requirement of virtually all analytic tools, such as timeline and spatial analysis, is structured data; however, much data is in text, an unstructured form. This article presents a new technology to bridge the gap between data buried in text and the requirement of structured data for analysis. The outcome should be an easy-to-maintain information technology component to support DoD and law enforcement applications. Our new approach uses statistical pattern recognition to learn to find data that is locally identifiable, e.g., that is not highly dependent on contexts. Examples are person names, organization names, locations, dates, times, monetary amounts, phone numbers, addresses, and social security numbers. The paper describes the statistical model employed, compares and contrasts the approach to previous approaches, numerically evaluates the adequacy of the technology on Government-supplied data, and illustrates the kind of examples needed for the system to learn to recognize the data desired from examples in documents.

We address the problem of determining what size test set guarantees statistically significant results in a character recognition task, as a function of the expected error rate. We provide a statistical analysis showing that if, for example, the expected character error rate is around 1 percent, then, with a test set of at least 10,000 statistically independent handwritten characters (which could be obtained by taking 100 characters from each of 100 different writers), we guarantee, with 95 percent confidence, that: (1) the expected value of the character error rate is not worse than 1.25 E, where E is the empirical character error rate of the best recognizer, calculated on the test set; and (2) a difference of 0.3 E between the error rates of two recognizers is significant. We developed this framework with character recognition applications in mind, but it applies as well to speech recognition and to other pattern recognition problems.

This paper presents a new method of topic spotting that attempts to retrieve detailed multiple simultaneous topics from broadcast news stories, each of which has about four different topics out of several thousand different topics. A new topic model uses a simple HMM where each state of the HMM represents one topic and the topic state emits topic-dependent keywords probabilistically. The model allows (unobserved) transitions among topics, word by word. These characteristics improve the discriminative ability between keywords and general words in a topic model and decrease the probabilistic overlap among the topic models more than the conventional topic models (such as a simple multinomial probability model). In addition, the model is not confused by words from multiple topics within one story. We applied the new method to topic spotting from manually transcribed texts of news shows. The new method showed better results in precision and recall rates than the conventional method.

We describe our work on automatic transcription of radio and television news broadcasts. This problem is very challenging for large vocabulary speech recognition because of the frequent and unpredictable changes that occur in speaker, speaking style, topic, channel, and background conditions. Faced with such a problem, there is a strong tendency to try to carve the input into separable classes and deal with each one independently. In our early work on this problem, however, we are finding that the rewards for condition-specific techniques are disappointingly small. This is forcing us to look for general, robust, and adaptive algorithms for dealing with extremely variable data. We describe the BBN BYB-LOS recognition system configured to handle off-line transcription and we characterize the speech contained in the 1996 DARPA Hub-4 testbed. On the partitioned development test set, we achieved a 29.4% overall word error rate.

We describe the first sentence understanding system that is completely based on learned methods both for understanding individual sentences, and determining their meaning in the context of preceding sentences. We divide the problem into three stages: semantic parsing, semantic classification, and discourse modeling. Each of these stages requires a different model. When we ran this system on the last test (December, 1994) of the ARPA Air Travel Information System (ATIS) task, we achieved a 13.7% error rate. The error rate for those sentences that are context-independent (class A) was 9.7%.

The authors present a set of techniques for omnifont, unlimited-vocabulary OCR, within the context of a system based on hidden Markov models (HMM). First, they address the issue of how to perform OCR on omnifont and multi-style data, such as plain and italic, without the need to have a separate model for each style. The amount of training data from each style, which is used to train a single model, becomes an important issue in the face of the conditional independence assumption inherent in the use of HMMs. They demonstrate mathematically and empirically how to allocate training data among the different styles to alleviate this problem. Second, they show how to use a word-based HMM system to perform character recognition with unlimited vocabulary. The method includes the use of a trigram language model on character sequences. Using all these techniques, they have achieved character error rates of 1.1% on data from the University of Washington English Document Image Database and 3.3% on data from the DARPA Arabic OCR Corpus.

In this paper, we describe the new BBN BYBLOS e#cient 2-Pass N-Best decoder used for the 1996 Hub-4 Benchmark Tests. The decoder uses a quick fastmatch to determine the likely word endings. Then in the second pass, it performs a time-synchronous beam search using a detailed continuousdensity HMM and a trigram language model to decide the word starting positions. From these word starts, the decoder, without looking at the input speech, constructs a trigram word lattice, and generates the top N likely hypotheses. This new 2-pass N-Best decoder maintains comparable recognition performance as the old 4-pass N-Best decoder, while its search strategy is simpler and much more e#cient. 1. INTRODUCTION As previously described in #2#, the old BBN BYBLOS decoder used a multi-pass search strategy consisting of 4 passes to generate the top N most likely hypotheses, which were then rescored using more detailed, but expensive knowledge sources. These N best hypotheses were then reordered and the to...

We describe a new algorithm for topic classification that allows discrimination among thousands of topics. A mixture of topics explicitly models the fact that each story has multiple topics, that different words are related to different topics, and that most of the words are not related to any topic. The resulting model, trained by EM, has sharper distributions of words that result in more accurate topic classification. We tested the algorithm on transcribed broadcast news texts. When trained on one year of stories containing over 5,000 different topics and tested on new (later) stories the first choice topic was among the manually annotated choices 76% of the time.

We present a natural language interface system which is based entirely on trained statistical models.The system consists of three stages of processing: parsing, semantic interpretation, and discourse.Each of these stages is modeled as a statistical process.The models are fully integrated, resulting in an end-to-end system that maps input utterances into meaning representation frames.

We use a hidden Markov model (HMM) based continuous speech recognition system to perform off-line character recognition (OCR) of Arabic printed text. The HMM trainer and recognizer are used without change, however we modify the feature extraction stage to compute features relevant to OCR. Although we begin by segmenting the page into a collection of lines, no further segmentation is necessary for either recognition or training. Experiments on the ARPA Arabic data corpus yield a range of character error rates from under one percent for a single computer font to 2.8% for multiple-font recognition of a wide range of material from books, magazines and newspapers.

We present a framework for maximum a posteriori (MAP) adaptation of large scale HMM recognizers. First we review the standard MAP adaptation for Gaussian mixtures. We then show how MAP can be used to estimated transformations which are shared across many parameters. Finally, we combine both techniques: each of the HMM models is adapted based on an interpolation of MAP estimates obtained under varying degrees of sharing. We evaluate this algorithm for adaptation of a continuous density HMM with 96 K Gaussians and show that very satisfactory improvements can be achieved, especially for adaptation of non-native speakers of American English.

In this paper we show how continuous speech recognition methods can be used for character recognition resulting in a technology that is language independent and does not require presegmentation of the data at the character and word levels. In multifont experiments on the ARPA Arabic OCR Corpus an average character error rate of 1.9% is obtained using the BBN BYBLOS continuous speech recognition system with no modifications. A first experiment using the identical system and procedures, trained and tested on a subset of the English Univ. of Washington OCR corpus resulted in 1.4% character error.

This paper presents a study of different methods for phoneme duration modeling in large vocabulary speech recognition. We investigate the employment of phoneme duration and the effect of context, speaking rate and lexical stress in the duration of phoneme segments in a large vocabulary speech recognition system. The duration models are used in a postprocessing phase of BYBLOS, our baseline HMM-based recognition system, to rescore the N-Best hypotheses. We describe experiments with the 5 K word ARPA Wall Street Journal (WSJ) corpus. The results show that integration of duration models that take into account context and speaking rate can improve the word accuracy of the baseline recognition system.

We present a framework for maximum a posteriori adaptation of large scale HMM speech recognizers. In this framework, we introduce mechanisms that take advantage of correlations present among HMM parameters in order to maximize the number of parameters that can be adapted by a limited number of observations. We are also separately exploring the feasibility of instantaneous adaptation techniques. Instantaneous adaptation attempts to improve recognition on a single sentence, the same sentence that is used to estimate the adaptation. We show that sizable gains (20-40% reduction in error rate) can be achieved by either batch or incremental adaptation for large vocabulary recognition of native speakers. The same techniques cut the error rate for recognition of non-native speakers by factors of 2 to 4, bringing their performance much closer to the native speaker performance. We also demonstrate that good improvements in performance (25-30%) are realized when instantaneous adaptation is used for recognition of non-native speakers.

We incorporate the hierarchical mixtures of experts (HME) method of probability estimation, developed by Jordan (see Neural Computation, 1994), into an HMM-based continuous speech recognition system. The resulting system can be thought of as a continuous-density HMM system, but instead of using Gaussian mixtures, the HME system employs a large set of hierarchically organized but relatively small neural networks to perform the probability density estimation. The hierarchical structure is reminiscent of a decision tree except for two important differences: each "expert" or neural net performs a "soft" decision rather than a hard decision, and, unlike ordinary decision trees, the parameters of all the neural nets in the HME are automatically trainable using the EM algorithm. We report results on the ARPA 5,000-word and 40,000-word Wall Street Journal corpus using HME models.

We describe and evaluate hidden understanding models, a statistical learning approach to natural language understanding. Given a string of words, hidden understanding models determine the most likely meaning for the string. We discuss 1) the problem of representing meaning in this framework, 2) the structure of the statistical model, 3) the process of training the model, and 4) the process of understanding using the model. Finally, we give experimental results, including results on an ARPA evaluation.

In this paper, we present several approaches designed to increase the robustness of BYBLOS, the BBN continuous speech, hidden Markov model (HMM) recognition system. We address the problem of increased degradation in performance when there is mismatch in the characteristics of the training and the test microphones. First we compare RASTA processing and cepstrum mean subtraction as preprocessing methods, to compensate for unknown channel transfer function effects, when we have no information about the new microphone. Then we introduce a new algorithm that computes a probabilistic transformation from the training microphone codebook to that of a new microphone, given some information about the new microphone. We test this algorithm in supervised mode and, combined with a microphone selection method, in unsupervised mode. We present experimental results which show that the proposed algorithm combined with cepstrum mean subtraction, improves the recognition accuracy when the system is tested on a microphone with different characteristics than the one on which it was trained.>

We describe recent changes to the BYBLOS system's training and recognition algorithms and report on numerous experiments in large vocabulary speech recognition. In earlier work, we performed five key experiments that were designed to answer questions related to different training scenarios. We investigated (1) the effect of varying the number of training speakers if the total amount of training data remains constant, (2) data pooling versus model averaging for generating speaker-independent (SI) HMMs, (3) the benefit of doubling the acoustic training data, (4) SI versus SD performance when the SI training data is twelve times greater, (5) the effect of cross-domain training for both the acoustic and language models. Our recent work was focused on four specific problem areas sharing the common thread that the test condition exposes the recognizer to phenomena not observed in the training data. Here we investigated (1) words outside the vocabulary, (2) spoken language effects due to subject variability and spontaneous dictation, (3) non-native dialects of the language, and (4) new microphones not used in training.>

A hidden Markov model (HMM) based continuous speech recognition system is applied to on-line cursive handwriting recognition. The base system is unmodified except for using handwriting feature vectors instead of speech. Due to inherent properties of HMMs, segmentation of the handwritten script sentences is unnecessary. A 1.1% word error rate is achieved for a 3050 word lexicon, 52 character, writer-dependent task and 3%-5% word error rates are obtained for six different writers in a 25,595 word lexicon, 86 character, writer-dependent task. Similarities and differences between the continuous speech and on-line cursive handwriting recognition tasks are explored; the handwriting database collected over the past year is described; and specific implementation details of the handwriting system are discussed.>

In this paper, we incorporate the Hierarchical Mixtures of Experts (HME) method of probability estimation, developed by Jordan [1], into an HMM(cid:173) based continuous speech recognition system. The resulting system can be thought of as a continuous-density HMM system, but instead of using gaussian mixtures, the HME system employs a large set of hierarchically organized but relatively small neural networks to perform the probability density estimation. The hierarchical structure is reminiscent of a decision tree except for two important differences: each "expert" or neural net performs a "soft" decision rather than a hard decision, and, unlike ordinary decision trees, the parameters of all the neural nets in the HME are automatically trainable using the EM algorithm. We report results on the ARPA 5,OOO-word and 4O,OOO-word Wall Street Journal corpus using HME models.

The current state-of-the-art in large-vocabulary, continuous speech recognition is based on the use of hidden Markov models (HMM). In an attempt to improve over HMM performance, the authors developed a hybrid system that combines the advantages of neural networks and HMM using a multiple hypothesis (or N-best) paradigm. The connectionist component of the system, the segmental neural net (SNN), models all the frames of a phonetic segment simultaneously, thus overcoming the well-known conditional-independence limitation of the HMM. They describe the hybrid system and discuss various aspects of SNN modeling, including network architectures, training algorithms and context modeling. Finally, they evaluate the hybrid system by performing several speaker-independent experiments with the DARPA Resource Management (RM) corpus, and demonstrate that the hybrid system shows a consistent improvement in performance over the baseline HMM system.

Up till recently, state-of-the-art, large vocabulary, continuous speech recognition (CSR) had employed hidden Markov modeling (HMM) to model speech sounds. In an attempt to improve over HMM we developed a hybrid system that integrates HMM technology with neural networks. We present the concept of a Segmental Neural Net (SNN) for phonetic modeling in CSR. By taking into account all the frames of a phonetic segment simultaneously, the SNN overcomes the well-known conditional-independence limitation of HMMs. We have developed a novel hybrid SNN/HMM system that combines the advantages of SNNs and HMMs using a multiple hypothesis (or N-best) paradigm. In this system, we generate likely phonetic segmentations from the HMM N-best list of word sequences, which are scored by the SNN. The HMM and SNN scores are then combined to optimize performance. In several speaker-independent, 1000-word CSR tests, the error rate for the hybrid system dropped 20% from that of a state-of-the-art HMM system alone.

The authors present the concept of a segmental neural net (SNN) for phonetic modeling in continuous speech recognition (CSR) and demonstrate how this can be used with a multiple hypothesis (or N-Best) paradigm to combine different CSR systems. In particular, the authors developed a system that combines the SNN with a hidden Markov model (HMM) system. In a speaker-independent, 1000-word CSR test using a word-pair grammar, the error rate for the hybrid system dropped 25% from that of a state-of-the-art HMM system alone. By taking into account all the frames of a phonetic segment simultaneously, the SNN overcomes the well-known conditional-independence limitation of HMMs. The hybrid SNN/HMM system generates likely phonetic segmentations from the HMM N-best list, which are scored by the SNN. The HMM and SNN scores are then combined to optimize performance.>

The authors describe four different ways in which they used the N-Best paradigm within the BYBLOS system. The most obvious use is for the efficient integration of speech recognition with a linguistic natural language understanding module. However, the authors have extended this principle to several other acoustic knowledge sources. They also describe a simple and efficient means for investigating and incorporating arbitrary knowledge sources. The N-Best hypotheses are used to provide close alternatives for discriminative training. Finally, the authors have developed a simple technique that allows them to optimize several weights and parameters within a system in a way that directly minimizes word error rate. Examples of each of these uses within the BYBLOS system are described.>

The authors report on the detection of new words for the speaker-dependent and speaker-independent paradigms. A useful operating point in a speaker-dependent paradigm is defined at 71% detection rate and 1% false alarm rate. The authors present a novel technique for obtaining a phonetic transcription for a new word, which is needed to add the new word to the system. The technique utilizes DECtalk's text-to-sound rules to obtain an initial phonetic transcription for the new word. Since these text-to-sound rules are imperfect, a probabilistic transformation technique is used that produces a phonetic pronunciation network of all possible pronunciations given DECtalk's transcription. The network is used to constrain a phonetic recognition process that results in an improved phonetic transcription for the new word. The resulting transcriptions are sufficient for speech recognition purposes.>

The authors introduce a technique that greatly speeds up expensive time-synchronous beam searches in speech recognition. The algorithm is called the forward-backward search and is mathematically related to the Baum-Welch forward-backward training algorithm. It uses a simplified forward pass followed by a detailed backward search. The information stored in the forward pass is used to decrease the computation in the backward pass by a large factor. An increase in speed of a factor of 40 with no increase in search errors was observed. The authors also describe how they have incorporated this algorithm into a real-time speaker-independent spoken language understanding system. One version of this is based on the 1000 word Resource Management vocabulary and is directed by a statistical class grammar. Another version has been incorporated into a military transportation planning application called DART (Dynamic Analysis Re-planning Tool).>

A paradigm for speaker-independent (SI) training of hidden Markov models (HMMs) is presented for continuous speech recognition. The method uses a large amount of speech from a few speakers instead of the traditional practice of using a little speech from many speakers. Furthermore, it avoids the common practice of pooling speech data from many speakers prior to training a SI model. The approach has been tested using the BBN Byblos system on the DARPA Resource Management corpus under standard test conditions. With only 12 speakers for training the SI models, recognition performance. comparable to that reported for other systems using 109 training speakers has been achieved. Besides surprisingly good recognition performance, this method offers many practical advantages that have implications for the way speech corpora, are designed and used for training SI models.>

The authors introduce a new, more efficient algorithm, the word-dependent N-best algorithm, for finding multiple sentence hypotheses. The proposed algorithm is based on the assumption that the beginning time of a word depends only on the preceding word. The authors compare this algorithm with two other algorithms for finding the N-best hypotheses: the exact sentence-dependent method and a computationally efficient lattice N-best method. Although the word-dependent algorithm is computationally much less expensive than the exact algorithm, it appears to result in the same accuracy. The lattice method, which is still more efficient, has a significantly higher error rate. It is demonstrated that algorithms that use Viterbi scoring have significantly higher error rates than those that use total likelihood scoring.>

A preliminary investigation of techniques that automatically detect when a speaker has used a word that is not in the vocabulary of a continuous-speech recognition system is described. A technique that uses a general model for the acoustics of any word to recognize the existence of new words is developed. Using this general word model, the correct detection of new words versus the false alarm rate is measured. Experiments were run using the DARPA 1000-word Resource Management Database indicate a detection rate for new words of 74% with a false alarm rate of 3.4%.>

A technique for using the speech of multiple reference speakers as a basis for speaker adaptation in large-vocabulary continuous-speech recognition is introduced. In contrast to other methods that use a pooled reference model, this technique normalizes the training speech from multiple reference speakers to a single common feature space before pooling it. The normalized and pooled speech is then treated as if it came from a single reference speaker for training the reference hidden Markov model (HMM). The usual probabilistic spectrum transformation can be applied to the reference HMM to model a new speaker. Preliminary experimental results are reported from applying this approach to over 100 reference speakers from the speaker-independent portion of the DARPA 1000-Word Resource Management Database.>

A search algorithm that provides a simple, clean, and efficient interface between the speech and natural language components of a spoken language system is introduced. The N-best algorithm is a time-synchronous Viterbi-style beam search procedure that is guaranteed to find the N most likely whole sentence alternatives that are within a given beam of the most likely sentence. The computation is linear with the length of the utterance, and faster than linear in N. When used together with a first-order statistical grammar, the correct sentence is usually within the first few sentence choices. The output of the algorithm, which is an ordered set of sentence hypotheses with acoustic and language model scores can easily be processed by natural language knowledge sources without the huge expansion of the search space that would be needed to include all possible knowledge sources in a top-down search. In experiments using a first-order statistical language model, the average rank of the correct answer was 1.8 and was within the first 24 choices 99% of the time.>

Several attempts to improve recognition accuracy with the use of supervised clustering techniques are described. These techniques modify the distance metric and/or the clustering procedure in a discrete hidden Markov model recognition system in an attempt to improve phonetic modeling. Three techniques considered are linear discriminant analysis, a hierarchical supervised vector quantization technique, and Kohonen's LVQ2 technique. All experiments were performed on the DARPA resource management speech corpus using the BBN BYBLOS system. Even though the techniques improved the phonetic recognition capability of the vector quantization, the overall word and sentence recognition accuracy did not improve.>

We introduce a new technique for using the speech of multiple reference speakers as a basis for speaker adaptation in large vocabulary continuous speech recognition. In contrast to other methods that use a pooled reference model, this technique normalizes the training speech from multiple reference speakers to a single common feature space before pooling it. The normalized and pooled speech can then be treated as if it came from a single reference speaker for training the reference hidden Markov model (HMM). Our usual prohabilistic spectrum transformation can be applied to the reference HMM to model a new (target) speaker. In this paper, we describe our baseline (single reference speaker) speakeradaptation system and give current performance results from a recent formal evaluation of the system. We also describe our proposal for adapting from multiple reference speakers and report on recent preliminary experimental results in support of the proposed technique. 1

The authors present several techniques to improve an algorithm presented last year for speaker-adaptive training in continuous speech recognition. The previous method uses a transformation matrix to modify the hidden Markov model (HMM) parameters of a prechosen prototype speaker to model a target speaker. To estimate the transformation matrix, it aligns a set of target speech with the same set of speech uttered by the prototype speaker using dynamic time warping. The authors focus on improving the previous method in the modeling of the spectral differences between two speakers, and the accuracy of the alignment. To improve the modeling of the spectral differences, they implemented a phoneme-dependent mapping procedure which transforms the prototype HMMs to the estimated target HMMs using a set of phoneme-dependent matrices. To improve the alignment, the authors developed a modeling of the silence, a linear duration normalization, and an iterative normalization procedure. They tested the new methods in the standard DARPA database with a grammar of perplexity 60. The performance shows a 30% word-error reduction compared to the previous algorithm.>

The authors consider the classification capabilities of feedforward two-layer neural nets with a single hidden layer and having threshold units only; that is they consider the type of decision regions that two-layer nets are capable of forming in the input space. It had been asserted previously that such nets are capable of forming only convex decision regions or nonconvex but connected regions. The authors show that two-layer nets are capable of forming disconnected decision regions as well. In addition to giving examples of the phenomena, they explain why and how disconnected decision regions are formed. They also derive an expression for the number of cells in the input space that are to be grouped together to form the decision regions. This expression can be useful in deciding how many nodes to have in the first layer. The results have bearing on neural networks where the nonlinear elements are smooth (sigmoid) functions rather than threshold functions.>

Three methods for smoothing discrete probability functions in discrete hidden Markov models for large-vocabulary continuous-speech recognition are presented. The smoothing is based on deriving a probabilistic co-occurrence matrix between the different vector-quantized spectra. Each estimated probability density is then multiplied by this matrix, ensuring that none of the probabilities are severely underestimated due to lack of training data. Experimental results show a 20-30% reduction in error rate when this smoothing is used. A word error rate of 3.0% is achieved with the DARPA 1000-word continuous speech recognition database and a word-pair grammar with a perplexity of 60.>

A novel text-dependent probabilistic spectral mapping method is presented for rapid speaker adaptation. The algorithm has been tested on the DARPA 1000-word resource management database with a grammar perplexity of 60. It results in significant better performance than the previous algorithms, and also provides recognition performance which is less than two times the word error rate for speaker-dependent training, using two minutes of adaptation speech.>

The system was trained in a speaker dependent mode on 28 minutes of speech from each of 8 speakers, and was tested on independent test material for each speaker. The system was tested with three artificial grammars spanning a broad perplexity range. The average performance of the system measured in percent word error was: 1.4% for a pattern grammar of perplexity 9, 7.5% for a word-pair grammar of perplexity 62, and 32.4% for a null grammar of perplexity 1000.>

In this paper, we describe BYBLOS, the BBN continuous speech recognition system. The system, designed for large vocabulary applications, integrates acoustic, phonetic, lexical, and linguistic knowledge sources to achieve high recognition performance. The basic approach, as described in previous papers [1, 2], makes extensive use of robust context-dependent models of phonetic coarticulation using Hidden Markov Models (HMM). We describe the components of the BYBLOS system, including: signal processing frontend, dictionary, phonetic model training system, word model generator, grammar and decoder. In recognition experiments, we demonstrate consistently high word recognition performance on continuous speech across: speakers, task domains, and grammars of varying complexity. In speaker-dependent mode, where 15 minutes of speech is required for training to a speaker, 98.5% word accuracy has been achieved in continuous speech for a 350-word task, using grammars with perplexity ranging from 30 to 60. With only 15 seconds of training speech we demonstrate performance of 97% using a grammar.

This paper presents research into the use of large-scale parallelism for a continuous speech recognition algorithm. The algorithm, developed for the BBN Byblos system [1], uses context dependent Hidden-Markov models to achieve high recognition accuracy. The multiprocessor used in the research, the BBN ButterflyTMParallel Processor, is a shared memory, MIMD machine. The algorithm was implemented using the Uniform System software methodology, a system that simplifies parallel programming without sacrificing efficiency. The algorithm is described, highlighting those portions critical to an efficient parallel implementation. Some of the problems encountered in trying to improve efficiency are presented as well as the solutions to those problems. The algorithm is shown to achieve 79% processor utilization on a 97-node Butterfly Parallel Processor. This is equivalent to a speedup by a factor of 77 over a single processor benchmark.

This paper deals with rapid speaker adaptation for speech recognition. We introduce a new algorithm that transforms hidden Markov models of speech derived from one "prototype" speaker so that they model the speech of a new speaker. The Speaker normalization is accomplished by a probabilistic spectral mapping from one speaker to another. For a 350 word task with a grammar and using only 15 seconds of speech for normalization, the recognition accuracy is 97% averaged over 6 speakers. This accuracy would normally require over 5 minutes of speaker dependent training. We derive the probabilistic spectral transformation of HMMs, describe an algorithm to estimate the transformation, and present recognition results.

This paper describes the results of our work in designing a system for large-vocabulary word recognition of continuous speech. We generalize the use of context-dependent Hidden Markov Models (HMM) of phonemes to take into account word-dependent coarticulatory effects, Robustness is assured by smoothing the detailed word-dependent models with less detailed but more robust models. We describe training and recognition algorithms for HMMs of phonemes-in-context. On a task with a 334-word vocabulary and no grammar (i.e., a branching factor of 334), in speaker-dependent mode, we show an average reduction in word error rate from 24% using context-independent phoneme models, to 10% when using robust context-dependent phoneme models.

We present a method for word recognition with input speech transduced simultaneously by several sensors in high levels of broadband acoustic background noise. In prior work on single-input multisensor systems, limited success in machine recognition was achieved by linearly combining multiple sensor signals to yield a robust estimate of the speech signal in the presence of noise. In this paper, we demonstrate that improved recognition results are obtained by using all available sensor signals jointly as a vector, which preserves information from all sensors, as input to the decision process. We report on multisensor configurations using close-talking pressure-gradient microphones and accelerometers placed at the throat and nose of the speaker. The recognition error rates obtained by using the joint output vector are 45% lower than the error rates obtained with the best constituent sensor in the multisensor system; single-input multisensor systems, on the other hand, produce error rates that are about equal to the error rates obtained with the best constituent sensor.

In this paper, we report the results of isolated-word speech recognition tests performed with the Verbex 4000 recognizer on speech data collected 1) in 95 dB and 115 dB SPL broad-band acoustic noise typical in a fighter aircraft cockpit and 2) using several single sensors and two-sensor configurations that we reported in ICASSP-85. The two-sensor systems and the gradient microphones we tested produced about the same recognition performance in 95 dB noise. In 115 dB noise, the recognizer failed to train for all but the accelerometer, which produced nearly constant performance in both noise levels. Also, we demonstrate the feasibility of a new speech recognition methodology that uses a parallel system of multiple input signals transduced simultaneously using different sensors. In selected phonetic discrimination tests involving a feature-based approach, a parallel-input multisensor system reduced the discrimination errors to between one-half and one-twelfth of the number produced by a gradient microphone alone.

In this paper, we examine several methods for text-independent speaker identification of telephone speech with limited duration data, The issue addressed is the assessment of channel characteristics, especially linear aspects, and methods for improving speaker identification performance when the speaker to be identified is on a different telephone channel than that data used for training. We show experimental evidence illustrating the cross-channel problem and also show that the direct approach, of using simple channel-invariant features, can discard much speaker dependent information. The methods we have found to be most effective rely on the training process to incorporate channel variability.

This paper describes the results of our work in designing a system for phonetic recognition of unrestricted continuous speech. We describe several algorithms used to recognize phonemes using context-dependent Hidden Markov Models of the phonemes. We present results for several variations of the parameters of the algorithms. In addition, we propose a technique that makes it possible to integrate traditional acoustic-phonetic features into a hidden Markov process. The categorical decisions usually associated with heuristic acoustic-phonetic algorithms are replaced by automated training techniques and global search strategies. The combination of general spectral information and specific acoustic-phonetic features is shown to result in more accurate phonetic recognition than either representation by itself.

This paper describes a syntax-directed real-time connected-word recognition system and compares the performance of different recognition algorithms. This is a speaker dependent system based on whole word template matching, and the recognition is done using a one-pass dynamic time-warping algorithm. The symmetric local constraints of Sakoe and Chiba are used for connected word recognition, and with proper normalization of the accumulated distances, result in better performance than the Itakura local constraints. By allowing a noise template to start and end the sentences during the recognition process, most of the errors due to a bad endpoint detection are eliminated. When memory and computational resources are limited, vector quantization allows more templates for each word in the dictionary, and as a consequence, the recognition performance increases. By carefully implementing the algorithm on standard hardware (VAX 11/780 and FPS AP-120B), real-time recognition is achieved for vocabularies of up to 100 templates, or up to 250 templates if vector quantization is used.

In this paper, we examine several methods for text-independent speaker identification and discuss the problem of achieving robust performance. In a previous paper [1], we have described a radio-channel database that contains highly variable speech data of poor quality. Here, we describe several experiments with the radio-channel database leading to the development of robust features and methods. These experiments show that robustness to one degradation may not be sufficient to improve speaker identification accuracy. A robust feature set and modeling and classification method should mitigate the effects of many of the degradations in the data.

This paper discusses the use of the Hidden Markov Model (HMM) in phonetic recognition. In particular, we present improvements that deal with the problems of modeling the effect of phonetic context and the problem of robust pdf estimation. The effect of phonetic context is taken into account by conditioning the probability density functions (pdfs) of the acoustic parameters on the adjacent phonemes, only to the extent that there are sufficient tokens of the phoneme in that context. This partial conditioning is achieved by combining the conditioned and unconditioned pdfs models with weights that depend on the confidence in each pdf estimate. This combination is shown to result in better performance than either model by itself. We also show that it is possible to obtain the computational advantages of using discrete probability densities without the usual requirement for large amounts of training data.

In this paper we investigate several methods for reducing the bit rate of a segment vocoder [1] by 35% to 150 b/s. In the original vocoder we used a random sample of vectors as a set of templates for vector quantization. We demonstrate in this paper that this random quantizer is near-optimal by comparing it with quantizers that use clustering algorithms for quantizing speech segments. The reduction of the bit rate of the segment vocoder was achieved primarily by using a segment network, i.e., not all segment templates are allowed to follow a given segment template. The spectral continuity of speech is used to determine the subset of templates, that can be used to quantize an input segment. To achieve the low rate of 150 b/s, we also reduced the bit rate for coding pitch, gain, and segment duration. Finally, we present the bit allocation used for transmitting speech at 150 b/s as a single speaker segment vocoder.

In this paper we discuss several algorithms that can be used to reduce the transmission rate for LPC vocoded speech to around 300 to 400 b/s, with only a modest degradation in speech quality relative to that of fixed-rate 2400 b/s LPC vocoders. We limit the discussion to vocoders that transmit information for single frames (as opposed to whole segments of speech). We start with vector quantization, which reduces the bit rate to around 800 b/s accompanied by a significant but tolerable loss in quality relative to a typical fixed-rate 2400 b/s vocoder. Then we reduce the frame rate using one of two techniques: Fixed-Rate Transmission with Variable Interpolation, or Optimal Variable-Frame-Rate Transmission. We also reduce the data rate necessary for the source parameters (pitch, voicing, gain) from 400 b/s to about 100 b/s by taking advantage of their statistical dependence on the spectrum and some perceptual factors. The final result at 300 b/s has a quality comparable to that of the fixed-rate 800 b/s vector quantization vocoder. At 400 b/s, the quality is, in many respects, better than that of the 800 b/s vocoder and comparable to the 2400 b/s LPC vocoder.

In this paper, we present the preliminary performance of four methods for text-independent speaker identification using speech transmitted over radio channels. In a previous paper [1], we showed that for both laboratory-quality and simulated noisy-channel data in a single-session paradigm, new probabilistic classifiers yielded performance superior to that of a minimum distance classifier. We have recently compiled a speech database consisting of speech transmissions over a radio-channel. The lower quality and higher variability of this database differ markedly from the laboratory-quality databases often used in speech processing research. We present preliminary results with the same four methods of text-independent speaker identification using the radio-channel database with several experimental paradigms including multi-session paradigms. These results show that the probabilistic methods perform significantly better than a minimum-distance classifier for the multi-session paradigm.

We present a method that reduces the bit rate of a low rate LPC vocoder by modelling the sequence of quantized spectra by a Markov chain. To minimize the bit rate, one would want to use a high-order chain. Unfortunately, a high-order chain would require an inordinate amount of data for training. We describe in this paper the use of a variable order Markov chain that maximizes the effective use of a given amount of speech data. To reduce the number of states of a high-order chain, we define an equivalence relation on the states, i.e., "similar" states are grouped together in an equivalence class and a single conditional distribution is associated with the equivalence class. We introduce two equivalence relations. In the first, called variable order Markov chain, the equivalence classes represent the most probable states of any order. In the second method, called variable resolution, the equivalence class is obtained by decreasing the quantization accuracy in representing a spectrum that belongs to a more remote past. For an LPC vocoder with 64 possible spectra (using 6-bit vector quantization), the second method is superior to the first and decreases the entropy from 6 bits to 4 bits per spectrum with 256 equivalence classes.

We introduce a new method for very-low-rate vocoding that the input speech as a sequence of variable-length segments. A segment is a by a spectrum of frames, where each frame is represented by a spectrum, pitch and gain. We use an automatic segmentation algorithm to obtain segments with an average duration comparable to that of a phoneme. A segment is quantized as a single block. The distance measure used for quantization incooporates the appropriate time alignment of two segments. We employ a computationally efficient metric that does not use the usual dynamic programming time warping. Two basic vocoders using the above approach of block quantization have been used to transmit intelligible speech at 200 b/s.

Most text-independent speaker identification methods to date depend on the use of some distance metric for classification. In this paper we develop the use of probability density function (pdf) estimation for text-independent speaker identification. We compare the performance of two parametric and one non-parametric pdf estimation methods to one distance classification method that uses the Mahalanobis distance. Under all conditions tested, the pdf estimation methods performed substantially better than the Mahalanobis distance method. The best method is a non-parametric pdf estimation method.

We review the variable frame rate (VFR) transmission methodology that we developed, implemented, and tested during the period 1973-1978 for efficiently transmitting LPC vocoder parameters extracted from the input speech at a fixed frame rate. In the VFR method, parameters are transmitted only when their values have changed sufficiently over the interval since their preceding transmission. We explored two distinct approaches to automatic implementation of the VFR method. The first approach bases the transmission decisions on comparisons of the parameter values of the present frame and the last transmitted frame. The second approach, which is based on a functional perceptual model of speech, compares the parameter values of all the frames that lie in the interval between the present frame and the last transmitted frame against a linear model of parameter variation over that interval. The application of VFR transmission to the design of narrow-band LPC speech coders with average bit rates of 2000-2400 bits/s is also considered. The transmission decisions are made separately for the three sets of LPC parameters, pitch, gain, and spectral parameters, using separate VFR schemes. A formal subjective spccch quality test of six selected LPC coders is described, and the results are presented and analyzed in detail. It is shown that a 2075 bit/s VFR coder produces speech quality equal to or better than that of a 5700 bit/s fixed frame rate coder.

We report on the initial development of a phonetic vocoder operating at 100 b/s. With each phoneme, the vocoder transmits the duration and a single pitch value. The synthesizer uses a large inventory of diphone templates to synthesize a desired phoneme string. To determine a phoneme string from input speech, the analyzer takes into account the synthesis model by using the same inventory of diphone templates, augmented by additional diphone templates to account for alternate pronunciations. The phoneme string is chosen to minimize the difference between the diphone templates and the input speech according to a distance measure.

This paper describes a method for enhancing speech corrupted by broadband noise. The method is based on the spectral noise subtraction method. The original method entails subtracting an estimate of the noise power spectrum from the speech power spectrum, setting negative differences to zero, recombining the new power spectrum with the original phase, and then reconstructing the time waveform. While this method reduces the broadband noise, it also usually introduces an annoying "musical noise". We have devised a method that eliminates this "musical noise" while further reducing the background noise. The method consists in subtracting an overestimate of the noise power spectrum, and preventing the resultant spectral components from going below a preset minimum level (spectral floor). The method can automatically adapt to a wide range of signal-to-noise ratios, as long as a reasonable estimate of the noise spectrum can be obtained. Extensive listening tests were performed to determine the quality and intelligibility of speech enhanced by our method. Listeners unanimously preferred the quality of the processed speech. Also, for an input signal-to-noise ratio of 5 dB, there was no loss of intelligibility associated with the enhancement technique.

We report on the synthesis of speech in the context of a phonetic vocoder operating at 100 b/s. With each phoneme, the vocoder transmits the duration and a single pitch value. The synthesizer uses a large inventory of diphone "models" to synthesize a desired phoneme string. The diphone inventory has been selected to differentiate between prevocalic and postvocalic allophones of sonorants, to account for changes in vowel color conditioned by postvocalic liquids, to allow exact specification of voice onset time, and to permit synthesis of glottal stops alveolar flaps and syllabic consonants. The diphones are extracted from carefully constructed short utterances and are stored as a sequence of LPC parameters. During synthesis, the requisite diphone models are time-warped, abutted and smoothed to produce a complete sequence of LPC parameters that are used in the synthesis. The algorithms used are described and compared with more conventional methods. Examples of the synthesized speech will be played.

This paper presents an excitation source model for speech compression and synthesis, which allows for a degree of voicing by mixing voiced (pulse) and unvoiced (noise) excitations in a frequency-selective manner. The mix is achieved by dividing the speech spectrum into two regions, with the pulse source exciting the low-frequency region and the noise source exciting the high-frequency region. A parameter Fcdetermines the degree of voicing by specifying the cut-off frequency between the voiced and unvoiced regions. For speech compression applications, Fccan be extracted automatically from the speech spectrum and transmitted. Experiments using the new model indicate its power in synthesizing natural sounding voiced fricatives, and in largely eliminating the "buzzy" quality of vocoded speech. A functional definition of buzziness and naturalness is given in terms of the model.

While gathering acoustic data for the acoustic-phonetic analysis of speech, it is necessary to consider many different sounds in varying phonetic environments to assure that the results are statistically significant. In order to reduce the amount of time required to test hypotheses, a facility has been developed which provides an interactive environment for performing a wide variety of acoustic-phonetic experiments on a large data base of continuous speech. Using this facility, one can formulate an experiment, run it on selected portions (or all) of the data base, and display or tabulate the results in a meaningful way. Another experiment may then be run based on the results. CPU time required to run an experiment on the entire data base is between 5 and 20 seconds, depending on the complexity of the experiment. Due to the ease of interactions, formulating or revising an experiment, running it, and displaying the results normally takes less than 5 minutes. This facility has been used in combination with a data base of 69 hand-labeled sentences to develop algorithms for acoustic-phonetic segmentation and labeling in a speech understanding system. Several examples of its use and the results obtained will be presented.

This paper describes the acoustic-phonetic analysis of continuous speech in a complete speech understanding system. The system accepts various parameters derived from the digital waveform and short-time spectra, and produces a segment lattice where segments can have overlapping boundaries and the description of segments is a list of labels. Acoustic-Phonetic as well as phonological knowledge of English is employed extensively in labeling the segments. Each label also has associated with it a score, reflecting the confidence in its identity. A description of the acoustic-phonetic analyzer, as well as statistics related to its performance will be presented in detail.