Most of the parameters in large vocabulary models are used in embedding layer to map categorical features to vectors and in softmax layer for classification weights. This is a bottleneck in memory constraint on-device training applications like federated learning and on-device inference applications like automatic speech recognition (ASR). One way of compressing the embedding and softmax layers is to substitute larger units such as words with smaller sub-units such as characters. However, often the sub-unit models perform poorly compared to the larger unit models. We propose WEST, an algorithm for encoding categorical features and output classes with a sequence of random or domain dependent sub-units and demonstrate that this transduction can lead to significant compression without compromising performance.

The acoustic models in state-of-the-art speech recognition systems are based on phones in context that are represented by hidden Markov models. This modeling approach may be limited in that it is hard to incorporate long-span acoustic context. Exemplar-based approaches are an attractive alter-native, in particular if massive data and computational power are available. Yet, most of the data at Google are unsupervised and noisy. This paper investigates an exemplar-based approach under this yet not well understood data regime. A log-linear rescoring framework is used to combine the exemplar-based features on the word level with the first-pass model. This approach guarantees at least baseline performance and focuses on the refined modeling of words with sufficient data. Experimental results for the Voice Search and the YouTube tasks are presented.

This paper describes a data-driven technique for optimizing the acoustic models for speech recognition systems that target commercial applications over telephones. Frame-averaged foreground log-likelihoods (foreground scores) correlate to recognition errors. These scores are used together with gender to optimize data weighting for the acoustic model. This process is interpreted as increasing the priors and associated parameters for poorly modeled data. The score-based optimization leads to about 7% fewer semantic errors on a live evaluation set collected after the last data used to estimate the acoustic model.

This paper focuses on the recognition of noisy speech. We show that the decoding of a noisy speech waveform can be facilitated if the recognizer has explicit knowledge of where it should hypothesize speech phones, and where it should map the acoustics to non-speech phones. We build a speech/non-speech detector and use its output as an additional front-end feature. We show that by appropriately weighting the contribution of this feature in the decoder and by modifying the acoustic models accordingly, we can penalize speech/non-speech confusions and consequently reduce the recognition error rate. This approach gives a 12% overall error rate reduction on a wide variety of recognition tasks and noise characteristics without degrading performance on clean test data. A simple extension of the approach boosts recognition improvements on noisy test sets to 14% overall.

Many speech recognition systems that provide over-the phone services, e.g. name dialers, stock quote providers, location finders, rely on the accurate recognition of proper names. For this to happen, the systems need to know how their users will pronounce these words. However, predicting the pronunciation of a proper name is a notoriously difficult problem as it depends on the origin of the name, the linguistic background of the speaker, and other cultural and sociological factors, in addition of course to the word spelling. In this paper, we describe a data-driven method that learns proper name pronunciations from audio samples of these words. The algorithm relies on the machinery of a general purpose speech recognizer to find the phone sequence that best matches the sample speech waveforms. In addition, it incorporates linguistic knowledge automatically acquired from a pronunciation dictionary to ensure that the learned pronunciations are "reasonable" from a linguistic viewpoint. We show on a corporate name dialing database that the proposed algorithm reduces the call routing error rate by 40% compared to a reference letter-to-phone pronunciation engine.

We describe an algorithm to learn word pronunciations from acoustic data. The algorithm jointly optimizes the pronunciation of a word using (a) the acoustic match of this pronunciation to the observed data, and (b) how “linguistically reasonable ” the pronunciation is. Variations of word pronunciations in the recognition dictionary (which was created by linguists), are used to train a model of whether new hypothesized pronunciations are reasonable or not. The algorithm is well-suited for proper name pronunciation learning. Experiments on a corporate name dialing database show 40 % error rate reduction with respect to a letter-to-phone pronunciation engine. 1.

This paper describes a new approach to acoustic modeling for large vocabulary continuous speech recognition (LVCSR) systems. Each phone is modeled with a large Gaussian mixture model (GMM) whose context-dependent mixture weights are estimated with a sentence-level discriminative training criterion. The estimation problem is cast in a neural network framework, which enables the incorporation of the appropriate constraints on the mixture weight vectors, and allows a straight-forward training procedure, based on steepest descent. Experiments conducted on the Callhome-English and Switchboard databases show a significant improvement of the acoustic model performance, and a somewhat lesser improvement with the combined acoustic and language models.

The paper presents the results of a series of experiments to assess the capability of a multi-microphone sub-band adaptive (MMSBA) signal processing scheme for improving the intelligibility of speech corrupted with noise recorded in an automobile. The noise corrupted speech signals were presented to 15 normal hearing volunteer subjects at various SNRs, and numbers and distributions of sub-bands. The results of listening tests were analysed to determine: (i) The direction and statistical significance (DSS) of any effect of processing treatment on intelligibility. (ii) The DSS of any effect of processing treatment on perceived quality. (iii) The DSS of any effect of the numbers of sub-bands used in sub-band processing (SBP). (iv) The DSS of any effect of the spacing of sub-bands used in SBP. In the experimental cases considered, the MMSBA scheme employing diverse sub-band processing is shown to deliver a statistically significant improvement in terms of both speech intelligibility and perceived quality when compared with both the wide-band processed and the noisy unprocessed case.

This paper addresses the issue of closed-set text-independent speaker identification from samples of speech recorded over the telephone. It focuses on the effects of acoustic mismatches between training and testing data, and concentrates on two approaches: (1) extracting features that are robust against channel variations and (2) transforming the speaker models to compensate for channel effects. First, an experimental study shows that optimizing the front end processing of the speech signal can significantly improve speaker recognition performance. A new filterbank design is introduced to improve the robustness of the speech spectrum computation in the front-end unit. Next, a new feature based on spectral slopes is described. Its ability to discriminate between speakers is shown to be superior to that of the traditional cepstrum. This feature can be used alone or combined with the cepstrum. The second part of the paper presents two model transformation methods that further reduce channel effects. These methods make use of a locally collected stereo database to estimate a speaker-independent variance transformation for each speech feature used by the classifier. The transformations constructed on this stereo database can then be applied to speaker models derived from other databases. Combined, the methods developed in this paper resulted in a 38% relative improvement on the closed-set 30-s training 5-s testing condition of the NIST'95 Evaluation task, after cepstral mean removal.

Statistics of frame-level pitch have recently been used in speaker recognition systems with good results [1, 2, 3]. Although they convey useful long-term information about a speaker's distribution of f 0 values, such statistics fail to capture information about local dynamics in intonation that characterize an individual's speaking style. In this work, we take a first step toward capturing such suprasegmental patterns for automatic speaker verification. Specifically, we model the speaker's f 0 movements by fitting a piecewise linear model to the f 0 track to obtain a stylized f 0 contour. Parameters of the model are then used as statistical features for speaker verification. We report results on 1998 NIST speaker verification evaluation. Prosody modeling improves the verification performance of a cepstrum-based Gaussian mixture model system (as measured by a task-specific Bayes risk) by 10%. 1. INTRODUCTION Statistics of frame-level pitch have recently been shown to improve the perf...

In the context of automatic speaker recognition, we propose a model transformation technique that renders speaker models more robust to acoustic mismatches and to data scarcity by appropriately increasing their variances. We use a stereo database containing speech recorded simultaneously under different acoustic conditions to derive a synthetic variance distribution. This distribution is then used to modify the variances of other speaker models from other telephone databases. The technique is illustrated with experiments conducted on a locally collected database and on the NIST'95 and '96 subsets of the Switchboard corpus.

This paper studies the effects of handset distortion on telephone-based speaker recognition performance, resulting in the following observations: (1) the major factor in speaker recognition errors is whether the handset type (e.g., electret, carbon) is different across training and testing, not whether the telephone lines are mismatched, (2) the distribution of speaker recognition scores for true speakers is bimodal, with one mode dominated by matched handset tests and the other by mismatched handsets, (3) cohort-based normalization methods derive much of their performance gains from implicitly selecting cohorts trained with the same handset type as the claimant, and (4) utilizing a handset-dependent background model which is matched to the handset type of the claimant's training data sharpens and separates the true and false speaker score distributions. Results on the 1996 NIST Speaker Recognition Evaluation corpus show that using handset-matched background models reduces false acceptances (at a 10% miss rate) by more than 60% over previously reported (handset-independent) approaches.

This paper proposes a probabilistic framework to define and evaluate confidence measures for word recognition. We describe a novel method to combine different knowledge sources and estimate the confidence in a word hypothesis, via a neural network. We also propose a measure of the joint performance of the recognition and confidence systems. The definitions and algorithms are illustrated with results on the Switchboard Corpus.

Statistics of pitch have recently been used in speaker recognition systems with good results. The success of such systems depends on robust and accurate computation of pitch statistics in the presence of pitch tracking errors. In this work, we develop a statistical model of pitch that allows unbiased estimation of pitch statistics from pitch tracks which are subject to doubling and/or halving. We first argue by a simple correlation model and empirically demonstrate by QQ plots that "clean" pitch is distributed with a lognormal distribution rather than the often assumed normal distribution. Second, we present a probabilistic model for estimated pitch via a pitch tracker in the presence of doubling/halving, which leads to a mixture of three lognormal distributions with tied means and variances for a total of four free parameters. We use the obtained pitch statistics as features in speaker verification on the March 1996 NIST Speaker Recognition Evaluation data (subset of Switchboard) and ...

We show that the standard hypothesis scoring paradigm used in maximum-likelihood-based speech recognition systems is not optimal with regard to minimizing the word error rate, the commonly used performance metric in speech recognition. This can lead to sub-optimal performance, especially in high-error-rate environments where word error and sentence error are not necessarily monotonically related. To address this discrepancy, we developed a new algorithm that explicitly minimizes expected word error for recognition hypotheses. First, we approximate the posterior hypothesis probabilities using N-best lists. We then compute the expected word error for each hypothesis with respect to the posterior distribution, and choose the hypothesis with the lowest error. Experiments show improved recognition rates on two spontaneous speech corpora. 1. INTRODUCTION The standard selection criterion for speech recognition hypotheses aims at maximizing the posterior probability of a hypothesis W given t...

There has been much interest in the area of adaptation for improved speech recognition in the presence of mismatches between the training and testing conditions. We focus on transformation-based maximum-likelihood (ML) adaptation. Some of the important adaptation parameters include whether the adaptation is performed in the feature-space or model-space, and whether the adaptation is supervised or unsupervised. An additional parameter is the adaptation data. For example adaptation may be performed using an independent dataset or the test data itself. The latter is referred to as transcription-mode adaptation. We experimentally study the effect of these various parameters, and report on our findings.

This paper compares three techniques for recognizing continuous speech in the presence of additive car noise: (1) transforming the noisy acoustic features using a mapping algorithm, (2) adaptation of the hidden Markov models (HMMs), and (3) combination of mapping and adaptation. To make the signal processing robust to additive noise, we apply a technique called probabilistic optimum filtering. We show that at low signal-to-noise ratio (SNR) levels, compensating in the feature and model domains yields similar performance. We also show that adapting the HMMs with the mapped features produces the best performance. The algorithms were implemented using SRI's DECIPHER speech recognition system and were tested on the 1994 ARPA-sponsored CSR evaluation test spoke 10.

A new scoring algorithm has been developed for generating wordspotting hypotheses and their associated scores. This technique uses a large-vocabulary continuous speech recognition (LVCSR) system to generate the N-best answers along with their Viterbi alignments. The score for a putative hit is computed by summing the likelihoods for all hypotheses that contain the keyword normalized by dividing by the sum of all hypothesis likelihoods in the N-best list. Using a test set of conversational speech from Switchboard Credit Card conversations, we achieved an 81% figure of merit (FOM). Our word recognition error rate on this same test set is 54.7%.

We present a new mapping algorithm for speech recognition that relates the features of simultaneous recordings of clean and noisy speech. The model is a piecewise linear transformation applied to the noisy speech feature. The transformation is a set of multidimensional linear least-squares filters whose outputs are combined using a conditional Gaussian model. The algorithm was tested using SRI's DECIPHER speech recognition system. Experimental results show how the mapping is used to reduce recognition errors when the training and testing acoustic environments do not match.>

In this paper we explore the effectiveness of constructing telephone acoustic models using a high-quality speech corpus. Results are presented for several front-end signal processing and feature mapping techniques. The algorithms were tested using SRI's DECIPHER speech recognition system on several telephone corpora. The results show that (a) most of the performance loss when testing on telephone speech is due to the loss of information associated with the telephone bandwidth; (b) telephone-bandwidth systems trained with high-quality speech can outperform systems that are trained on telephone speech when tested on telephone speech; and (c) robust signal processing can allow speech recognizers to maintain performance when wide-bandwidth acoustic models are tested with telephone speech.>

This paper describes a method for utilizing the quasi-periodicity of speech in a minimum mean-square error (MMSE) estimation of the discrete Fourier transform (DFT) log-amplitude, either for speech enhancement or for noise-robust speech recognition. The estimator takes into account the periodicity by conditioning the estimate of voiced speech on the distance between the frequency of any given DFT coefficient and the nearest harmonic: if the DFT coefficient lies in the vicinity of a harmonic, the a priori probability distribution (PD) of its amplitude centers around higher values than if it lies halfway between two harmonics. Thus, knowing the pitch narrows down the a priori PD, improving the estimate. The DFT estimator is combined with a mixture model for the broadband spectral PD, so that correlations between distant frequencies are partially taken into account. The algorithm has been tested with computer-room noise using an MSE criterion for the spectral envelope, defined by Mel-scale filter-bank log energies, and in recognition experiments. The incorporation of correlations in the broadband spectrum improves recognition accuracy significantly; the periodicity conditioning reduces the MSE for voiced speech, but recognition accuracy is not improved because the overwhelming majority of errors occur in unvoiced speech.

We describe an approach for the estimation of acoustic phonetic models that will be used in a hidden Markov model (HMM) recognizer operating over the telephone. We explore two complementary techniques to developing telephone acoustic models. The first technique presents two new channel compensation algorithms. Experimental results on the Wall Street Journal corpus show no significant improvement over sentence-based cepstral-mean removal. The second technique uses an existing "high-quality" speech corpus to train acoustic models that are appropriate for the switchboard credit card task over long-distance telephone lines. Experimental results show that cross-database acoustic training yields performance similar to that of conventional task-dependent acoustic training.>

An estimation algorithm for noise robust speech recognition, the minimum mean log spectral distance (MMLSD), is presented. The estimation is matched to the recognizer by seeking to minimize the average distortion as measured by a Euclidean distance between filterbank log-energy vectors, approximating the weighted-cepstral distance used by the recognizer. The estimation is computed using a clean speech spectral probability distribution, estimated from a database, and a stationary, ARMA model for the noise. When trained on clean speech and tested with additive white noise at 10-dB SNR, the recognition accuracy with the MMLSD algorithm is comparable to that achieved with training the recognizer at the same constant 10-dB SNR. The algorithm is also highly efficient with a quasi-stationary environmental noise, recorded with a desktop microphone, and requires almost no tuning to differences between this noise and the computer-generated white noise.>

An estimation algorithm to improve the noise robustness of filterbank-based speech recognition systems is presented. The algorithm is based on a minimum mean square error (MMSE) estimation of the filter log-energies, introducing a significant improvement over related published algorithms by conditioning the estimate on the total frame energy. The algorithm was evaluated with DECIPHER, SRI's continuous-speech speaker-independent recognizer, on two types of noisy speech: a standard database with added white Gaussian noise, and recordings made in a noisy environment. With white noise the recognition accuracy obtained while training on clean speech and testing in noise approached that obtained with training and testing in noise. In the noisy environment, the estimation algorithm boosted the recognition system's performance with a table mounted microphone almost to the level achieved with a close talking microphone.>

A method for utilizing the quasi-periodicity of speech in a minimum-mean-square-error (MMSE) estimation of the DFT log-amplitude, either for speech enhancement or for noise-robust speech recognition, is described. The estimator takes into account the periodicity by conditioning the estimate of voiced speech on the distance between the frequency of any given DFT coefficient and the nearest harmonic. The DFT estimator is also made conditional on the broadband spectrum, so that the correlation between distant frequencies is partially taken into account. The algorithm has been tested with computer-room noise using an MSE criterion for the spectral envelope, defined by Mel-scale filterbank log-energies, and in recognition experiments. The MSE for voiced speech is reduced significantly by the periodicity-conditioning. Recognition accuracy is not improved because the overwhelming majority of errors occur in unvoiced speech.>

A large-vocabulary, continuous-speech system, called DECIPHER, which is based on a hidden Markov model (HMM) approach and is designed to achieve high word accuracy in a speaker-independent mode is described. The results of a series of experiments that test acoustic and phonological adaptation of the DECIPHER system to the pronunciations of a single speaker in a speaker-dependent task are presented. Estimating the probabilities of alternative pronunciations and speaker-dependent phonology are discussed.>

A spectral-estimation algorithm designed to improve the noise robustness of speech-recognition systems is presented and evaluated. The algorithm is tailored for filter-bank-based systems, where the estimation seeks to minimize the distortion as measured by the recognizer's distance metric. This minimization is achieved by modeling the speech distribution as consisting of clusters; the energies at different frequency channels are assumed to be uncorrelated within each cluster. The algorithm was tested with a continuous-speech, speaker-independent hidden Markov model (HMM) recognition system using the NIST Resource Management Task speech database. When trained on a clean speech database and tested with additive white Gaussian noise, the recognition accuracy with the new algorithm is comparable to that under the ideal condition of training and testing at constant SNR. When trained on clean speech and tested with a desktop microphone in a noisy environment, the error rate is only slightly higher than that with a close-talking microphone.>

SRI is developing a system that uses real time speech recognition to diag nose, evaluate and provide training in spoken English. The paper first describes the methods and results of a study of the feasibility of automati cally grading the performance of Japanese students when reading English aloud. Utterances recorded from Japanese speakers were independently rated by expert listeners. Speech grading software was developed from a speaker independent hidden-Markov-model speech recognition system. The auto matic grading procedure first aligned the speech with a model and then com pared the segments of the speech signal with models of those segments that have been developed from a database of speech from native speakers of English. The evaluation study showed that ratings of speech quality by experts are very reliable and automatic grades correlate well (r > 0.8) with those expert ratings. SRI is now extending this technology and integrating it in a spoken-language training system. This effort involves (1) porting SRI's DECIPHER speech recognition system to a microcomputer platform, and (2) extending the speech-evaluation software to more exactly diagnose a learner's pronuncia tion deficits and lead the learner through an appropriate regimen of exer cises.

A speaker-independent, continuous-speech, large-vocabulary speech recognition system, DECIPHER, has been developed. It provides state-of-the-art performance on the DARPA standard speaker-independent resource management training and testing materials. The approach is to integrate speech and linguistic knowledge into the HMM (hidden Markov model) framework. Performance improvements arising from detailed phonological modeling and from the incorporation of cross-word coarticulatory constraints are described. It is concluded that speech and linguistic knowledge sources can be used to improve the performance of HMM-based speech recognition systems provided that care is taken to incorporate these knowledge sources appropriately.>

An algorithm based on hidden Markov models is applied to the task of speaker-independent continuous-speech recognition for a vocabulary of 1000 words with no syntactic constraints. The signal is limited to 4000 Hz. Word models were built from three-state representations of phonetic units, concatenated according to entries in a lexicon. Performance as measured on DARPAs resource management database was 40% correct word recognition. It was found that the use of several different acoustic features and the use of word-specific phonetic modeling, where possible, improved system performance.>

This paper describes an alternative approach to lexical access in the CMU ANGEL speech recognition system. Using this approach, the asynchronous phonetic hypotheses generated by an acoustic-phonetics module are converted to a directed graph. This graph is compared to a pronunciation dictionary. Performance results for this approach and the original CMU approach are similar. An error analysis indicates promising directions for further work.

This paper describes a computational model that attempts to separate two simultaneous talkers. The goal of this model is to improve a speech recognition system's ability to recognize what each of the two talkers say. The model consists of the following stages: (1) an iterative dynamic programming algorithm to track the pitch period for each of the two talkers, (2) a Markov model to determine the characteristics (e.g. voiced-unvoiced) of each speaker's voice, (3) a recursive algorithm that uses both local periodicity information and local spectral continuity constraints to compute a spectral estimate of each talker, (4) a resynthesis algorithm to convert the spectral estimate of each talker into a speech waveform, and (5) a speaker-independent continuous-digit-recognition system that attempts to recognize what each of two talkers is saying. The system was trained and tested on a database of simultaneous digit strings spoken by a male and female talker. An evaluation of the different stages of this model is presented.

Any speech recognition system designed to function in environments with other sounds present must deal with extraneous sounds. A hierarchical separation and recognition system is proposed as a computational model of how the auditory system processes sounds. The GRASP system (for Grouping Research on Auditory Sound Processing) is a part of that overall framework for sound separation. GRASP is a computational model of the data-driven aspects of auditory separation. It uses the physical cues present in the acoustic signal (such as pitch and onsets) to decide how many sounds are present and of what each sound consists. Its initial use is to separate simultaneously spoken digits from different speakers.