This paper describes a robust feature extraction technique for continuous speech recognition. Central to the technique is the minimum variance distortionless response (MVDR) method of spectrum estimation. We consider incorporating perceptual information in two ways: 1) after the MVDR power spectrum is computed and 2) directly during the MVDR spectrum estimation. We show that incorporating perceptual information directly into the spectrum estimation improves both robustness and computational efficiency significantly. We analyze the class separability and speaker variability properties of the features using a Fisher linear discriminant measure and show that these features provide better class separability and better suppression of speaker-dependent information than the widely used mel frequency cepstral coefficient (MFCC) features. We evaluate the technique on four different tasks: an in-car speech recognition task, the Aurora-2 matched task, the Wall Street Journal (WSJ) task, and the Switchboard task. The new feature extraction technique gives lower word-error-rates than the MFCC and perceptual linear prediction (PLP) feature extraction techniques in most cases. Statistical significance tests reveal that the improvement is most significant in high noise conditions. The technique thus provides improved robustness to noise without sacrificing performance in clean conditions

We introduce a class of Gaussian mixture models (GMMs) in which the covariances or the precisions (inverse covariances) are restricted to lie in subspaces spanned by rank-one symmetric matrices. The rank-one basis are shared between the Gaussians according to a sharing structure. We describe an algorithm for estimating the parameters of the GMM in a maximum likelihood framework given a sharing structure. We employ these models for modeling the observations in the hidden-states of a hidden Markov model based speech recognition system. We show that this class of models provide improvement in accuracy and computational efficiency over well-known covariance modeling techniques such as classical factor analysis, shared factor analysis and maximum likelihood linear transformation based models which are special instances of this class of models. We also investigate different sharing mechanisms. We show that for the same number of parameters, modeling precisions leads to better performance when compared to modeling covariances. Modeling precisions also gives a distinct advantage in computational and memory requirements

In this paper, we consider the problem of quantifying the amount of information contained in a set of features, to discriminate between various classes. We explore these ideas in the context of a speech recognition system, where an important classification sub-problem is to predict the phonetic class, given an observed acoustic feature vector. The connection between information content and speech recognition system performance is first explored in the context of various feature extraction schemes used in speech recognition applications. Subsequently, the idea of optimizing the information content to improve recognition accuracy is generalized to a linear projection of the underlying features. We show that several prior methods to compute linear transformations (such as linear/heteroscedastic discriminant analysis) can be interpreted in this general framework of maximizing the information content. Subsequently, we extend this reasoning and propose a new objective function to maximize a penalized mutual information (pMI) measure. This objective function is seen to be very well correlated with the word error rate of the final system. Finally experimental results are provided that show that the proposed pMI projection consistently outperforms other methods for a variety of cases, leading to relative improvements in the word error rate of 5%-16% over earlier methods.

We consider the problem of enrollment for low-resource speech recognition systems designed for noisy environments. Noise robustness concerns, memory and computational constraints, along with the use of compact acoustic models for fast Gaussian computation, make adaptation especially challenging. We derive a maximum a posteriori (MAP) algorithm especially designed for the fast off-line adaptation of these compact acoustic models. It requires less computation and memory than standard feature-space maximum likelihood linear regression (FMLLR) which is another technique well suited for compact acoustic models. In our experiments of speaker enrollment for speech recognition in the car, we present a computationally efficient procedure to simulate noisy conditions with the adaptation data. In these experiments, MAP compares favorably with FMLLR in terms of recognition accuracy. Besides, combining FMLLR and MAP significantly outperforms each technique individually, thus providing an efficient alternative for systems with larger resources.

We propose a non-linear feature space transformation for speaker/environment adaptation which forces the individual dimensions of the acoustic data for every speaker to be Gaussian distributed. The transformation is given by the preimage under the Gaussian cumulative distribution function (CDF) of the empirical CDF on a per dimension basis. We show that, for a given dimension, this transformation achieves minimum divergence between the density function of the transformed adaptation data and the normal density with zero mean and unit variance. Experimental results on both small and large vocabulary tasks show consistent improvements over the application of linear adaptation transforms only.

In this paper, we describe an adaptation method for speech recognition systems that is based on a nonlinear transformation of the feature space. In contrast to most existing adaptation methods which assume some form of affine transformation of either the feature vectors or the acoustic models that model the feature vectors, our proposed method composes a general nonlinear transformation from two transformations, one of these being an affine transformation that combines the dimensions of the original feature space, and the other being a nonlinear transformation that is applied independently to each dimension of the previously transformed feature space leading to a general multidimensional nonlinear transformation of the original feature space. This method also differs from other affine techniques in the way the parameters of the transform are shared. In most previous methods, the parameters of the transformation are shared on the basis of the phonetic class, in our method, the parameters of the nonlinear transformation are shared not on the basis of the phonetic class, but rather on the location in the feature space. Experimental results show that the method outperforms affine methods providing up to a 25% relative improvement in the word error rate in an in-car speech recognition task.

We introduce a class of Gaussian mixture models for HMM states in continuous speech recognition. In these models, the covariances or the precisions (inverse covariances) are restricted to lie in subspaces spanned by rank-one symmetric matrices. In both cases, the rank-one matrices are shared across classes of Gaussians. We show that, for the same number of parameters, modeling precisions leads to better performance when compared to modeling covariances. Modeling precisions however gives a distinct advantage in computational and memory requirements. We also show that this class of models provides improvement in accuracy (for the same number of parameters) over classical factor analysed models and the recently proposed EMLLT (extended maximum likelihood linear transform) models which are special instances of this class of models.

This paper describes a robust feature extraction technique for continuous speech recognition. Central to the technique is the minimum variance distortionless response (MVDR) method of spectrum estimation. We incorporate perceptual information directly in to the spectrum estimation. This provides improved robustness and computational efficiency when compared with the previously proposed MVDR-MFCC technique. On an in-car speech recognition task this method, which we refer to as PMCC, is 15% more accurate in WER and requires approximately a factor of 4 times less computation than the MVDR-MFCC technique. On the same task PMCC yields 20% relative improvement over MFCC and 11% relative improvement over PLP frontends. Similar improvements are observed on the Aurora 2 database.

This paper discusses building gender dependent gaussian mixture models (GMMs) and how to integrate these with an efficient gender detection scheme. Gender specific acoustic models of half the size of a corresponding gender independent acoustic model substantially outperform the larger gender independent acoustic models. With perfect gender detection, gender dependent modeling should therefore yield higher recognition accuracy without consuming more memory. Furthermore, as certain phonemes are inherently gender independent (e.g. silence) much of the male and female specific acoustic models can be shared. This paper proposes how to discover which phonemes are inherently similar for male and female speakers and how to efficiently share this information between gender dependent GMMs. A highly accurate gender detection scheme is suggested that takes advantage of computations inherently done in the speech recognizer to detect the gender at a computational cost that is negligible. By making the gender assignment probabilistic an increase in word error rate (WER) seen for erroneously gender labeled speakers is avoided. The method of gender detection and probabilistic use of gender is novel and should be of interest beyond mere gender detection. The only requirement for the method to work is that the training data be appropriately labeled. 1.

This paper describes a robust, accurate, efficient, low-resource, medium-vocabulary, grammar-based speech recognition system using hidden Markov models for mobile applications. Among the issues and techniques we explore are improving robustness and efficiency of the front-end, using multiple microphones for removing extraneous signals from speech via a new multichannel CDCN technique, reducing computation via silence detection, applying the Bayesian information criterion (BIC) to build smaller and better acoustic models, minimizing finite state grammars, using hybrid maximum likelihood and discriminative models, and automatically generating baseforms from single new-word utterances.

In this paper, we present a fast, vocabulary independent, algorithm for spotting words in speech. The algorithm consists of a phone-ngram representation (indexing) stage and a coarse-to-detailed search stage for spotting a word/phone sequence in speech. The phone-ngram representation stage provides a phoneme-level representation of the speech that can be searched efficiently. We present a novel method for phoneme-recognition using a vocabulary prefix tree to guide the creation of the phone-ngram index. The coarse search, consisting of phone-ngram matching, identifies regions of speech as putative word hits. The detailed acoustic match is then conducted only at the putative hits identified in the coarse match. This gives us vocabulary independence and the desired accuracy and speed in wordspotting. Current lattice-based phoneme-matching algorithms are similar to the coarse-match step of our algorithm. We show that our combined algorithm gives a factor of two improvement over the coarse match. The algorithm has wide-ranging use in distributed and pervasive speech recognition applications such as audio-indexing, spoken message retrieval and video-browsing.

We report the results of investigations in acoustic modeling, language modeling and decoding techniques, for the DARPA Communicator, a speaker-independent, telephone-based dialog system. By a combination of methods, including enlarging the acoustic model, augmenting the recognizer vocabulary, conditioning the language model upon the dialog state, and applying a post-processing decoding method, we lowered the overall word error rate from 21.9% to 15.0%, a gain of 6.9% absolute and 31.5% relative.

Describes a robust feature extraction method for continuous speech recognition. Central to the method is the minimum variance distortionless response (MVDR) method of spectrum estimation and a feature trajectory smoothing technique for reducing the variance in the feature vectors. The above method, when evaluated on continuous speech recognition tasks in a stationary and moving car, gave an average relative improvement in WER of greater than 30%.

Abstract In this paper, we describe an adaptation method for speechrecognition systems that is based on a piecewise-linear approx-imation to a non-linear transformation of the feature space.The method extends a previously proposed non-linear trans-formation (NLT) technique by making the transformation func-tion more sophisticated (piecewise-linear instead of piecewise-constant), and by computing the transformation to maximizethe likelihood of the adaptation data given its transcription (in-stead of just matching the global statistics of the test and train-ing data). This method also differs from other linear techniques(such as MLLR, linear feature space transforms, etc.) in twoways - first, the computed transformation is non-linear, second,the tying structure of the transformation depends not on the pho-netic class but rather on the location in the feature space. Exper-imental results show that the method performs well for the caseof limited adaptation data, and the performance gains appear tobe additive to those provided by MLLR - yielding upto 3.4%relative improvement over MLLR.

This paper describes a computationally inexpensive, nonlinear feature transformation technique for rapid adaptation of a speech recognition system to new acoustic conditions. One of the advantages of the method is that it does not require any initial decoding of the adaptation data for computing the nonlinear transform. This technique performs as well as the more expensive unsupervised MLLR technique. Furthermore, it signi cantly adds to the improvement when combined with unsupervised MLLR. 1.

There has been a recent upsurge in the deployment of emerging technologies such as speech and speaker recognition which are reaching maturity. We discuss the details of the components required to build a system for audio indexing and retrieval for spoken documents using content and speaker based information facilitated by speech and speaker recognition. The real power of spoken document analysis is in using both content and speaker information together in retrieval by combining the results. The experiments described here are in the broadcast news domain, but the underlying techniques can easily be extended to other speech-centric applications and transactions.

We present a technique for the segmention of a sound track into two classes of segments. Each frame of signal is preprocessed by extracting cepstral coefficients and their first order derivatives. For each class, the distribution of the frame parameter vectors is modeled by a Gaussian Mixture Model (GMM). GMM order is selected using two criteria : the Minimum Description Length (MDL) criterion and the Akaike Information Criterion (AIC). Frame score is based on a weighted loglikelihood ratio in a window around the frame. Decision for each frame is taken by comparing its score to a threshold. Experiments are presented on speech / music segmentation in audio tracks. In these experiments, the MDL criterion leads to a reasonable GMM order. Using the MDL criterion for GMM order selection, frame classification error rate is around 20%. However, using GMMs with much lower orders, only decreases marginally performances.

In applications such as audio-indexing, spoken message retrieval and video-browsing, it is necessary to have the ability to detect spoken words that are outside the vocabulary of the speech recognizer used in these systems, in large amounts of speech at speeds many times faster than real-time. We present a fast, vocabulary independent, algorithm for spotting words in speech. The algorithm consists of a preprocessing stage and a coarse-to-detailed search strategy for spotting a word/phone sequence in speech. The preprocessing method provides a phone-level representation of the speech that can be searched efficiently. The coarse search, consisting of phone-ngram matching, identifies regions of speech as putative word hits. The detailed acoustic match is then conducted only at the putative hits identified in the coarse match. This gives us the desired accuracy and speed in word spotting. Overall, the algorithm has a speed of execution that is 2400 times faster than real-time.

Modeling data with Gaussian distributions is an important statistical problem. To obtain robust models one imposes constraints the means and covariances of these distributions [6, 4, 10, 8]. Constrained ML modeling implies the existence of optimal feature spaces where the constraints are more valid [2, 3]. This paper introduces one such constrained ML modeling technique called factor analysis invariant to linear transformations (FACILT) which is essentially factor analysis in optimal feature spaces. FACILT is a generalization of several existing methods for modeling covariances. This paper presents an EM algorithm for FACILT modeling. 1.

In this paper we describe our initial eorts in build-ing a natural language understanding (NLU) system across multiple languages. The system allows users to switch lan-guages seamlessly in a single session without requiring any switch in the speech recognition system. Context depen-dence is maintained across sentences, even when the user changes languages. Towards this end we have begun build-ing a universal speech recognizer for English and French languages. We experiment with a universal phonology for both French and English with a novel mechanism to han-dle language dependent variations. Our best results so far show about 5 % relative performance degradation for Eng-lish relative to a unilingual English system and a 9 % rela-tive degradation in French relative to a unilingual French system. The NLU system uses the same statistical un-derstanding algorithms for each language, making system development, maintenance, and portability vastly superior to systems built customly for each language. 1.

The problem of signal reconstruction from linear measurements and convex set constraints is addressed in this paper. The currently popular POCS algorithm suffers from slow convergence and non-unique limits, whereas the recently proposed faster converging Newton algorithm requires projection onto the intersection of convex sets. In this paper, new algorithms, that retain the simplicity of the POCS algorithm, but have increased generality, and superior convergence rates, are presented. Superior convergence rates are achieved by incorporating derivative information using recent results on the differentiability of the projection operator onto a convex set. Efficient implementation of these algorithms using conjugate-gradient iterations for matrix inversion, make them suitable for large-scale problems. Computer simulations demonstrating the effectiveness of these algorithms, are also presented.

Resolution analysis for the problem of signal recovery from finitely many linear samples is the subject of the paper. The classical Rayleigh limit serves only as a lower bound on resolution since it does not assume any recovery strategy and is based only on observed data. The authors show that details finer than the Rayleigh limit can be recovered by simple linear processing that incorporates prior information. They first define a measure of resolution based on allowable levels of error that is more appropriate for current signal recovery strategies than the Rayleigh definition. In the practical situation in which only finitely many noisy observations are available, one must restrict the class of signals in order to make the resolution measure meaningful. They consider the set of bandlimited and essentially time-limited signals since it describes most signals encountered in practice. For this set they show how to precompute resolution limits from knowledge of measurement functionals, signal-to-noise ratio, passband, energy concentration regions, energy concentration factor, and a prescribed level of error tolerance. In the process they also derive an algorithm for high resolution signal recovery. They illustrate the results with an example.

The problem of tomographic reconstruction from a few discrete projections is addressed. When the projection data are discrete and few in number, the image formed by the convolution back-projection algorithm may not be consistent with the observed projections and is known to exhibit artifacts. Hence, the problem formulated here is one of finding an image that is closest to a nominal and is consistent with the projection data and other convex constraints such as positivity. The measure of closeness used is a Hilbert space norm, typically a weighted sum/integral of squares, with weights used to reflect expected deviation from the nominal in different regions. In the absence of constraints, this approach leads to a direct, noniterative algorithm (based on a simple matrix-vector computation) for construction of the image. When additional convex constraints such as positivity and upper-bounds need to be enforced on the reconstructed image to improve resolution, a quadratically convergent Newton algorithm is suggested.>