We introduce Trustera1, the first functional system that redacts personally identifiable information (PII) in real-time spoken conversations to remove agents’ need to hear sensitive information while preserving the naturalness of live customer-agent conversations. As opposed to post-call redaction, audio masking starts as soon as the customer begins speaking to a PII entity. This significantly reduces the risk of PII being intercepted or stored in insecure data storage. Trustera’s architecture consists of a pipeline of automatic speech recognition, natural language understanding, and a live audio redactor module. The system’s goal is three-fold: redact entities that are PII, mask the audio that goes to the agent, and at the same time capture the entity, so that the captured PII can be used for a payment transaction or caller identification. Trustera is currently being used by thousands of agents to secure customers’ sensitive information.

Compared to the landline network environment, the wireless environment presents new factors affecting ASR performance (or accuracy). Our goal here is to determine these factors, and their relative importance, and then to devise methods to mitigate them. We approach this goal, first, by conducting a set of experiments where we use a state of the art ASR system trained on landline speech data, and then compare its performance in landline network conditions to its performance across a variety of wireless network conditions. Based on the results of these experiments, we determine critical factors affecting ASR accuracy. We then use multi-condition acoustic models to mitigate these factors and show that the resulting ASR system is able to not only achieve high accuracy across the various wireless network conditions, but also maintain its high accuracy across landline network conditions. This leads to a recognition system that is channel independent which is a very desirable property for telecom based applications.

In this work, an integrated approach to vector dynamic feature extraction is described in the design of a hidden Markov model (VVD-IHMM) based speech recognizer. The new model contains state-dependent, vector-valued weighting functions responsible for transforming static speech features into the dynamic ones. In this paper, the minimum classification error (MCE) is extended from the earlier formulation of VVD-IHMM that applies to a novel maximum-likelihood based training algorithm. The experimental results on alphabet classification demonstrate the effectiveness of the MCE-trained new model relative to VVD-IHMM using dynamic features that have been subject to optimization during MLE-training.

This paper addresses the problem of speech recognition under accent variations in English language. It has been demonstrated in previous research efforts that the multi-transitional model architecture is one of the solutions for robust speech recognition. In this study, we describe an universal hybrid system that is trained with data from American, Australian, and British accented speech. Experimental results on connecteddigit recognition task show an average string error rate reduction of about 62% and 8% when compared to our best monolingual and multi-transitional systems respectively. The result indicates that the universal model is about three times faster and half time smaller than the multi-transitional or multilingual models and this makes it an ideal choice for practical accentindependent speech recognition applications.

A formulation of the maximum a posteriori (MAP) approach to speaker adaptation is presented with use of the trended or nonstationary-state hidden Markov model (HMM), where the Gaussian means in each HMM state are characterized by time-varying polynomial trend functions of the state sojourn time. Assuming uncorrelatedness among the polynomial coefficients in the trend functions, we have obtained analytical results for the MAP estimates of the parameters including time-varying means and time-invariant precisions. We have implemented a speech recognizer based on these results in speaker adaptation experiments using the TI46 corpora. The experimental evaluation demonstrates that the trended HMM, with use of either the linear or the quadratic polynomial trend function, consistently outperforms the conventional, stationary-state HMM. The evaluation also shows that the unadapted, speaker-independent models are outperformed by the models adapted by the MAP procedure under supervision with as few as a single adaptation token. Further, adaptation of polynomial coefficients alone is shown to be better than adapting both polynomial coefficients and precision matrices when fewer than four adaptation tokens are used, while the reverse is found with a greater number of adaptation tokens.

A new approach for linear prediction (LP) analysis is explored, where predictor can be computed from a mel-warped subband-based autocorrelation functions obtained from the power spectrum. For spectral representation a set of multi-resolution cepstral features are proposed. The general idea is to divide up the full frequency-band into several subbands, perform the IDFT on the mel power spectrum for each subband, followed by Durbin's algorithm and the standard conversion from LP to cepstral coefficients. This approach can be extended to several levels of different resolutions. Multi-resolution feature vectors, formed by concatenation of the subband cepstral features into an extended feature vector, are shown to yield better performance than the conventional mel-warped LPCCs over the full voice-bandwidth for a connected digit recognition task.

Linear prediction (LP) analysis is widely used in speech recognition for representing the short time spectral envelope information of speech. The predictive residues are usually ignored in LP analysis based speech recognition system. In this study, the normalized residual error based on LP is introduced and the performance of the recognizer has been further improved by the addition of this new feature along with its first and second order derivative parameters. The convergence property of the training procedure based on the minimum classification error (MCE) approach is investigated, and experimental results on the city name recognition task demonstrated a 8% string error rate reduction by using the extended feature set as compared to conventional feature set.

We investigate a class of features related to voicing parameters that indicate whether the vocal chords are vibrating. Features describing voicing characteristics of speech signals are integrated with an existing 38-dimensional feature vector consisting of first and second order time derivatives of the frame energy and of the cepstral coefficients with their first and second derivatives. HMM-based connected digit recognition experiments comparing the traditional and extended feature sets show that voicing features and spectral information are complementary and that improved speech recognition performance is obtained by combining the two sources of information.

In this letter, we report our recent work on applications of the combined maximum likelihood linear regression (MLLR) and the minimum classification error training (MCE) approach to estimating the time-varying polynomial Gaussian mean functions in the trended hidden Markov model (HMM). We call this integrated approach the minimum classification error linear regression (MCELR), which has been developed and implemented in speaker adaptation experiments using TI46 corpora. Results show that the adaptation of linear regression on time-varying mean parameters is always better when fewer than three adaptation tokens are used.

In this letter, we investigate the interactions of front-end feature extraction and back-end classification techniques in nonstationary state hidden Markov model (NSHMM) based speech recognition. The proposed model aims at finding an optimal linear transformation on the mel-warped discrete Fourier transform (DFT) features according to the minimum classification error (MCE) criterion. This linear transformation, along with the NSHMM parameters, are automatically trained using the gradient descent method. An error rate reduction of 8% is obtained on a standard 39-class TIMIT phone classification task in comparison with the MCE-trained NSHMM using conventional preprocessing techniques.

In this paper, we extend the maximum likelihood (ML) training algorithm to the minimum classification error (MCE) training algorithm for discriminatively estimating the state-dependent polynomial coefficients in the stochastic trajectory model or the trended hidden Markov model (HMM) originally proposed in Deng (1992). The main motivation of this extension is the new model space for smoothness-constrained, state-bound speech trajectories associated with the trended HMM, contrasting the conventional, stationary-state HMM, which describes only the piecewise-constant "degraded trajectories" in the observation data. The discriminative training implemented for the trended HMM has the potential to utilize this new, constrained model space, thereby providing stronger power to disambiguate the observational trajectories generated from nonstationary sources corresponding to different speech classes. Phonetic classification results are reported which demonstrate consistent performance improvements with use of the MCE-trained trended HMM both over the regular ML-trained trended HMM and over the MCE-trained stationary-state HMM.

We report our recent work on applications of the MAP approach to estimating the time-varying polynomial Gaussian mean functions in the nonstationary-state or trended HMM. Assuming uncorrelatedness among the polynomial coefficients in the trended HMM, we have obtained analytical results for the MAP estimates of the time-varying mean and precision parameters. We have implemented a speech recognizer based on these results in speaker adaptation experiments using the T146 corpora. Experimental results show that the trended HMM always outperforms the standard, stationary-state HMM and that adaptation of polynomial coefficients only is better than adapting both polynomial coefficients and precision matrices when fewer than four adaptation tokens are used.

In this study, a new hidden Markov model that integrates generalized dynamic feature parameters into the model structure is developed and evaluated using maximum-likelihood (ML) and minimum-classification-error (MCE) pattern recognition approaches. In addition to the motivation of direct minimization of error rate, the MCE approach automatically eliminates the necessity of artificial constraints, which were essential for the model formulation based on the ML approach, on the weighting functions in the definition of the generalized dynamic parameters. We design the loss function for minimizing error rate specifically for the new model, and derive an analytical form of the gradient of the loss function that enables the implementation of the MCE approach. The convergence property of the training procedure based on the MCE approach is investigated, and the experimental results from a standard TIMIT phonetic classification task demonstrate a 13.4% error rate reduction compared with the ML approach.

In the study reported in this paper, we investigate interactions of front-end feature extraction and back-end classification techniques in hidden Markov model-based (HMM-based) speech recognition. The proposed model focuses on dimensionality reduction of the mel-warped discrete Fourier transform (DFT) feature space subject to maximal preservation of speech classification information, and aims at finding an optimal linear transformation on the mel-warped DFT according to the minimum classification error (MCE) criterion. This linear transformation, along with the HMM parameters, are automatically trained using the gradient descent method to minimize a measure of overall empirical error counts. A further generalization of the model allows integration of the discriminatively derived state-dependent transformation with the construction of dynamic feature parameters. Experimental results show that state-dependent transformation on mel-warped DFT features is superior in performance to the mel-frequency cepstral coefficients (MFCC's). An error rate reduction of 15% is obtained on a standard 39-class TIMIT phone classification task, in comparison with the conventional MCE-trained HMM using MFCC's that have not been subject to optimization during training.

We investigate the interactions of front-end feature extraction and back-end classification techniques in HMM based speech recognizer. This work concentrates on finding the optimal linear transformation of Mel-warped short-time DFT information according to the minimum classification error criterion. These transformations, along with the HMM parameters, are automatically trained using the gradient descent method to minimize a measure of overall empirical error count. The discriminatively derived state-dependent transformations on the DFT data are then combined with their first time derivatives to produce a basic feature set. Experimental results show that Mel-warped DFT features, subject to appropriate transformation in a state-dependent manner, are more effective than the Mel-frequency cepstral coefficients that have dominated current speech recognition technology. The best error rate reduction of 9% is obtained using the new model, tested on a TIMIT phone classification task, relative to conventional HMM.

In this study we implemented a speech recognizer based on the integrated view, proposed first by Deng (see IEEE Signal Processing Letters, vol.1, no.4, p.66-69, 1994), on the speech preprocessing and speech modeling problems in the recognizer design. The integrated model we developed generalizes the conventional, currently widely used delta-parameter technique, which has been confined strictly to the preprocessing domain only, in two significant ways. First, the new model contains state-dependent weighting functions responsible for transforming static speech features into the dynamic ones in a slowly time-varying manner. Second, novel maximum-likelihood and minimum-classification-error based learning algorithms are developed for the model that allows joint optimization of the state-dependent weighting functions and the remaining conventional HMM parameters. The experimental results obtained from a standard TIMIT phonetic classification task provide preliminary evidence for the effectiveness of our new, general approaches to the use of the dynamic characteristics of speech spectra.