In this paper, we propose a dual-encoder ASR architecture for joint modeling of close-talk (CT) and far-talk (FT) speech, in order to combine the advantages of CT and FT devices for better accuracy. The key idea is to add an encoder selection network to choose the optimal input source (CT or FT) and the corresponding encoder. We use a single-channel encoder for CT speech and a multi-channel encoder with Spatial Filtering neural beamforming for FT speech, which are jointly trained with the encoder selection. We validate our approach on both attention-based and RNN Transducer end-to-end ASR systems. The experiments are done with conversational speech from a medical use case, which is recorded simultaneously with a CT device and a microphone array. Our results show that the proposed dual-encoder architecture obtains up to 9% relative WER reduction when using both CT and FT input, compared to the best single-encoder system trained and tested in matched condition.

In this paper, we apply Semi-Supervised Learning (SSL) along with Data Augmentation (DA) for improving the accuracy of End-to-End ASR.We focus on the consistency regularization principle, which has been successfully applied to image classification tasks, and present sequence-to-sequence (seq2seq) versions of the FixMatch and Noisy Student algorithms.Specifically, we generate the pseudo labels for the unlabeled data onthe-fly with a seq2seq model after perturbing the input features with DA.We also propose soft label variants of both algorithms to cope with pseudo label errors, showing further performance improvements.We conduct SSL experiments on a conversational speech data set (doctor-patient conversations) with 1.9 kh manually transcribed training data, using only 25 % of the original labels (475 h labeled data).In the result, the Noisy Student algorithm with soft labels and consistency regularization achieves 10.4 % word error rate (WER) reduction when adding 475 h of unlabeled data, corresponding to a recovery rate of 92 %.Furthermore, when iteratively adding 950 h more unlabeled data, our best SSL performance is within 5 % WER increase compared to using the full labeled training set (recovery rate: 78 %).

Deep Neural Network (DNN) acoustic models are sensitive to the mismatch between training and testing environments. When a trained model is tested on unseen speakers, domain, or environment, recognition accuracy can degrade substantially. In such a case, offline adaptation with a fair amount of field data can improve recognition accuracy significantly, and is commonly applied to ASR systems in practice. Ideally, such kind of adaptation should be done online as well in order to catch any unexpected dynamic changes in the environments during the inference process. However, online adaptation is subject to strict constraints on computational cost. On the other hand, the small amount of available data and the nature of unsupervised adaptation make online adaptation a very challenging task, especially for DNN acoustic models which normally contain millions of parameters. In this paper, we introduce a simple and effective online adaptation technique to compensate training and testing mismatch for DNN acoustic models. It is done via online adaptation of the parameters associated with the batch normalization applied to the model training process. Our results show that this technique can improve accuracy significantly in a domain mismatched scenario for different DNN architectures.

This paper describes the conceptual and algorithmic evolutions of Memory Based Parametric Equalization (MPEQ) needed to exploit the potentialities of the method within the state-of-the-art Loquendo ASR. MPEQ is the memory-based evolution of Parametric Non-Linear Equalization (PEQ) introduced to overcome the problem of unreliable statistics estimation in presence of very limited acoustic information in the test utterance to be normalized. The main limitations of the method that prevented its practical application were the lack of online implementation, the unrealistic unimodal assumption about the training statistics, the unconditioned application of equalization, and the need for retraining the acoustic models. The paper describes how these limitations have been overcome and reports a large experimentation on many corpora that shows improvements in a variety of mismatched conditions, while preserving performances in matched conditions.

In this paper we describe the implementation of a complete ANN training procedure for speech recognition using the block mode back-propagation learning algorithm. We exploit the high performance SIMD architecture of GPU using CUDA and its C-like language interface. We also compare the speed-up obtained implementing the training procedure only taking advantage of the multi-thread capabilities of multi-core processors. Our approach has been tested by training acoustic models for large vocabulary speech recognition tasks, showing a 6 times reduction of the time required to train real-world large size networks with respect to an already optimized implementation using the Intel MKL libraries.

This paper analyzes the benefits and drawbacks of PEQ, a features normalization technique based on the parametric equalization of the MFCC parameters to match a reference probability distribution. Three evolutions of PEQ are presented as solutions to the limitations encountered. The effects of those proposed evolutions are studied on three speech corpora, namely WSJ0, AURORA-3 and HIWIRE cockpit database. Simple and more complex evaluation tasks with different mismatch conditions given by convolutional and/or additive noise and non-native speakers, are evaluated to analyze the improvements.

In this paper, we propose a word confidence measure based on phone durations depending on large contexts.The measure is based on the expected duration of each recognized phone in a word.In the approach here proposed the duration of each phone is in principle context-dependent, and the measure is a function of the distance between the observed and expected phone duration distributions within a word.Our experiments show that, since the "duration confidence" does not make use of any acoustic information, its Equal Error Rate (EER) in terms of False Accept and False Rejection rates is not as good as the one obtained by using the more informed acoustic confidence measure.However, combining the two measures by a simple linear interpolation, the system EER improves by 6% to 10% relative on an isolated word recognition task in several languages.

This paper presents a front-end consisting of an Artificial Neural Network (ANN) architecture trained with multilingual corpora.The idea is to train an ANN front-end able to integrate the acoustic variations included in databases collected for different languages, through different channels, or even for specific tasks.This ANN front-end produces discriminant features that can be used as observation vectors for language or task dependent recognizers.The approach has been evaluated on three difficult tasks: recognition of non-native speaker sentences, training of a new language with a limited amount of speech data, and training of a model for car environment using a clean microphone corpus of the target language and data collected in car environment in another language.

Training Artificial Neural Networks (ANNs) with large amounts of speech data is a time intensive task due to the intrinsically sequential nature of the back-propagation algorithm.This paper presents an approach for training ANNs using sentence and frame selection.The goal is to speed-up the training process, and to balance the phonetic coverage of the selected frames, trying to mitigate the classification problems related to the prior probabilities of the individual phonetic classes.These techniques, together with a three-step training approach and software optimizations, reduced by an order of magnitude the training time of our models.

A method is described for predicting acoustic feature variability by analyzing the consensus and relative entropy of phoneme posterior probability distributions obtained with different acoustic models having the same type of observations. Variability prediction is used for diagnosis of automatic speech recognition (ASR) systems. When errors are likely to occur, different feature sets are considered for correcting recognition results. Experimental results are provided on the CH1 Italian portion of AURORA3

A technique is proposed for the adaptation of automatic speech recognition systems using hybrid models combining artificial neural networks with hidden Markov models. The application of linear transformations not only to the input features, but also to the outputs of the internal layers is investigated. The motivation is that the outputs of an internal layer represent a projection of the input pattern into a space where it should be easier to learn the classification or transformation expected at the output of the network. A new solution, called conservative training, is proposed that compensates for the lack of adaptation samples in certain classes. Supervised adaptation experiments with different corpora and for different adaptation types are described. The results show that the proposed approach always outperforms the use of transformations in the feature space and yields even better results when combined with linear input transformations

Many techniques for speaker or channel adaptation have been successfully applied to automatic speech recognition. Most of these techniques have been proposed for the adaptation of hidden Markov models (HMMs). Far less proposals have been made for the adaptation of the artificial neural networks (ANNs) used in the hybrid HMM-ANN approach. This paper presents an adaptation technique for ANNs that, similar to the framework of MAP estimation, tries to exploit in the adaptation process prior information that is particularly useful to deal with the problem of sparse training data. We show that the integration of a priori information can be simply achieved by linear interpolation of the weights of an "a priori" network and of a speaker specific network. Good improvements with respect to the baseline results are reported evaluating this technique on the Wall Street Journal WSJ0 and WSJ1 databases and on TIMIT corpus using different amounts of adaptation data

In connectionist learning, one relevant problem is "catastrophic forgetting" that may occur when a network, trained with a large set of patterns, has to learn new input patterns, or has to be adapted to a different environment. The risk of catastrophic forgetting is particularly high when a network is adapted with new data that do not adequately represent the knowledge included in the original training data. Two original solutions are proposed to reduce the risk that the network focuses on new data only, loosing its generalization capability. The first one, conservative training, is a variant to the target assignment policy, while the second approach, support vector rehearsal, selects from the training set the patterns that lay near the borders of the classes not included in the adaptation set. These patterns are used as sentinels that try to keep unchanged the original boundaries of these classes. Moreover, we investigated the extension of the classical approach consisting in applying linear transformations not only to the input features, but also to the outputs of the internal layers. The motivation is that the outputs of an internal layer represent a projection of the input pattern into a space where it should be easier to learn the classification or transformation expected at the output of the network. We illustrate the problems using an artificial test-bed, and apply our techniques to a set of adaptation tasks in the domain of automatic speech recognition (ASR) based on artificial neural networks. Supervised ASR adaptation experiments with several corpora and for different adaptation types are described. We report on the adaptation potential of different techniques, and on the generalization capability of the adapted networks. The results show that the combination of the proposed approaches mitigates the catastrophic forgetting effects, and always outperforms the use of the classical transformations in the feature space.

A soft decision gain modification is introduced and applied to the Ephraim-Malah gain function based on maximum mean-square-error estimation after amplitude compression. Nonlinear evaluations of the noise overestimation factor and spectral floor are used in the same way for the proposed gain modification and for nonlinear spectral subtraction (NSS). Consistent and statistically significant automatic speech recognition improvements of the proposed approach with respect to NSS are observed for different noise conditions considered in the Aurora-2 and Aurora-3 corpora. As the nonlinearity affects the two approaches in the same way, the result of comparison is particularly interesting.

International audience

A soft decision gain modification is introduced and applied to the Ephraim-Malah gain function based on maximum mean square error estimation (MMSE) (Ephraim, Y. and Malah, D., IEEE Trans. Acoust. Speech Sig. Process., vol.ASSP-32, no.6, p.1109-21, 1984; vol.ASSP-33, no.2, p.443-5, 1985) after amplitude compression. Non-linear evaluations of the noise overestimation factor and spectral floor are used in the same way for the proposed gain modification and for non-linear spectral subtraction (NSS). Consistent and statistically significant ASR improvements of the proposed approach with respect to NSS are observed for different noise conditions considered in the AURORA2 and AURORA3 corpora. As the non-linearity affects the two approaches in the same way, the comparison result is particularly interesting.

The successful deployment of a telephone speech application cannot only rely on the accuracy of the recognition results, but also on their reliability.Reliable confidence measures are, thus, necessary in all practical applications to decide whether a recognized wordor sentence -should be accepted or rejected.Since most of the applications are based on continuous speech recognition, controlled by grammars, we present the results of a set of experiments aiming at assessing the quality and the limitations of different confidence measures for six different grammars that can be embedded in several applications.We show that using application independent confidence scoring techniques, good performance are obtained across all six grammars.We introduce also a sentence level confidence measure that allows a significant reduction of the system error rate due to ill-formed sentences.

Compares the performance of an operational automatic speech recognition system when Mel frequency-scaled cepstral coefficients (MFCCs), J-Rasta perceptual linear prediction coefficients (J-Rasta PLP) and energies from a multi resolution analysis (MRA) tree of filters are used as input features to a hybrid system consisting of a neural network (NN) which provides observation probabilities for a network of hidden Markov models (HMM). Furthermore, the paper compares the performance of the system when various combinations of these features are used showing a WER reduction of 16% w.r.t. the use of J-Rasta PLP coefficients, when J-Rasta PLP coefficients are combined with the energies computed at the output of the leaves of an MRA filter tree. Such a combination is practically feasible thanks to the NN architecture used in the system. Recognition is performed without any language model on a very large test set including many speakers uttering proper names from different locations of the Italian public telephone network.

The paper provides a theoretical justification that gravity centers (GC) in frequency bands computed from zero-crossing information are far more robust to additive telephone noise than GCs computed from FFT spectra. Experiments on two different corpora confirm the theoretical results when GCs are added to standard mel frequency-scaled cepstral coefficients (MFCC) and their time derivatives. A 20.1% word error reduction is observed on a large telephone corpus of Italian cities, with an average signal-to-noise ratio (SNR) of 15 dB, if GCs are computed from zero-crossings, while performance deteriorates when GCs are computed from FFT spectra.

Neural networks have found their place among speech recognition technologies mainly with hybrid models that integrates the time warping ability of hidden Markov models (HMM) with the pattern recognition capability of neural networks (NN). Hybrid HMM-NN models have been investigated by several research teams, and constitute now a mature technology highly competitive with HMM. The authors' contribution to this research field is the introduction of a hybrid HMM-NN model whose original points are a training procedure which employs an integrated gradual movement of bootstrap speech segmentations to shorten training time, a time/feature architecture of the first hidden layer devoted to exploit a better feature selection using some a priori knowledge of speech, the use of a particular kind of acoustical modeling more suitable for a discriminative training, and a method to speed-up the execution of the neural component that makes possible develop real time applications using low cost hardware. Recently, the synergy of several input speech parameterizations has been experimented, leading to a further improvement in the recognition accuracy.

Different parameterizations of the speech signal may potentially extract complementary information useful to increase the accuracy in discriminating between confusable sound classes. In spite of this a single parameterization has nearly universally been used in speech recognition because the most diffused matching technology (hidden Markov models) is bound by theoretical and practical constraints that limit the use of multiple features derived from the speech signal with different processing algorithms. On the contrary neural networks are capable of incorporating multiple heterogeneous input features, which do not need to be treated as independent, finding the optimal combination of these features for classification. The purpose of this work is the exploitation of this potentiality of neural networks to improve the speech recognition accuracy. The multiple input features coming from different parameterization algorithms are combined through a network architecture called multi-source NN, designed to obtain the best synergy from them. In this work, we report the last results obtained on this research line by combining the basic spectral features with two auditory inspired features, a formant like feature and the frequency derivatives. The results show that multi-source NN leads to significant error reductions on both isolated words and continuous speech test sets.

The combined use of different set of features extracted from the speech signal with different processing algorithms is a promising approach to improve speech recognition performances.Artificial Neural Networks are well suited to this task since they are able to use directly multiple heterogeneous input features to estimate a near optimal combination of them for classification, without being constrained by a priori assumptions on the stochastic independence of the input sources.This work shows how we have taken advantage of these characteristics of Neural Networks to improve the recognition accuracy of our systems.In particular, three set of input features have been considered as sources in this work: Mel based Cepstral Coefficients derived from the FFT spectrum, RASTA-PLP Cepstral Coefficients, and a set of features that describe the dynamics of the FFT power spectrum along the frequency dimension, instead of the usual time dimension.The experimental results confirm the usefulness of the proposed approach of feature integration that leads to a significant error reduction both on isolated and continuous speech recognition tasks on a large telephone speech test set.

In speech recognition the most diffused technology (hidden Markov models) is constrained by the condition of stochastic independence of its input features. That limits the simultaneous use of features derived from the speech signal with different processing algorithms. On the contrary artificial neural networks (ANN) are capable of incorporating multiple heterogeneous input features, which do not need to be treated as independent, finding the optimal combination of these features for classification. The purpose of this work is the exploitation of this characteristic of ANNs to improve the speech recognition accuracy through the combined use of input features coming from different sources (different feature extraction algorithms). We integrate two input sources: the Mel based cepstral coefficients (MFCC) derived from FFT and the RASTA-PLP cepstral coefficients. The results show that this integration leads to an error reduction of 26% on a telephone quality test set.

This paper presents a new database collected from a bilingual speakers set (49), in two different languages: Spanish and Catalan.Phonetically there are significative differences between both languages.These differences have let us to establish several conclusions on the relevance of language in speaker recognition, using two methods: vector quantization and covariance matrices.

This paper presents Dialogos, a real-time system for human-machine spoken dialogue on the telephone in task-oriented domains. The system has been tested in a large trial with inexperienced users and it has proved robust enough to allow spontaneous interactions both to users which get good recognition performance and to the ones which get lower scores. The robust behavior of the system has been achieved by combining the use of specific language models during the recognition phase of analysis, the tolerance toward spontaneous speech phenomena, the activity of a robust parser, and the use of pragmatic-based dialogue knowledge. This integration of the different modules allows to deal with partial or total breakdowns of the different levels of analysis. We report the field trial data of the system and the evaluation results of the overall system and of the submodules.