In this paper, we apply Semi-Supervised Learning (SSL) along with Data Augmentation (DA) for improving the accuracy of End-to-End ASR.We focus on the consistency regularization principle, which has been successfully applied to image classification tasks, and present sequence-to-sequence (seq2seq) versions of the FixMatch and Noisy Student algorithms.Specifically, we generate the pseudo labels for the unlabeled data onthe-fly with a seq2seq model after perturbing the input features with DA.We also propose soft label variants of both algorithms to cope with pseudo label errors, showing further performance improvements.We conduct SSL experiments on a conversational speech data set (doctor-patient conversations) with 1.9 kh manually transcribed training data, using only 25 % of the original labels (475 h labeled data).In the result, the Noisy Student algorithm with soft labels and consistency regularization achieves 10.4 % word error rate (WER) reduction when adding 475 h of unlabeled data, corresponding to a recovery rate of 92 %.Furthermore, when iteratively adding 950 h more unlabeled data, our best SSL performance is within 5 % WER increase compared to using the full labeled training set (recovery rate: 78 %).

Deep Neural Network (DNN) acoustic models are sensitive to the mismatch between training and testing environments. When a trained model is tested on unseen speakers, domain, or environment, recognition accuracy can degrade substantially. In such a case, offline adaptation with a fair amount of field data can improve recognition accuracy significantly, and is commonly applied to ASR systems in practice. Ideally, such kind of adaptation should be done online as well in order to catch any unexpected dynamic changes in the environments during the inference process. However, online adaptation is subject to strict constraints on computational cost. On the other hand, the small amount of available data and the nature of unsupervised adaptation make online adaptation a very challenging task, especially for DNN acoustic models which normally contain millions of parameters. In this paper, we introduce a simple and effective online adaptation technique to compensate training and testing mismatch for DNN acoustic models. It is done via online adaptation of the parameters associated with the batch normalization applied to the model training process. Our results show that this technique can improve accuracy significantly in a domain mismatched scenario for different DNN architectures.

This paper describes the conceptual and algorithmic evolutions of Memory Based Parametric Equalization (MPEQ) needed to exploit the potentialities of the method within the state-of-the-art Loquendo ASR. MPEQ is the memory-based evolution of Parametric Non-Linear Equalization (PEQ) introduced to overcome the problem of unreliable statistics estimation in presence of very limited acoustic information in the test utterance to be normalized. The main limitations of the method that prevented its practical application were the lack of online implementation, the unrealistic unimodal assumption about the training statistics, the unconditioned application of equalization, and the need for retraining the acoustic models. The paper describes how these limitations have been overcome and reports a large experimentation on many corpora that shows improvements in a variety of mismatched conditions, while preserving performances in matched conditions.

In this paper we describe the implementation of a complete ANN training procedure for speech recognition using the block mode back-propagation learning algorithm. We exploit the high performance SIMD architecture of GPU using CUDA and its C-like language interface. We also compare the speed-up obtained implementing the training procedure only taking advantage of the multi-thread capabilities of multi-core processors. Our approach has been tested by training acoustic models for large vocabulary speech recognition tasks, showing a 6 times reduction of the time required to train real-world large size networks with respect to an already optimized implementation using the Intel MKL libraries.

This paper analyzes the benefits and drawbacks of PEQ, a features normalization technique based on the parametric equalization of the MFCC parameters to match a reference probability distribution. Three evolutions of PEQ are presented as solutions to the limitations encountered. The effects of those proposed evolutions are studied on three speech corpora, namely WSJ0, AURORA-3 and HIWIRE cockpit database. Simple and more complex evaluation tasks with different mismatch conditions given by convolutional and/or additive noise and non-native speakers, are evaluated to analyze the improvements.

This paper presents a front-end consisting of an Artificial Neural Network (ANN) architecture trained with multilingual corpora.The idea is to train an ANN front-end able to integrate the acoustic variations included in databases collected for different languages, through different channels, or even for specific tasks.This ANN front-end produces discriminant features that can be used as observation vectors for language or task dependent recognizers.The approach has been evaluated on three difficult tasks: recognition of non-native speaker sentences, training of a new language with a limited amount of speech data, and training of a model for car environment using a clean microphone corpus of the target language and data collected in car environment in another language.

Training Artificial Neural Networks (ANNs) with large amounts of speech data is a time intensive task due to the intrinsically sequential nature of the back-propagation algorithm.This paper presents an approach for training ANNs using sentence and frame selection.The goal is to speed-up the training process, and to balance the phonetic coverage of the selected frames, trying to mitigate the classification problems related to the prior probabilities of the individual phonetic classes.These techniques, together with a three-step training approach and software optimizations, reduced by an order of magnitude the training time of our models.

A method is described for predicting acoustic feature variability by analyzing the consensus and relative entropy of phoneme posterior probability distributions obtained with different acoustic models having the same type of observations. Variability prediction is used for diagnosis of automatic speech recognition (ASR) systems. When errors are likely to occur, different feature sets are considered for correcting recognition results. Experimental results are provided on the CH1 Italian portion of AURORA3

A technique is proposed for the adaptation of automatic speech recognition systems using hybrid models combining artificial neural networks with hidden Markov models. The application of linear transformations not only to the input features, but also to the outputs of the internal layers is investigated. The motivation is that the outputs of an internal layer represent a projection of the input pattern into a space where it should be easier to learn the classification or transformation expected at the output of the network. A new solution, called conservative training, is proposed that compensates for the lack of adaptation samples in certain classes. Supervised adaptation experiments with different corpora and for different adaptation types are described. The results show that the proposed approach always outperforms the use of transformations in the feature space and yields even better results when combined with linear input transformations

Many techniques for speaker or channel adaptation have been successfully applied to automatic speech recognition. Most of these techniques have been proposed for the adaptation of hidden Markov models (HMMs). Far less proposals have been made for the adaptation of the artificial neural networks (ANNs) used in the hybrid HMM-ANN approach. This paper presents an adaptation technique for ANNs that, similar to the framework of MAP estimation, tries to exploit in the adaptation process prior information that is particularly useful to deal with the problem of sparse training data. We show that the integration of a priori information can be simply achieved by linear interpolation of the weights of an "a priori" network and of a speaker specific network. Good improvements with respect to the baseline results are reported evaluating this technique on the Wall Street Journal WSJ0 and WSJ1 databases and on TIMIT corpus using different amounts of adaptation data

In connectionist learning, one relevant problem is "catastrophic forgetting" that may occur when a network, trained with a large set of patterns, has to learn new input patterns, or has to be adapted to a different environment. The risk of catastrophic forgetting is particularly high when a network is adapted with new data that do not adequately represent the knowledge included in the original training data. Two original solutions are proposed to reduce the risk that the network focuses on new data only, loosing its generalization capability. The first one, conservative training, is a variant to the target assignment policy, while the second approach, support vector rehearsal, selects from the training set the patterns that lay near the borders of the classes not included in the adaptation set. These patterns are used as sentinels that try to keep unchanged the original boundaries of these classes. Moreover, we investigated the extension of the classical approach consisting in applying linear transformations not only to the input features, but also to the outputs of the internal layers. The motivation is that the outputs of an internal layer represent a projection of the input pattern into a space where it should be easier to learn the classification or transformation expected at the output of the network. We illustrate the problems using an artificial test-bed, and apply our techniques to a set of adaptation tasks in the domain of automatic speech recognition (ASR) based on artificial neural networks. Supervised ASR adaptation experiments with several corpora and for different adaptation types are described. We report on the adaptation potential of different techniques, and on the generalization capability of the adapted networks. The results show that the combination of the proposed approaches mitigates the catastrophic forgetting effects, and always outperforms the use of the classical transformations in the feature space.

A soft decision gain modification is introduced and applied to the Ephraim-Malah gain function based on maximum mean-square-error estimation after amplitude compression. Nonlinear evaluations of the noise overestimation factor and spectral floor are used in the same way for the proposed gain modification and for nonlinear spectral subtraction (NSS). Consistent and statistically significant automatic speech recognition improvements of the proposed approach with respect to NSS are observed for different noise conditions considered in the Aurora-2 and Aurora-3 corpora. As the nonlinearity affects the two approaches in the same way, the result of comparison is particularly interesting.

International audience

A soft decision gain modification is introduced and applied to the Ephraim-Malah gain function based on maximum mean square error estimation (MMSE) (Ephraim, Y. and Malah, D., IEEE Trans. Acoust. Speech Sig. Process., vol.ASSP-32, no.6, p.1109-21, 1984; vol.ASSP-33, no.2, p.443-5, 1985) after amplitude compression. Non-linear evaluations of the noise overestimation factor and spectral floor are used in the same way for the proposed gain modification and for non-linear spectral subtraction (NSS). Consistent and statistically significant ASR improvements of the proposed approach with respect to NSS are observed for different noise conditions considered in the AURORA2 and AURORA3 corpora. As the non-linearity affects the two approaches in the same way, the comparison result is particularly interesting.

The renewed interest in grapheme to phoneme conversion (G2P), due to the need of developing multilingual speech synthesizers and recognizers, suggests new approaches more efficient than the traditional rule&exception ones. A number of studies have been performed to investigate the possible use of machine learning techniques to extract phonetic knowledge in a automatic way starting from a lexicon. In this paper, we present the results of our experiments in this research field. Starting from the state of art, our contribution is in the development of a language-independent learning scheme for G2P based on Classification and Regression Trees (CART). To validate our approach, we realized G2P converters for the following languages: British English, American English, French and Brazilian Portuguese.

The paper provides a theoretical justification that gravity centers (GC) in frequency bands computed from zero-crossing information are far more robust to additive telephone noise than GCs computed from FFT spectra. Experiments on two different corpora confirm the theoretical results when GCs are added to standard mel frequency-scaled cepstral coefficients (MFCC) and their time derivatives. A 20.1% word error reduction is observed on a large telephone corpus of Italian cities, with an average signal-to-noise ratio (SNR) of 15 dB, if GCs are computed from zero-crossings, while performance deteriorates when GCs are computed from FFT spectra.

Neural networks have found their place among speech recognition technologies mainly with hybrid models that integrates the time warping ability of hidden Markov models (HMM) with the pattern recognition capability of neural networks (NN). Hybrid HMM-NN models have been investigated by several research teams, and constitute now a mature technology highly competitive with HMM. The authors' contribution to this research field is the introduction of a hybrid HMM-NN model whose original points are a training procedure which employs an integrated gradual movement of bootstrap speech segmentations to shorten training time, a time/feature architecture of the first hidden layer devoted to exploit a better feature selection using some a priori knowledge of speech, the use of a particular kind of acoustical modeling more suitable for a discriminative training, and a method to speed-up the execution of the neural component that makes possible develop real time applications using low cost hardware. Recently, the synergy of several input speech parameterizations has been experimented, leading to a further improvement in the recognition accuracy.

Different parameterizations of the speech signal may potentially extract complementary information useful to increase the accuracy in discriminating between confusable sound classes. In spite of this a single parameterization has nearly universally been used in speech recognition because the most diffused matching technology (hidden Markov models) is bound by theoretical and practical constraints that limit the use of multiple features derived from the speech signal with different processing algorithms. On the contrary neural networks are capable of incorporating multiple heterogeneous input features, which do not need to be treated as independent, finding the optimal combination of these features for classification. The purpose of this work is the exploitation of this potentiality of neural networks to improve the speech recognition accuracy. The multiple input features coming from different parameterization algorithms are combined through a network architecture called multi-source NN, designed to obtain the best synergy from them. In this work, we report the last results obtained on this research line by combining the basic spectral features with two auditory inspired features, a formant like feature and the frequency derivatives. The results show that multi-source NN leads to significant error reductions on both isolated words and continuous speech test sets.

In speech recognition the most diffused technology (hidden Markov models) is constrained by the condition of stochastic independence of its input features. That limits the simultaneous use of features derived from the speech signal with different processing algorithms. On the contrary artificial neural networks (ANN) are capable of incorporating multiple heterogeneous input features, which do not need to be treated as independent, finding the optimal combination of these features for classification. The purpose of this work is the exploitation of this characteristic of ANNs to improve the speech recognition accuracy through the combined use of input features coming from different sources (different feature extraction algorithms). We integrate two input sources: the Mel based cepstral coefficients (MFCC) derived from FFT and the RASTA-PLP cepstral coefficients. The results show that this integration leads to an error reduction of 26% on a telephone quality test set.

This paper presents a new database collected from a bilingual speakers set (49), in two different languages: Spanish and Catalan.Phonetically there are significative differences between both languages.These differences have let us to establish several conclusions on the relevance of language in speaker recognition, using two methods: vector quantization and covariance matrices.