This contribution presents our efforts to develop the automatic speech recognition (ASR) systems for three low resource languages: Kurmanji Kurdish, Cree and Inuktut. As a first step, we generate multilingual models from acoustic training data from 12 different languages in the hybrid DNN/HMM framework. We explore different strategies for combining the phones from different languages: either keep the phone labels separate for each language or merge the common phones. For Kurmanji Kurdish and Inuktut, keeping the phones separate gives much lower word error rate (WER), while merging phones gives lower WER for Cree. These WER are lower than training the acoustic models separately for each language. We also compare two different DNN architectures: factored time delay neural network (TDNN-F), and bidirectional long short-term memory (BLSTM) acoustic models. The TDNN-F acoustic models give significantly lower WER for Kurmanji Kurdish and Cree, while BLSTM acoustic models give significantly lower WER for Inuktut. We also show that for each language, training multilingual acoustic models by one more epoch with acoustic data from that language reduces the WER significantly. We also added 512-dimensional embedding features from cross-lingual pre-trained wav2vec2.0 XLSR-53 models, but they lead to only a small reduction in WER.

Roland Kuhn, Fineen Davis, Alain Désilets, Eric Joanis, Anna Kazantseva, Rebecca Knowles, Patrick Littell, Delaney Lothian, Aidan Pine, Caroline Running Wolf, Eddie Santos, Darlene Stewart, Gilles Boulianne, Vishwa Gupta, Brian Maracle Owennatékha, Akwiratékha’ Martin, Christopher Cox, Marie-Odile Junker, Olivia Sammons, Delasie Torkornoo, Nathan Thanyehténhas Brinklow, Sara Child, Benoît Farley, David Huggins-Daines, Daisy Rosenblum, Heather Souter. Proceedings of the 28th International Conference on Computational Linguistics. 2020.

We introduce the first attempt at automatic speech recognition (ASR) in Inuktitut, as a representative for polysynthetic, low-resource languages, like many of the 900 Indigenous languages spoken in the Americas. As most previous work on Inuktitut, we use texts from parliament proceedings, but in addition we have access to 23 hours of transcribed oral stories. With this corpus, we show that Inuktitut displays a much higher degree of polysynthesis than other agglutinative languages usually considered in ASR, such as Finnish or Turkish. Even with a vocabulary of 1.3 million words derived from proceedings and stories, held-out stories have more than 60% of words out-of-vocabulary. We train bi-directional LSTM acoustic models, then investigate word and subword units, morphemes and syllables, and a deep neural network that finds word boundaries in subword sequences. We show that acoustic decoding using syllables decorated with word boundary markers results in the lowest word error rate.

Distributed representations, or embeddings, are commonly learned without supervision on very large unannotated corpora for natural language processing. In speech processing, deep network-based representations such as bottlenecks and x-vectors have had some success,but are limited to supervised or partly supervised settings where annotations are available and are not optimized to separate underlying factors. Here, we propose a generative model with deep encoders and decoders that can learn interpretable speech representations without supervision. Our inductive biases operate as prior distributions in a variational autoencoder model and allow us to separate several latent variables along a continuous range of time-scale properties, as opposed to binary oppositions or hierarchical factorization that have been previously proposed. On simulated data, we confirm that these biases enable the model to accurately recover phonetic and speaker underlying factors. On TIMIT and LibriSpeech, they yield representations that separate phonetic and speaker information, as evidenced by unsupervised results on downstream phoneme and speaker classification tasks using a simple k-means classifier.

The Multi-Genre Broadcast Challenge at ASRU 2015 is a controlled evaluation of speech recognition, speaker diarization, and lightly supervised alignment using BBC TV recordings. CRIM and LIUM teams participated in the speech recognition part of the challenge with a joint submission. This paper presents the CRIM and LIUM's contributions. Each team made different choices to develop its ASR system. By the way, it was expected to compare and to evaluate different approaches to diarization and acoustic modeling, and to get complementary ASR systems for effective merging. CRIM's main contributions are the use of a training scenario similar to multi-lingual training to estimate the deep neural net (DNN) acoustic models with most of the data, the use of a pruned trigram model for search, in addition to the use of a genre-dependent quadgram language model for rescoring the lattice from the search. For LIUM, the focus was on fast decoding with high accuracy. The final word error rates (WER) after merging show that it is possible to get reasonable WER with automatically aligned files. The final global WER of 25.1% corresponds to a WER reduction of about 20% absolute in comparison to the ASR baseline system provided by the organizers.

Voice passphrase verification is the task of deciding whether an audio recording contains a given passphrase. It is usually done by evaluating the likelihood of the passphrase reference text given the audio, which requires a different ASR system for each language. Here we look at verification when the passphrase reference is an audio recording instead of a text. We propose a decision likelihood ratio derived from a generative model. Training is unsupervised and needs only audio, without labelling, so the method applies to any language for which recorded audio exists. We report experiments on English and Urdu telephone speech, and show that our model-based likelihood ratio largely outperforms a baseline of DTW based on MFCC feature vectors.

Several unsupervised methods have been proposed to segment a continuous text stream into individual topics. A simple HMM formulation of the most successful of these methods exposes their underlying assumptions and suggests the use of a new prior for segmentation probability. Under this formulation, we explore the space of possible modeling choices on databases of English and French TV and radio programs. We show that the proposed prior improves segmentation results and can also accommodate additional knowledge sources within the HMM efficient dynamic programming.

In this paper we look at real-time computing issues in large vocabulary speech recognition. We use the French broadcast audio transcription task from ETAPE 2011 for this evaluation. We compare word error rate (WER) versus overall computing time for hidden Markov models with Gaussian mixtures (GMM-HMM) and deep neural networks (DNN-HMM). We show that for a similar computing during recognition, the DNNHMM combination is superior to the GMM-HMM. For a realtime computing scenario, the error rate for the ETAPE dev set is 23.5% for DNN-HMM versus 27.9% for the GMM-HMM: a significant difference in accuracy for comparable computing. Rescoring lattices (generated by DNN-HMM acoustic model) with a quadgram language model (LM), and then with a neural net LM reduces the WER to 22.0% while still providing realtime computing.

The speed of modern processors has remained constant over the last few years but the integration capacity continues to follow Moore's law and thus, to be scalable, applications must be parallelized. The parallelization of the classical Viterbi beam search has been shown to be very difficult on multi-core processor architectures or massively threaded architectures such as Graphics Processing Unit (GPU). The problem with this approach is that active states are scattered in memory and thus, they cannot be efficiently transferred to the processor memory. This problem can be circumvented by using the A* search which uses a heuristic to significantly reduce the number of explored hypotheses. The main advantage of this algorithm is that the processing time is moved from the search in the recognition network to the computation of heuristic costs, which can be designed to take advantage of parallel architectures. Our parallel implementation of the A* decoder on a 4-core processor with a GPU led to a speed-up factor of 6.13 compared to the Viterbi beam search at its maximum capacity and an improvement of 4% absolute in accuracy at real-time.

The speed of modern processors has remained constant over the last few years but the integration capacity continues to follow Moore's law and thus, to be scalable, applications must be parallelized. This paper presents results in using the A* search algorithm in a large vocabulary speech recognition parallel system. This algorithm allows better parallelization over the Viterbi algorithm. First experiments with a “unigram approximation” heuristic resulted in approximatively 8.7 times less states being explored compared to our classical Viterbi decoder. The multi-thread implementation of the A* decoder led to a speed-up factor of 3 over its sequential counterpart.

We describe a lattice generation method that is exact, i.e. it satisfies all the natural properties we would want from a lattice of alternative transcriptions of an utterance. This method does not introduce substantial overhead above one-best decoding. Our method is most directly applicable when using WFST decoders where the WFST is “fully expanded”, i.e. where the arcs correspond to HMM transitions. It outputs lattices that include HMM-state-level alignments as well as word labels. The general idea is to create a state-level lattice during decoding, and to do a special form of determinization that retains only the best-scoring path for each word sequence. This special determinization algorithm is a solution to the following problem: Given a WFST A, compute a WFST B that, for each input-symbol-sequence of A, contains just the lowest-cost path through A.

We report results on audio copy detection for TRECVID 2008 copy detection task. This task involves searching for transformed audio queries in over 200 hours of test audio. The queries were transformed in seven different ways, three of them involved mixing unrelated speech to the original query, making it a much more difficult task. We give results with a few promising algorithms and show that mapping each test frame to the nearest query frame results in robust audio copy detection. The minimal normalized detection cost rate (NDCR) for even the worst case transformations is less than 0.03. The algorithm provides easy parallel processing on a graphics processing unit, leading to a very fast search.

All in-text\treferences\tunderlined\tin\tblue\tare\tlinked\tto\tpublications\ton\tResearchGate, letting you\taccess\tand\tread\tthem\timmediately.

The speed of modern processors has remained constant over the last few years and thus, to be scalable, applications must be parallelized.In addition to the main CPU, almost every computer is equipped with a Graphics Processors Unit (GPU) which is in essence a specialized parallel processor.This paper explores how performances of speech recognition systems can be enhanced by using GPU for the acoustic computations and multicore CPUs for the Viterbi search in a large vocabulary application.The multi-core implementation of our speech recognition system runs 1.3 times faster than the single-threaded CPU implementation.Addition of the GPU for dedicated acoustic computations increases the speed by a factor of 2.8, leading to a word accuracy improvement of 16.6% absolute at real-time, compared to the the single-threaded CPU implementation.

We report results on speaker diarization of French broadcast news and talk shows on current affairs. This speaker diarization process is a multistage segmentation and clustering system. One of the stages is agglomerative clustering using state-of-the-art speaker identification methods (SID). For the QMMs used in this stage, we tried many different feature parameters, including MFCCs, Gaussianized MFCCs, Gaussianized MFCCs with cepstral mean subtraction, and Gaussianized MFCCs with cepstral mean substraction containing only frames with high energy. We found that this last set of feature parameters gave the best results. Compared to Gaussianized MFCCs, these features reduced the diarization error rate (DER) by 12% on a development set and by 19% on a test set. We also combined clusters resulting from Gaussianized and non-Gaussianized feature sets. This cluster combination resulted in another 4% reduction in DER for both the development and the test sets. The best DER we have achieved is 15.4% on the development set, and 14.5% on the test set.

This paper introduces the use of Graphics Processors Unit (GPU) for computing acoustic likelihoods in a speech recog-nition system. In addition to their high availability, GPUs pro-vide high computing performance at low cost. We have used a NVidia GeForce 8800GTX programmed with the CUDA (Com-pute Unified Device Architecture) which shows the GPU as a parallel coprocessor. The acoustic likelihoods are computed as dot products, operations for which GPUs are highly efficient. The implementation in our speech recognition system shows that GPU is 5x faster than the CPU SSE-based implementation. This improvement led to a speed up of 35 % on a large vocabu-lary task. Index Terms: Speech recognition, GPU 1.

Real-time speech recognition captioning has not progressed much, beyond television broadcast, to other tasks like meetings in the workplace. A number of obstacles prevent this transition, such as proper means to receive and display captions, or on-site shadow speakers costs. More problematic is the insufficient performance of speech recognition for less formal and one-time events. We describe how we developed a mobile platform for remote captioning during trials in several conferences and meetings. We also show that sentence selection based on relative entropy allows training of adequate language models with small amounts of in-domain data, making real-time captioning of an event possible with only a few hours of preparation.

We report results on speaker diarization of telephone conversations. This speaker diarization process is similar to the multistage segmentation and clustering system used in broadcast news. It consists of an initial acoustic change point detection algorithm, iterative Viterbi re-segmentation, gender labeling, agglomerative clustering using a Bayesian information criterion (BIC), followed by agglomerative clustering using state-of-the-art speaker identification methods (SID) and Viterbi re-segmentation using Gaussian mixture models (GMMs). The Viterbi re-segmentation using GMMs is new, and it reduces the diarization error rate (DER) by 10%. We repeat these multistage segmentation and clustering steps twice: once with MFCCs as feature parameters for the GMMs used in gender labeling, SID and Viterbi re-segmentation steps, and another time with Gaussianized MFCCs as feature parameters for the GMMs used in these three steps. The resulting clusters from the parallel runs are combined in a novel way that leads to a significant reduction in the DER. On a development set containing 30 telephone conversations, this combination step reduced the DER by 20%. On another test set containing 30 telephone conversations, this step reduced the DER by 13%. The best error rate we have achieved is 6.7% on the development set, and 9.0% on the test set.

We report results on speaker diarization of telephone conversations. This speaker diarization process is similar to the multistage segmentation and clustering system used in broadcast news. It consists of an initial acoustic change point detection algorithm, iterative Viterbi re-segmentation, gender labeling, agglomerative clustering using a Bayesian information criterion (BIC), followed by agglomerative clustering using state-of-the-art speaker identification (SID) methods and Viterbi re-segmentation using Gaussian mixture models (GMMs). We repeat these multistage segmentation and clustering steps twice: once with mel-frequency cepstral coefficients (MFCCs) as feature parameters for the GMMs used in gender labeling, SID, and Viterbi re-segmentation steps and another time with Gaussianized MFCCs as feature parameters for the GMMs used in these three steps. The resulting clusters from the parallel runs are combined in a novel way that leads to a significant reduction in the diarization error rate (DER). On a development set containing 30 telephone conversations, this combination step reduced the DER by 20%. On another test set containing 30 telephone conversations, this step reduced the DER by 13%. The best error rate we have achieved is 6.7% on the development set and 9.0% on the test set.

We compare two approaches to the problem of session variability in Gaussian mixture model (GMM)-based speaker verification, eigenchannels, and joint factor analysis, on the National Institute of Standards and Technology (NIST) 2005 speaker recognition evaluation data. We show how the two approaches can be implemented using essentially the same software at all stages except for the enrollment of target speakers. We demonstrate the effectiveness of zt-norm score normalization and a new decision criterion for speaker recognition which can handle large numbers of t-norm speakers and large numbers of speaker factors at little computational cost. We found that factor analysis was far more effective than eigenchannel modeling. The best result we obtained was a detection cost of 0.016 on the core condition (all trials) of the evaluation.

We present a corpus-based approach to speaker verification in which maximum-likelihood II criteria are used to train a large-scale generative model of speaker and session variability which we call joint factor analysis. Enrolling a target speaker consists in calculating the posterior distribution of the hidden variables in the factor analysis model and verification tests are conducted using a new type of likelihood II ratio statistic. Using the NIST 1999 and 2000 speaker recognition evaluation data sets, we show that the effectiveness of this approach depends on the availability of a training corpus which is well matched with the evaluation set used for testing. Experiments on the NIST 1999 evaluation set using a mismatched corpus to train factor analysis models did not result in any improvement over standard methods, but we found that, even with this type of mismatch, feature warping performs extremely well in conjunction with the factor analysis model, and this enabled us to obtain very good results (equal error rates of about 6.2%).

This work explores a new way of fusing audio and visual information for audio-visual automatic speech recognition in the context of a large vocabulary application. Mouth shape information is extracted off-line and integrated into a speech recognition system using a phoneme-based Dempster-Shafer fusion approach. The fusion methodology assumes that the audio information about the phonemes is a precise Bayesian source while the visual information is an imprecise evidential source. This ensures that the visual information does not degrade significantly the audio information in situation where the audio performs well in controlled noiseless environment. Bayesian and simple consonance belief structures are explored and compared, along with standard stack-based fusion

We present the results of speaker verification experiments conducted on the NIST 2005 evaluation data using a factor analysis of speaker and session variability in 6 telephone speech corpora distributed by the Linguistic Data Consortium. We demonstrate the effectiveness of zt-norm score normalization and a new decision criterion for speaker recognition which can handle large numbers of t-norm speakers and large numbers of speaker factors at little computational cost. The best result we obtained was a detection cost of 0.016 on the core condition (all trials) of the evaluation

Growing needs for French closed-captioning of live TV broadcasts in Canada cannot be met only with stenography-based technology because of a chronic shortage of skilled stenographers. Using speech recognition for live closed-captioning, however, requires several specific problems to be solved, such as the need for low-latency real-time recognition, remote operation, automated model updates, and collaborative work. In this paper we describe our solutions to these problems and the implementation of a live captioning system based on the CRIM speech recognizer. We report results from field deployment in several projects. The oldest in operation has been broadcasting real-time closed-captions for more than 2 years. Index Terms: speech recognition, closed-captioning, model adaptation.

We propose a method for estimating the parameters of SPLICElike transformations from individual utterances so that this type of transformation can be used to normalize acoustic feature vectors for speech recognition on an utterance-by-utterance basis in a similar manner to cepstral mean normalization. We report results on an in-house French language multi-speaker database collected while deploying an automatic closed-captioning system for live broadcast news. An unusual feature of this database is that there are very large amounts of training data for the individual speakers (typically several hours) so that it is very difficult to improve on multi-speaker modeling by using standard methods of speaker adaptation. We found that the proposed method of feature normalization is capable of achieving a 6 % relative improvement over cepstral mean normalization on this task. Index Terms: speech recognition, SPLICE, eigenvoice MAP 1.

We show how the factor analysis model for speaker verification can be successfully implemented using some fast approximations which result in minor degradations in accuracy and open up the possibility of training the model on very large databases such as the union of all of the Switchboard corpora. We tested our algorithms on the NIST 1999 evaluation set (carbon data as well as electret). Using warped cepstral features we obtained equal error rates of about 6.3% and minimum detection costs of about 0.022.

In this paper, we present the approach we used to produce a training database from a set of recorded newscasts for which we had inaccurate transcriptions. These transcribed segments correspond to a set of prepared anchor texts and journalist stories, not necessarily in chronological order of their actual presentation. No segmental time boundary information is provided. Our main concern is thus to establish time marks that delimit the audio segments of the corresponding texts. To resolve this problem, we have developped a time marking procedure using our speech recognition engine. We obtain a segmentation accuracy of 80%. 1.

We derive an exact solution to the problem of maximum likelihood estimation of the supervector covariance matrix used in extended MAP (or EMAP) speaker adaptation and show how it can be regarded as a new method of eigenvoice estimation. Unlike other approaches to the problem of estimating eigenvoices in situations where speaker-dependent training is not feasible, our method enables us to estimate as many eigenvoices from a given training set as there are training speakers. In the limit as the amount of training data for each speaker tends to infinity, it is equivalent to cluster adaptive training.

We describe a new method of estimating speaker-dependent hidden Markov models for speakers in a closed population. Our method differs from previous approaches in that it is based on an explicit model of the correlations between all of the speakers in the population, the idea being that if there is not enough data to estimate a Gaussian mean vector for a given speaker then data from other speakers can be used provided that we know how the speakers are correlated with each other. We explain how to estimate inter-speaker correlations using a Kullback-Leibler divergence minimization technique which can be applied to the problem of estimating the parameters of all of the hyperdistributions that are currently used in Bayesian speaker adaptation.

In film post-production, efficient methods for re-recording a dialogue or dubbing in a new language require a precisely time-aligned text, with individual letters time-coded to video frame resolution. Currently, this time alignment is performed by experts in a painstaking and slow process. To automate this process, we used CRIM’s largevocabulary HMM speech recognizer as a phoneme segmenter and measured its accuracy on typical film extracts in French and English. Our results reveal several characteristics of film dialogues, in addition to noise, that affect segmentation accuracy, such as speaking style or reverberant recordings. Despite these difficulties, an HMM-based segmenter trained on clean speech can still provide more than 89 % acceptable phoneme boundaries on typical film extracts. We also propose a method which provides the correspondence between aligned phonemes and graphemes of the text. The method does not use explicit rules, but rather computes an optimal string alignment according to an edit-distance metric. Together, HMM phoneme segmentation and phonemegrapheme correspondence meet the needs of film postproduction for a time-aligned text, and make it possible to automate a large part of the current post-synch process. 1.

This paper describes the system currently under development at CRIM whose aim is to provide real-time closed captioning of live TV broadcast news in Canadian French. This project is done in collaboration with TVA Network, a national TV broadcaster and the RQST (a Québec association which promotes the use of subtitling). The automated closed-captioning system will use CRIM’s transducer-based large vocabulary French recognizer. The system will be totally integrated to the existing broadcaster’s equipment and working methods. First ”on-air” use will take place in February 2004. 1.

This article describes a new approach for cues discrimination between speakers addressed to a speaker identification task. To this end, we make use of elements of decision theory. We propose to decompose the conventional feature space (MFCCs) into two subspaces which carry information about discriminative and confusable sections of the speech signal. The method is based on the idea that, instead of adapting the speakers models to a new test environment, we require the test utterance to fit the speakers models environment. Discriminative sections of training speech are used to estimate the probability density function (pdf) of a discriminative world model (DM), and confusable sections to estimate the probability density function of a confusion world model (CM). The two models are then used as a maximum likelihood detector (filter) at the input of the recogniser. The method was experimented on highly mismatched telephone speech and achieves a considerable improvement (averaging 16 % gain in performance) over the baseline GMM system. 1.

We describe a new algorithm for estimating eigenvoices (or, equivalently, EMAP correlation matrices) for large vocabulary speech recognition tasks. The algorithm is an EM procedure based on a novel maximum likelihood formulation of the estimation problem which is similar to the mathematical model underlying probabilistic principal components analysis. It enables us to extend eigenvoice/EMAP adaptation in a natural way to adapt variances as well as mean vectors. It differs from other approaches in that it does not require that speaker dependent or speaker adapted models for the training speakers be given in advance (these are derived as a byproduct of the estimation procedure). Accordingly our algorithm can be applied directly to large vocabulary tasks even if the training data is sparse in the sense that only a small fraction of the total number of Gaussians is observed for each training speaker. 1.

There are two types of prior distribution that can be viewed as natural for extended MAP (or EMAP) speaker adaptation.One arises from modeling the correlations between speakers (assumed to be constant across HMM Gaussians) and the other from modeling the correlations between HMM Gaussians (assumed to be constant across speakers).In this paper we present new results establishing the usefulness of correlations of the first type for speaker adaptation and we outline a tensor product construction which enables both types of correlation to be integrated in a common mathematical framework.We also present the results of some experiments which suggest that the two types of correlation are equally effective for speaker adaptation and that there is no incremental improvement to be gained by modeling both of them simultaneously.

Although finite-state transducers have been widely used in linguistics, their application to speech recognition has begun only recently (M. Mohri, 1997). We describe our implementation of French large vocabulary recognition based on transducers, and how we take advantage of this approach to integrate automatic pronunciation rules and cross-word phenomena such as French "liaison". We also show that a simple, single-level Viterbi algorithm can efficiently decode speech recognition transducers and handle cross-word context models and cross-word phonological rules. In our experiments we compared network size, error rate and decoding speed of our transducer based recognizer against a baseline HTK recognizer, on a large vocabulary French dictation task. Transducers reduced search time by a factor of 25 compared to our HTK recognizer. We also examined the effect of automated pronunciation rules, and their combination with crossword phonological rules that control "liaison". We obtained a 23% relative reduction in the word error rate on a 5000 word task.

Decision trees have been used in speech recognition with large numbers of context-dependentHMM models, to provide models for contexts not seen in training.Trees are usually created by successive node splitting decisions, based on how well a single Gaussian or Poisson density fits the data associated with a node.We introduce a new node splitting criterion, derived from the maximum likelihood fitting of the complex node distributions with Gaussian tiedmixture densities.We also carry the use of decision trees for tying HMM models a step further.In addition to questions about phonetic class of neighbouring phonemes, we allow questions about the HMM model state to be asked.The resulting decision tree maximizes the likelihood by adjusting the amount of parameter tying simultaneously across state and context.Accuracy improvement and model size reduction were evaluated on a gender-dependent 5K closed-vocabulary WSJ task, using the SI-84 and SI-284 training sets, for tied-mixture and continuous HMM models.The new decision trees are shown to reduce both error rate and model size, while being computationally cheap enough to allow consideration of two preceding and two following phones for the context.

First pass acoustic models in a two pass, large vocabulary, speaker dependent, continuous speech recognizer are discussed. The models are based on single state VQ, compared to our earlier 5-state continuous HMMs. The loss in performance was recovered using context dependent models, multiple codebooks and improvements in the computation of the observation vector. This resulted in a first pass inclusion of >97% while reducing the computational burden by a factor of 30. Currently, the system runs real-time on an AT&T DSP32C module and an HP-735 workstation, with recognition accuracies of up to 90% for a 5000 word vocabulary task.>