sponsorship: Laszlo Toth was supported by the J ' anos Bolyai Research Scholarship of the Hungarian Academy of Sciences and the UNKP19-4 New Excellence Program of the Hungarian Ministry of Innovation and Technology. (Hungarian Academy of Sciences, New Excellence Program of the Hungarian Ministry of Innovation and Technology|UNKP19-4)

Deep neural networks (DNNs) have shown a great promise in exploiting out-of-language data, particularly for under-resourced languages. The common trend is to merge data from various source languages to train a multilingual DNN and then reuse the hidden layers as language-independent feature extractors for a low-resource target language. While there is a consensus that using as much data from various languages results in a better and more general multilingual DNN, employing only source languages similar to the target language has proven effective. In this study, we propose a novel framework for multilingual DNN training, which employs all the available training data and exploits complementary information from individual source languages at the same time. Toward this goal, we borrow the idea of an ensemble with one generalist and many specialists. The generalist is derived from a multilingual DNN acoustic model trained on all available multilingual data; the specialists are the DNNs derived from the source languages individually. Then, the constituents in the ensemble are combined using weighted averaging schemes, where the combination weights are trained to minimize the cross-entropy objective function. In this framework, we seek for complementary information among the constituents while it is possible to get at least the performance equal to the baseline. Moreover, unlike previous well-known system combination schemes, only one model is required during decoding. We successfully examined two combination methodologies and demonstrated their usefulness in different scenarios using the multilingual GlobalPhone dataset. It is observed that, specifically, speech recognition systems developed in low-resource settings profit from the proposed strategy.

DNNs have shown remarkable performance in multilingual scenarios; however, these models are often too large in size that adaptation to a target language with relatively small amount of data cannot be well accomplished. In our previous work, we utilized Low-Rank Factorization (LRF) using singular value decomposition for multilingual DNNs to learn compact models which can be adapted more successfully. In this paper, we address two problems associated with that LRF scheme and we propose a compellingly simple methodology to overcome them. First, factorizing all layers results in a huge drop in performance and consequently a long recovery process is required which is not practically efficient. Secondly, LRF can be viewed as a regularization by which some noise is added to weight layers; however, factorizing all layers together equates to adding too much noise which results in bad performance. To mitigate these problems, we propose to apply LRF sequentially. We demonstrate that the lost information after factorizing one layer is small and can be rapidly retrieved; hence, sequential factorization is more efficient. Moreover, the sequential LRF adds only a small amount of noise sequentially which is a better regularization. Our experiments are conducted on five languages from the GlobalPhone dataset.

Speech signals are produced by the smooth and continuous movements of the human articulators. An articulatory representation of speech is considered to be a more compact, more universal, and language-independent speech feature space and can, therefore, improve crosslingual and multilingual speech recognition systems, especially when porting components from one language to another in low-resource scenarios. However, learning the acoustic-to-articulatory conversion has proven to be a very challenging task. In this paper, we utilize a manifold learning technique to derive a nonlinear feature transformation from the conventional filterbank feature space to an articulatory-like feature space. The coordinates in the resultant representation of which some have demonstrable phonological meaning are shown to be highly portable across languages. We propose a proper framework in terms of data selection and graph construction to train coordinates from multilingual data, which allows for training the coordinate space when we have abundant out-of-language data. Deep neural network (DNN) bottleneck features are demonstrated to exhibit a greater degree of language independence when using this representation than in the case of filterbank features as inputs. The usability of this representation is further demonstrated in a number of speech recognition experiments using DNNs in a variety of crosslingual and multilingual scenarios using the multilingual GlobalPhone dataset. Especially, speech recognition systems developed in low-resource settings profit from the improved portability across languages.

Multilingual Deep Neural Networks (DNNs) have been successfully used to exploit out-of-language data to improve under-resourced ASR. In this paper, we improve on a multilingual DNN by utilizing low-rank factorization (LRF) of weight matrices via Singular Value Decomposition (SVD) to sparsify a multilingual DNN. LRF was previously used for monolingual DNNs, yielding large computational savings without a significant loss in recognition accuracy. In this work, we show that properly applying LRF on a multilingual DNN can improve recognition accuracy for multiple low-resource ASR configurations. First, only the final weight layer is factorized. Since the output weight layer needs to be trained with language specific data, reducing the number of parameters is beneficial for under-resourced languages. It is common in multilingual DNN speech recognition, to further adapt the full neural network through retraining of the multilingual DNN on target language data. Again we observe that in low-resource situations, this adaptation can bring significant improvement if LRF is applied to all hidden layers. We demonstrate the positive effect of LRF in two very different scenarios: one is a phone recognition task for two related languages and the other is a word recognition task using five different languages from the GlobalPhone dataset.

Copyright © 2015 ISCA. Conventional acoustic modeling involves estimating many parameters to effectively model feature distributions. The sparseness of speech and text data, however, degrades the reliability of the estimation process and makes speech recognition a challenging task. In this paper, we propose to use a nonlinear feature transformation based on the speech manifold called Intrinsic Spectral Analysis (ISA) for under-resourced speech recognition. First, we investigate the usefulness of ISA features in low resource scenarios for both Gaussian mixture and deep neural network (DNN) acoustic modeling. Moreover, due to the connection of ISA features to the articulatory configuration space, this feature space is potentially less language dependent than other typical spectral-based features, and therefore exploiting out-of-language data in this feature space is beneficial. We demonstrate the positive effect of ISA in the frame work of multilingual DNN systems where Flemish and Afrikaans are used as donor and under-resourced target languages respectively. We compare the performance of ISA with conventional features in both multilingual and under-resourced monolingual conditions.

Intrinsic Spectral Analysis (ISA) has been formulated within a manifold learning setting allowing natural extensions to out-of-sample data together with feature reduction in a learning framework. In this paper, we propose two approaches to improve the performance of supervised ISA, and then we examine the effect of applying Linear Discriminant technique in the intrinsic subspace compared with the extrinsic one. In the interest of reducing complexity, we propose a preprocessing operation to find a small subset of data points being well representative of the manifold structure; this is accomplished by maximizing the quadratic Renyi entropy. Furthermore, we use class based graphs which not only simplify our problem but also can be helpful in a classification task. Experimental results for phone classification task on TIMIT dataset showed that ISA features improve the performance compared with traditional features, and supervised discriminant techniques outperform in the ISA subspace compared to conventional feature spaces.

Reading tutor software uses automatic speech recognition technology to support children in developing their reading skills. In many forms of exercise and evaluation, tracking the reading position is a relevant task or even a prerequisite, e.g. to provide assistance on the pronunciation of a word or to advance the screen to the next page. In this paper, we introduce a new robust tracking algorithm, which measures the similarity between the recognized phones and the phonetic transcription of words displayed on a screen using an efficient dynamic programming algorithm. The criteria for accepting a word reading attempt and thus advancing the cursor can hence be expressed phonetically. In addition, the most likely state of the Hidden Markov Model (HMM) used to decode the speech serves as a fallback for cases of phone matching failure. The new tracker's performance is compared with two other trackers which use either the most likely HMM state or phone matching. The evaluation metrics quantify both the frequency of timely movements and loss of tracking synchronicity. The proposed approach performs significantly better than the others achieving a Timing Accuracy of Tracking of 81.03% compared to 50.63% of the phone matching approach and 32.36% of the state-based approach.

Exemplar based recognition systems are characterized by the fact that, instead of abstracting large amounts of data into compact models, they store the observed data enriched with some annotations and infer on-the-fly from the data by finding those exemplars that resemble the input speech best. One advantage of exemplar based systems is that next to deriving what the current phone or word is, one can easily derive a wealth of meta-information concerning the chunk of audio under investigation. In this work we harvest meta-information from the set of best matching exemplars, that is thought to be relevant for the recognition such as word boundary predictions and speaker entropy. Integrating this meta-information into the recognition framework using segmental conditional random fields, reduced the WER of the exemplar based system on the WSJ Nov92 20k task from 8.2% to 7.6%. Adding the HMM-score and multiple HMM phone detectors as features further reduced the error rate to 6.6%.

In this paper we present a number of improvements that were recently made to the template based speech recognition system developed at ESAT. Combining these improvements resulted in a decrease in word error rate from 9.6% to 8.2% on the Nov92, 20k trigram, Wall Street Journal task. The improvements are along different lines. Apart from the time warping already applied within the DTW, it was found beneficial to apply additional length compensation on the template score. The single best score was replaced by a weighted k-NN average, while maintaining natural successor information as an ensemble cost. The local geometry of the acoustic space is now taken into account by assigning a diagonal covariance matrix to each input frame. Context sensitivity of short templates is increased by taking cross boundary scores into account for sorting the N best templates. Furthermore boundaries on the template segmentations may be relaxed. Finally context dependent word templates are now being used for short words. Several other variants that were not retained in the final system are discussed as well.

This paper summarizes the 2010 CLSP Summer Workshop on speech recognition at Johns Hopkins University. The key theme of the workshop was to improve on state-of-the-art speech recognition systems by using Segmental Conditional Random Fields (SCRFs) to integrate multiple types of information. This approach uses a state of-the-art baseline as a springboard from which to add a suite of novel features including ones derived from acoustic templates, deep neural net phoneme detections, duration models, modulation features, and whole word point-process models. The SCRF framework is able to appropriately weight these different information sources to produce significant gains on both die Broadcast News and Wall Street Journal tasks.

In this paper, we use prosodic information to improve the accuracy of our template-based automatic speech recognizer. Prosodic information is harvested adopting a data-driven approach. A number of prosodic features is extracted, then combined into major groups, and finally studied separately and together. All acoustic evidence, both segmental and suprasegmental, is modelled non-parametrically. The different sources of information are conveniently combined with segmental conditional random fields. Prosody enhances the accuracy of the state-of-the-art baseline by reducing the word error rate by 7% relative on the nov92, 20κ trigram, Wall Street Journal task. Copyright © 2011 ISCA.

Over the years, the focus in noise robust speech recognition has shifted from noise robust features to model based techniques such as parallel model combination and uncertainty decoding. In this paper, we contrast prime examples of both approaches in the context of large vocabulary recognition systems such as used for automatic audio indexing and transcription. We look at the approximations the techniques require to keep the computational load reasonable, the resulting computational cost, and the accuracy measured on the Aurora4 benchmark. The results show that a well designed feature based scheme is capable of providing recognition accuracies at least as good as the model based approaches at a substantially lower computational cost. © 2010 ISCA.

In this paper we describe and analyze a data pruning method in combination with template-based automatic speech recognition. We demonstrate the positive effects of polishing the template database by minimizing the word error rate scores. Data pruning allowed to effectively reduce the database size, and therefore the model size, by an impressive 30%, with consequent benefits on the computation time and memory usage. © 2010 ISCA.

This paper describes the ESAT 2008 Broadcast News transcription system for the N-Best 2008 benchmark, developed in part for testing the recent SPRAAK Speech Recognition Toolkit. ESAT system was developed for the Southern Dutch Broadcast News subtask of N-Best using standard methods of modern speech recognition. A combination of improvements were made in commonly overlooked areas such as text normalization, pronunciation modeling, lexicon selection and morphological modeling, virtually solving the out-of-vocabulary (OOV) problem for Dutch by reducing OOV-rate to 0.06% on the N-Best development data and 0.23% on the evaluation data. Recognition experiments were run with several configurations comparing one-pass vs. two-pass decoding, high-order vs. low-order n-gram models, lexicon sizes and different types of morphological modeling. The system achieved 7.23% word error rate (WER) on the broadcast news development data and 20.3% on the much more difficult evaluation data of N-Best.

International audience

International audience

Acoustic differences are so subtle in a native accent identification (AID) task that a brute force frame-based Gaussian Mixture Model (GMM) fails to discover the tiny distinctions [1]. Apart from the frame-based framework, in this paper we propose a vector-based speaker modeling method, to which common support vector machine (SVM) kernels can be applied. The vector-based speaker model is composed of the concatenation of the average acoustic representations of all phonemes. SVM and GMM classifiers are compared on the speaker models. Moreover, based on the observation that accents only differ in a limited number of phonemes, a variable selection framework is indispensable to select accent relevant features. We investigate a forward selection method, Analysis of Variance (ANOVA) , and a backward selection method, SVM- Recursive Feature Elimination (SVM-RFE). We find that the multiclass SVM-RFE achieves comparable performance with the ANOVA on optimally selected variable sets, while it obtains excellent performance with very few features in low dimensions. Results demonstrate the effectiveness of the proposed speaker models together with the SVM classifier both in low dimensions and in high dimensions as well as the necessity of variable selection. Index Terms: variable selection, native accent identification, support vector machines, recursive feature elimination, cross

Example based speech recognition is critically dependent on the quality of the acoustic distance measure between input and reference vectors. In the past, the commonly used Euclidean distance has been refined to take into account the covariance of the different sounds, resulting in a class dependent distance measure. However, using the same measure for the whole class is still too crude: vectors in the tails of the distribution (outliers) are unduly considered equally representative of the class as those in the centre. In this paper, we derive two techniques inspired by non-parametric density estimation that explicitly adjust the distance measure based on the position of the reference vector in its class. Experiments on three low-level acoustic tasks show that "data sharpening" results in a substantial improvement, while "adaptive kernels" have minimal effect.

In this paper we investigate the behaviour of different acoustic distance measures for template based speech recognition in light of the combination of acoustic distances, linguistic knowledge and template concatenation fluency costs. To that end, different acoustic distance measures are compared on tasks with varying levels of fluency/linguistic constraints. We show that the adoption of those constraints invariably results in an acoustically clearly suboptimal template sequence being chosen as the winning hypothesis. There are strong implications for the design of acoustic distance measures: distance measures that are optimal for frame based classification may prove to be suboptimal for full sentence recognition. In particular, we show this is the case when comparing the Euclidean and the recently introduced adaptive kernel local Mahalanobis distance measures. Index Terms: template based, example based, episodic, nonparametric 1.

In previous study we proposed algorithms to select representative frames from a segment for phoneme likelihood evaluation. In this paper we show that this frame selection behavior is phoneme dependent. We observe that some phonemes benefit from frame selection while others do not, and that this separation matches the phonetic categories. For those phonemes sensitive to frame selection, we find that selecting frames at some pre-defined positions in the segment enhances the discrimination between phonemes. These phoneme-dependent positions are explicitly retrieved and used in a phoneme classification task. Experimental results on the TIMIT phonetic database show that the frame selection method significantly outperforms decoding by the classical Viterbi decoder. Index Terms: frame selection, phoneme classification 1.

Despite their known weaknesses, hidden Markov models (HMMs) have been the dominant technique for acoustic modeling in speech recognition for over two decades. Still, the advances in the HMM framework have not solved its key problems: it discards information about time dependencies and is prone to overgeneralization. In this paper, we attempt to overcome these problems by relying on straightforward template matching. The basis for the recognizer is the well-known DTW algorithm. However, classical DTW continuous speech recognition results in an explosion of the search space. The traditional top-down search is therefore complemented with a data-driven selection of candidates for DTW alignment. We also extend the DTW framework with a flexible subword unit mechanism and a class sensitive distance measure—two components suggested by state-of-the-art HMM systems. The added flexibility of the unit selection in the template-based framework leads to new approaches to speaker and environment adaptation. The template matching system reaches a performance somewhat worse than the best published HMM results for the Resource Management benchmark, but thanks to complementarity of errors between the HMM and DTW systems, the combination of both leads to a decrease in word error rate with 17% compared to the HMM results.

In this paper, we propose a maximum likelihood (ML) based frame selection approach. A fixed frame rate adopted in most state-of-the-art speech recognition systems can face some problems, such as accidentally meeting noisy frames, assigning the same importance to each frame, and pitch asynchronous representation. As an attempt to avoid those problems, our approach selects reliable frames from a fine resolution along the time axis. In a phoneme recognition task, we show that significant improvements are achieved with the frame selection approach comparing to a system with a fixed frame rate.

Most ASR systems adopt an all-in-one approach: acoustic model, lexicon and language model are all applied simultaneously, thus forming a single large search space. This way, both lexicon and language model help in constraining the search at an early stage which greatly improves its efficiency. However, such close integration comes at a cost: all resources must be kept simple. Achieving higher accuracy in unconstrained LVCSR tasks will require more complex resources while at the same time the 'unconstrainedness' of the task reduces the effectiveness of the all-in-one approach. Therefore, we propose a modular two-layered architecture. First, a pure acoustic-phonemic search generates a dense phone network. Next a robust decoder finds those words from the lexicon that match well with the phone sequences encoded in the phone network. In this paper we investigate the properties the robust word decoder must have and we propose an efficient search algorithm.

The state-of-the-art in automatic speech recognition is distinctly Markovian. The ubiquitous 'beads-on-a-string' approach, where sentences are explained as a sequence of words, words as a sequence of phones and phones as a sequence of acoustically stable states, is bound to lose a lot of dynamic information. In this paper we show that a combination with example-based recognition can be used to recapture some of that information. A new approach to combine Hidden Markov Model (HMM) and phone-examplebased continuous speech recognition is presented. Experiments show that the combination outperforms the HMM recognizer, and indicate that adding long-span information is especially beneficial.

Our former study [1] has shown that maximum likelihood (ML) based frame selection, which selects reliable frames from a high resolution along the time axis, helps to improve the discrimination between phonemes. In this paper, we present our recent research on single frame selection for a phoneme classification task. A new single selection, which only selects one frame for one state in an Hidden Markov Model (HMM), is proposed. The new technique takes likelihoods of frames and their positions in a phoneme segment into account at the same time, and selects very few frames to represent the spectral evolution of the phoneme. Furthermore, we also show that for a low model complexity, a phoneme model trained by selected frames is more discriminative than a model using all frames.

Acoustic differences between native accents may prove to be too subtle for straightforward brute force techniques such as blindly clustered Gaussian mixture model (GMM) classifiers to yield satisfactory discrimination performance while these methods work well for classifying more pronounced differences such as language, gender or channel. In this paper it is shown that small channel differences are easier to detect by such coarse classifiers than native accent differences. Performance of native accent classification can be improved considerably by incorporating the knowledge of the underlying phoneme sequence and using phoneme specific GMMs. Further improvements are obtained if optimal feature selection is combined with the phoneme dependent GMMs, resulting in usage of less than 10% of the original features. The presented methods result in a reduction of more than 40% in relative error rate in a 5-class classification task.

State-of-the-art speech recognition relies on a state-dependent distance measure. In HMM systems, the distance measure is trained into state-dependent covariance matrices using a maximum likelihood or discriminative criterion. This "automatic" adjustment of the distance measure is traditionally considered an inherent advantage of HMMs over DTW (dynamic time warping) recognizers, as those typically rely on a uniform Euclidean distance. We show how to incorporate a non-uniform weighted distance measure into an example-based recognition system. By doing so, we manage to combine the superior segmental behaviour of DTW with the near-optimal acoustic distance measure as found in HMMs. The non-uniform distance measure enforces modifications to the k nearest neighbours search, an essential component in our large vocabulary DTW approach. We show that the complexity of our solution remains within bounds. The validity of the full approach is verified by experimental results on the resource management and TIDigits tasks.

The merits of different signal preprocessing schemes for speech recognizers are usually assessed purely on the basis of the resulting recognition accuracy. Such benchmarks give a good indication as to whether one preprocessing is better than another, but little knowledge is acquired about why it is better or how it could be further improved. In order to gain more insight in the preprocessing, we seek to re-synthesize speech from speech recognition features. This way, we are able to pin-point some deficiencies in our current preprocessing scheme. Additional analysis of successful new preprocessing schemes may allow us one day to identify precisely those properties that are desirable in a feature set. Next to these purely scientific aims, the re-synthesis of speech from recognition features is of interest to thin-client speech applications, and as an alternative to the classical LPC source-filter model for speech manipulation.

This paper describes a new architecture for large vocabulary continuous speech recognition (LVCSR), which will be developed within the project FLaVoR (Flexible Large Vocabulary Recognition). The proposed architecture abandons the standard all-in-one search strategy with integrated acoustic, lexical and language model information. Instead, a modular framework is proposed which allows for the integration of more complex linguistic components. The search process consists of two layers. First, a pure acoustic-phonemic search generates a dense phoneme network enriched with meta-data. Then, the output of the first layer is used by sophisticated language technology components for word decoding in the second layer. Preliminary experiments prove the feasibility of the approach.

The performance of large vocabulary recognition systems, for instance in a dictation application, typically deteriorates severely when used in a reverberant environment. This can be partially avoided by adding a dereverberation algorithm as a speech signal preprocessing step. The purpose of this paper is to compare the effect of different speech dereverberation algorithms on the performance of a recognition system. Experiments were conducted on the Wall Street Journal dictation benchmark. Reverberation was added to the clean acoustic data in the benchmark both by simulation and by re-recording the data in a reverberant room. Moreover additive noise was added to investigate its effect on the dereverberation algorithms. We found that dereverberation based on a delay-and-sum beamforming algorithm has the best performance of the investigated algorithms.

The dominant acoustic modeling methodology based on Hidden Markov Models is known to have certain weaknesses. Partial solutions to these flaws have been presented, but the fundamental problem remains: compression of the data to a compact HMM discards useful information such as time dependencies and speaker information. In this paper, we look at pure example based recognition as a solution to this problem. By replacing the HMM with the underlying examples, all information in the training data is retained. We show how information about speaker and environment can be used, introducing a new interpretation of adaptation. The basis for the recognizer is the well-known DTW algorithm, which has often been used for small tasks. However, large vocabulary speech recognition introduces new demands, resulting in an explosion of the search space. We show how this problem can be tackled using a data driven approach which selects appropriate speech examples as candidates for DTW-alignment.

In this paper, we investigate the use of the total likelihood (the weighted sum of the likelihoods of all possible state sequences) instead of the approximation with the Viterbi likelihood (the like-lihood of the best state sequence) normally used in speech recognition. Next to its use in a recognizer, the use of total likelihoods in the context of an automatic word aligrunent task is also addressed shortly. We describe how the search algorithm must be modified and how word lattices based on total likelihoods can be constructed. The total likelihood framework also requires us to make a distinction between upgrading the language model scores or downgrading the acoustic model scores in the recognizer. To help in deciding between these two alternatives, some theoretical foundation is given to the practice of making a weighted combination of language and acoustic scores. Finally, the total likelihood and the Viterbi framework are compared in terms of accuracy and computational effort on the Wall Street Journal recognition task, while the accuracy of word alignments is evaluated on a large Dutch corpus.

In this paper, we describe a method to identify Flemish accents and examine the contribution of different features. Although Flanders is quite small, we found that, even when speakers are asked to read a text in standard Dutch, regional accents are still present in the speech. Our method uses formant and phoneme duration features in a text-dependent model. A relevant set of features is automatically selected and normalized in a way reminiscent of the eigenvoices approach for speaker adaptation. We found good discrimination results between the accent of Antwerp and Brabant versus the accents of other provinces. However, the confusion between the accent of Limburg versus the accent of East- and West-Flanders shows that the feature set is too restricted,

In this paper the performance and assumptions of linear prediction acoustical modelling are assessed on the free field cough sound. Four distinct free field cough classes originating from animal and human species in different health conditions are considered. Firstly based on the prediction signal-to-noise ratio the model order for each cough class was chosen to 14. Secondly for each cough class the vocal tract formants are estimated from the linear prediction parameters. Finally the occurrence of subglottal resonances in the cough-sound is investigated.

The aim of discriminant feature analysis techniques in the signal processing of speech recognition systems is to find a feature vector transformation which maps a high dimensional input vector onto a low dimensional vector while retaining a maximum amount of information in the feature vector to discriminate between predefined classes. This paper points out the significance of the definition of the classes in the discriminant feature analysis technique. Three choices for the definition of the classes are investigated: the phonemes, the states in context independent acoustic models and the tied states in context dependent acoustic models. These choices for the classes were applied to (1) standard LDA (linear discriminant analysis) for reference and to (2) MIDA, an improved, mutual information based discriminant analysis technique. Evaluation of the resulting linear feature transforms on a large vocabulary continuous speech recognition task shows, depending on the technique, the best choice for the classes.

Harmonic selection is a useful mechanism for speech enhancement. However, there is difficulty in detecting harmonic structure in the spectrum in the presence of noise or interfering speech. In this paper we address the problem of applying the reassignment method to produce a higher-resolution spectrum for improving the harmonic selection. The reassignment method is to assign the value of the spectrogram computed by the short-time Fourier transform to the gravity center of the region rather than the geometric center of the region. We analyze the resolution capability of the reassigned Fourier spectrum and show that it has better frequency separation than Fourier spectrum. We incorporate the reassignment method with the amplitude-modulation based harmonic selection to segregate a target vowel from interfering vowels. Experimental results show that the target vowel recognition performance is improved by using the reassignment method for increasing the readability of the harmonic structure in speech signals.

This paper presents two different directions to build HMM models which give enough acoustic resolution and t in limited user resources. They both refer to scaling down the acoustic models which are built with tied gaussian HMMs. The total number of gaussians is reduced by a pairwise merging, and the number of gaussians per state is reduced by selecting them based on the so called occupancy criterion. Experiments carried out on the WSJ recognition task show that after scaling down, no further training is needed when the number of gaussians or the number of gaussians per state is reduced up to a factor three. This is an advantage as retraining can not be executed by the final system user.

In most HMM-based recognition systems, a mixture of diagonal covariance gaussians is used to model the observation density functions in the states. The use of diagonal covariance gaussians however assumes that the underlying data vectors have uncorrelated vector components: if each gaussian is replaced with its full covariant counterpart, the off-diagonal elements in the covariance matrices should be small. To that end, most recognition systems have some kind of decorrelation matrix near the end of the preprocessing. Examples are the inverse cosine transform used with cepstral coefficients, and principal component analysis (PCA) or linear discriminant analysis (LDA) of the features. However, none of these transforms is optimal if it comes to reducing the mismatch introduced by setting the off-diagonal elements in the covariance matrices to zero. The algorithm described in this paper reduces the local correlations between feature vector components inside the gaussians with a single global linear transform at the end of the preprocessing stage. The algorithm is optimal in the sense that we calculate the linear transformation that minimises the sum of the square of all off-diagonal elements over all gaussians. The algorithm is compared with principal component analysis, linear discriminant analysis and the recently published maximum likelihood modelling for semi-tied covariance matrices. The decorrelation method is also evaluated on two speech recognition tasks. A significant relative improvement was achieved in both cases.

In an HMM based large vocabulary continuous speech recognition system, the evaluation of - context dependent - acoustic models is very time consuming. In Semi-Continuous HMMs, a state is modelled as a mixture of elementary - generally gaussian - probability density functions. Observation probability calculations of these states can be made faster by reducing the size of the mixture of gaussians used to model them. In this paper, we propose different criteria to decide which gaussians should remain in the mixture for a state, and which ones can be removed. The performance of the criteria is compared on context dependent tied state models using the WSJ recognition task. Our novel criterion, which decides to remove a gaussian in a state if it is based on too few acoustic data, outperforms the other described criteria.

This paper proposes a method to transform acoustic models that have been trained with a certain group of speakers for use on different speech in hidden Markov model based (HMM-based) automatic speech recognition. Features are transformed on the basis of assumptions regarding the difference in vocal tract length between the groups of speakers. First, the vocal tract length (VTL) of these groups has been estimated based on the average third formant F/sub 3/. Second, the linear acoustic theory of speech production has been applied to warp the spectral characteristics of the existing models so as to match the incoming speech. The mapping is composed of subsequent nonlinear submappings. By locally linearizing it and comparing results in the output, a linear approximation for the exact mapping was obtained which is accurate as long as the warping is reasonably small. The feature vector, which is computed from a speech frame, consists of the mel scale cepstral coefficients (MFCC) along with delta and delta/sup 2/-cepstra as well as delta and delta/sup 2/ energy. The method has been tested for TI digits data base, containing adult and children speech, consisting of isolated digits and digit strings of different length. The word error rate when trained on adults and tested on children with transformed adult models is decreased by more than a factor of two compared to the nontransformed case.

This paper proposes a method to transform acoustic models (HMM gaussian mixtures) that have been trained on a certain group of speakers for use on speech from a different group of speakers. Cepstral features are transformed on the basis of assumptions regarding the difference in vocal tract length (VTL) between the groups of speakers (VTL normalisation, VTLN). Firstly, the VTL of these groups has been estimated based on the average third formant F . Secondly, the linear acoustic theory of speech production has been applied to warp the spectral characteristics of the existing models so as to match the incoming speech. The mapping is composed of subsequent non-linear submappings. By locally linearizing it, a linear approximation was obtained which is accurate as long as warping is reasonably small. The method has been tested for the TI digits database, containing adult and kids speech, consisting of isolated digits and digit strings of different length. The word error rate when trained on adults and tested on kids with transformed adult models is decreased by more than a factor of 2 compared to the non-transformed case.

To cope with the prohibitive growth of lexical tree based search-graphs when using cross-word context dependent (CD) phone models, an efficient novel search-topology was developed. The lexicon is stored as a compact static network with no language model (LM) information attached to it. The static representation avoids the cost of dynamic tree expansion, facilitates the integration of additional pronunciation information (e.g. assimilation rules) and is easier to integrate in existing search engines. Moreover, the network representation also results in a compact structure when words have alternative pronunciations, and due to its construction, it offers partial LM forwarding at no extra cost. Next, all knowledge sources (pronunciation information, language model and acoustic models) are combined by a slightly modified token-passing algorithm, resulting in a one pass time-synchronous recognition system.

A new normalisation technique for speech recognition in adverse conditions is presented. Specifically the influence of additive noise in combination with convolutive distortions is considered. In the proposed method a masking constant is added to the outputs of a mel scale triangular filterbank. This is done for testing and training samples. The goal is to normalize the signal-to-noise ratio (SNR) in each frequency band by adapting the masking constant depending on the measured SNR or dynamic range in each band. This makes the extracted parameters less sensitive to the noise level, but also the influence of channel distortions is suppressed. The method is easy to implement and works on-line. Experimental results are given on the NOISEX-92 database and on real car data.

The problem addressed in this paper, is the incorporation of user specific words in a speaker independent speech recognition system. No transcription is used to model the new words, modeling is based on a very small number of training utterances only. We investigated two different modeling methods. The first is intended for small vocabulary recognisers. The HMM models for the new words are enhanced by averaging their states with the nearest speaker independent state. This way, the recognition error was reduced by a factor two, and even the noise robustness of the speaker independent models seems to be transferred to the new models. The second method can be used in large vocabulary recognisers. Using a CSR algorithm, a transcription for the new words is found in terms of the subword models in the recogniser. The resulting models perform equally well as the models based on phonetic transcriptions.

Speech recognition in adverse conditions remains a difficult but challenging problem.It is already shown [1] that normalisation of the dynamic range (SNR 1 ) of the frequency channels in a mel scale triangular filterbank (MFCC) [2], improves the robustness against both additive and convolutional noise.Nevertheless, because the method is based on a masking-technique, the improvement is small in the case of SNR values that are smaller than the target (normalised) SNR.A solution for this problem can be found in first enhancing the filterbank energies before the masking-technique is applied.For this purpose we developed a Non-linear Spectral Estimator (NSE) for speech recognition that operates on the log filterbank energies.NSE enhances these filterbank energies and makes use of SNR-normalisation also effective at very low SNRs.Experimental results are given on the NOISEX-92 [3] database.Better recognition performance is seen even at 0dB SNR.

Semi-continuous Density HMM's have -due to the decoupling between the set of gaussians and the other HMM-parameters -more possibilities than Continuous Density HMM's to match the number of parameters in the model to the available train data.The computational load of the SC-HMM's however is huge compared to the load of their continuous counterparts, because of the large mixture weighting vector and because of the fact that for each frame all gaussians have to be evaluated.This paper describes the different steps taken to reduce the computational load of the SC-HMM's, resulting in faster and better models.

Describes a recurrent neural network based, isolated word speech recognizer. The recognizer uses 2 MLPs. A first, static MLP is used for classification of frames in phonemes. Next, a time compression step is applied. The resulting pseudo-segments are then used as inputs for a second, dynamic MLP that integrates the information over time to decide the current word. The authors apply this approach on an isolated digit recognition task and compare the results with hybrid MLP/HMM approach using the same static MLP.>

In this paper we describe a method to separate a scalar mixture of two signals. The method is based on the use of decorrelation as a signal separation criterion. It is proven analytically that decorrelating the output signals at different time lags is sufficient provided that the normalised autocorrelation functions of the source signals are sufficiently distinct. The method involves an iterative least-squares solution of a set of nonlinear equations. Alternatively, a gradient search algorithm can also be used to find the minimum of the sum of squares of these equations. Both time- and frequency-domain formulations are given. Some convergence and stability issues are discussed and a small example is given at the end.>

One of the problems with speech recognition in the car is the position of the far talk microphone. This position not only implies more or less noise, coming from the car (engine, tires,...) or from other sources (traffic, wind noise,...) but also a different acoustical transfer function. In order to compare the microphone positions in the car, we recorded a multispeaker database in a car with 7 different positions and compared them on the basis of SNR and recognition rate. The position at the ceiling right in front of the speaker gave the best results.>

In this paper we present a family of nonlinear spectral estimators for noise reduction which are approximated and implemented by a multilayer perceptron neural network. The estimators are approximations of the true minimum mean square error estimator in the logarithmic or a related perceptual domain. Training data for the neural networks is generated from relevant statistical speech and noise models. One single estimator network is generated for all frequency channels. Parameters describing both the noise and speech distribution are estimated on line and provided as extra inputs to the neural net. Including these parameters significantly improves performance over standard spectral estimators which are based on a global speech model and a noise model described by a single parameter, the noise mean.>

Variable frame rate (VFR) analysis is a technique used in speech processing and recognition for discarding frames that are too much alike. The article introduces a new method for VFR. Instead of calculating the distance between frames, the norm of the derivative parameters is used in deciding to retain or to discard a frame, informal inspection of speech spectrograms shows that this new method puts more emphasis on the transient regions of the speech signal. Experimental results with a hidden Markov model (HMM) based system show that the new method outperforms the classical method.>

A novel combination of multilayer perceptrons (MLPs) and hidden Markov models (HMMs) is presented. Instead of using MLPs as probability generators for HMMs, the authors propose to use MLPs as labelers for discrete parameter HMM's. Compared with the probabilistic interpretation of MLPs, this gives them the advantage of flexibility in system design (e.g., the use of word models instead of phonetic models while using the same MLPs). Moreover, since they do not need to reach a global minimum, they can do with MLPs with fewer hidden nodes, which can be trained faster. In addition, they do not need to retrain the MLPs with segmentations generated by a Viterbi alignment. Compared with Euclidean labeling, their method has the advantages of needing fewer HMM parameters per state and obtaining a higher recognition accuracy. Several improvements of the baseline MLP labeling are investigated. When using one MLP, the best results are obtained when giving the labels a fuzzy interpretation. It is also possible to use parallel MLPs where each is based on a different parameter set (e.g., basic parameters, their time derivatives, and their second-order time derivatives). This strategy increases the recognition results considerably. A final improvement is the training of MLPs for subphoneme classification.>

An algorithm for adaptive noise cancellation and signal separation is presented. It is an extension of the classical Widrow least-mean-square (LMS) noise canceler for the case of signal leakage into the noise reference. The algorithm is derived intuitively from the interpolation of the adaptive noise canceler as a decorrelator between signal estimate and noise in which the noise reference is replaced by a signal-free noise estimate. Thus, a symmetric adaptive decorrelator is obtained for signal separation, as the artificial distinction between signal and noise concepts has disappeared. The algorithm has its limitations as convergence to the desired solution and stability around it can only be guaranteed for a subclass of signal separation problems. These restrictions are rarely violated in real life problems, however, and seem to be fundamental to the signal separation problem rather than algorithm dependent.>

Concentrates on techniques for parameter and frame reduction. Standard parameter sets of contemporary speech recognition systems include some set of basic parameters (e.g. cepstra and energy), their time derivatives, and their second time derivatives. The large number of parameters (typically 25 to 45) induces the investigation of methods for reducing the amount of computation, without loss of recognition accuracy. The paper presents the variable frame rate analysis, a technique for leaving out frames that are too resemblant, and describes methods for decreasing the number of parameters in a frame.>

Switching adaptive filters, suitable for speech beamforming, with no prior knowledge about the speech source are presented. The filters have two sections, of which only one section at any given time is allowed to adapt its coefficients. The switch between both is controlled by a speech detection function. The first section implements an adaptive look direction and cues in on the desired speech. This section only adapts when speech is present. The second section acts as a multichannel adaptive noise canceller. The obtained noise references are typically very bad; hence, adaptation must be restricted to silence-only periods. Several ideas were explored for the first section. The most robust solution, and the one with the best sound quality, was given by the simplest solution, i.e. a delay and sum beamformer that cues in on the direct path only and neglects all multipath contributions. Tests were performed with a four-microphone array in a highly reverberant room with both music and fan type noise as jammers, SNR improvements of 10 dB were typical with no audible distortion.>

A system which combines the good short-time classification properties of the time delay neural network (TDNN) with the good integration and overall recognition capabilities of hidden Markov models (HMMs) is proposed for a speaker-independent speech recognizer. The standard vector quantization is replaced by a TDNN labeler giving phonelike labels. In order to avoid hand segmentation for the training of the TDNN, a separate HMM and a Viterbi alignment derived from it are used. This gives a coarse phonetic segmentation of the training data.>

A novel algorithm is presented for the estimation of a signal in noise. The distortion criterion used is based on the distance between log spectra. In many signal-processing applications, such as speech recognition, log spectra are much closer to the parameters used in a discriminator than power spectra. Therefore, it is believed that this spectral estimation technique should lead to better results than previously developed techniques such as spectral subtraction. The present technique performed better than spectral subtraction in noise immunity experiments on the IBM isolated word speech-recognition system, although at the expense of additional computational requirements.>

The Speech Recognition Group at IBM Research in Yorktown Heights has developed a real-time, isolated-utterance speech recognizer for natural language based on the IBM Personal Computer AT and IBM Signal Processors. The system has recently been enhanced by expanding the vocabulary from 5,000 words to 20,000 words and by the addition of a speech workstation to support usability studies on document creation by voice. The system supports spelling and interactive personalization to augment the vocabularies. This paper describes the implementation, user interface, and comparative performance of the recognizer.

This paper presents several ways of making the signal processing in the IBM speech recognition system more robust with respect to variations in the background noise level. The underlying problem is that the speech recognition system trains on the specific noise circumstances of the training session. A simple solution lays in the controlled addition of noise. The level of noise that has to be added in to effectively mask all background noise is rather high and causes a significant reduction in accuracy. Spectral subtraction does a better job in a limited number of cases, but the thresholding in spectral subtraction often leads to training problems in the hidden Markov model based recognition system. The best results were obtained by reintroducing a semi-natural background by adding noise after applying spectral subtraction.

This paper describes a computational model of the cochlea with the appropriate interface for a cochlear prosthesis. This interface consists of two parts. First neural activity is mapped to electrical current density at the neurons. Then the optimal stimulation pattern is derived through a least squares fit, given the current spread function between the electrodes and the nerve endings. Practical limitations which limit the usage of this scheme, as well as the first tests performed with our patients, are described.