An algorithm for the generation of word graphs in a cross-word decoder that uses long span m-gram language models (LMs) is presented. The generation of word hypotheses within the graph relies on the word m-tuple-based boundary optimization. The graphs contain the full word history knowledge information since the graph structure reflects all LM constraints used during the search. This results in better word boundaries and in enhanced capabilities to prune the graphs. Furthermore, the memory costs for expanding the m-gram constrained word graphs to apply very long span LMs (e.g. ten-grams that are constructed by log linear LM combination) are considerably reduced. Experiments for lattice generation and rescoring have been carried out on the 5K-word WSJ task and the 64K-word NAB task.

The subject of this paper is the integration of the traditional vector quantizer (VQ) and discrete hidden Markov models (HMM) combination in the mixture emission density framework commonly used in automatic speech recognition (ASR). It is shown that the probability density of a system that consists of a VQ and a discrete classifier can be interpreted as a special case of a semi-continuous mixture model. Thus, the VQ parameters and the classifier can be trained jointly. In this framework, a gradient based VQ training method for single and multiple feature stream systems is derived. This leads to an approach that is directly related to the paradigm of maximum mutual information (MMI) neural networks, that has been successfully applied as VQ in ASR earlier. In continuous speech recognition experiments that were carried out for the Resource Management and Wall Street Journal databases the presented systems achieve recognition accuracies that compete well with comparable Gaussian mixture HMMs. Thus, we demonstrate that the performance degradations, often reported for discrete HMM systems, are not mainly caused by the vector quantization process in itself, but that they are due to the traditional separation of the VQ and the HMM during parameter estimation. These degradations can be avoided by training of the entire system as described here, while keeping the attractive computational speed of discrete HMMs.

With this paper, we present the DUcoder, the LVCSR decoder developed at Duisburg University. The decoder performs the Viterbi search for the most probable word sequence in recognition systems that make use of HMMs and backo N-gram language models. In principle, the decoding strategy is similar to the one of the so-called stackdecoders. During the development of the decoder, emphasis has been laid upon innovations for rapidly speeding up decoding by carefully performing approximations. Besides a brief presentation of the decoder's overall design, this paper points out the crucial issues with respect to speed and recognition performance. Evaluations are carried out on a German LVCSR system with a vocabulary of 100; 000 words, wordinternal triphones and a trigram language model. Closeto -real-time performance is achieved with 12% additional error while a decoder conguration which runs in around 40 times real-time causes no search error on the evaluation set. 1. INTRODUCTION The realiz...

A new approach to identification of handwritten symbols in arbitrary complex environments is presented. 20 different pictograms drawn in different backgrounds can be identified with a recognition accuracy of 90%. In order to perform this challenging task, we use pattern spotting techniques based on pseudo 2-D hidden Markov models (P2DHMMs). Practical applications of our approach can be found in many typical multimedia document processing tasks, such as localization and recognition of non-rigid objects in image databases, detection of objects in complex scenes, finding trademarks in presence of clutter within videos, processing distorted document images in digital libraries, or content-based image retrieval based on handwritten query symbols.

The paper deals with the problem of extracting topic information from news show stories by statistical methods. It is shown that the traditional topic-dependent n-gram language modeling approach can be decomposed in order to apply neural networks for topic indexing. Two specific problems in training of these networks are addressed: a very sparse data distribution in the stories and the superposition of different topics in a story. The first problem is tackled by an integrated smoothing approach in the backpropagation method; an expansion of the neural network structure can be used to cope with topic mixtures in stories. Due to the efficient parameter sharing the application of neural networks results in a small improvement in topic indexing performance on a small corpus of broadcast news compared to the traditional topic-dependent n-gram method.

Decision tree-based state clustering has emerged in as the most popular approach for clustering the states of context dependent hidden Markov model based speech recognizers. The application of sets of phones, mainly phonetically motivated, that limit the possible clusters, results in a reasonably good modeling of unseen phones while it still enables to model specific phones very precisely whenever this is necessary and enough training data is available. Formal concept analysis, a young mathematical discipline, provides means for the treatment of sets and sets of sets that are well suited for further improving tree-based state clustering. The possible refinements are outlined and evaluated in this paper. The major merit is the proposal of procedures for the adaptation of the number of sets used for clustering to the amount of available training data, and of a method that generates suitable sets automatically without the incorporation of additional knowledge.

This paper describes, how to perform speaker adaptation for a hybrid large vocabulary speech recognition system. The hybrid system is based on a Maximum Mutual Information Neural Network (MMINN), which is used as a Vector Quantizer (VQ) for a discrete HMM speech recognizer. The combination of MMINNs and HMMs has shown good performance on several large vocabulary speech recognition tasks like RM and WSJ. This paper now presents two approaches to perform speaker adaptation with this hybrid system. The first approach is a transformation of the feature space, which is performed by a neural network with maximum likelihood (ML) as objective function for the complete system, which means, that the parameters of the NN are estimated in order to match the HMM-parameters of the pre-trained speaker independent system. The second approach is to adapt the HMM parameters depending on the amount of training data available per HMM, using a regularization approach. Both approaches can be applied join...

In previous work about hybrid speech recognizers with discrete HMMs we have shown that VQs, that are trained according to a maximum mutual information (MMI) criterion, are well suited for ML estimated Bayes classifiers. This is only valid for single VQ systems. In this paper we extend the theory to speech recognizers with multiple VQs. This leads to a joint training criterion for arbitrary multiple neural VQs that considers the inter VQ correlation during parameter estimation. The idea of a gradient based joint training method is derived. Experimental results indicate that inter VQ correlations can cause some degradation of recognition performance. The joint multiple VQ training decorrelates the quantizer labels and improves system performance. In addition the new training criterion allows for a less careful way of splitting up the feature vector into multiple streams that do not have to be statistically independent. In particular the use of highly correlated features in conjunction with the novel training criterion in the experiments leads to important gains in recognition performance for the speaker independent resource management database and gives the lowest error rate of 5.0% we ever obtained in this framework.

We present a new adaptation technique for our hybrid large vocabulary continuous speech recognition system. In most adaptation approaches the HMM parameters are reestimated. In our approach, however, we train a speaker independent continuous speech recognizer, then we keep the HMM parameters fixed and we train a second network, which transforms the features of the adaptation data to fit the HMM parameters. Thus, less parameters have to be estimated, and therefore this approach performs well even for a small number of adaptation data. With this approach we achieve relative improvements in recognition rates on the Wall Street Journal (WSJ) task of 16.5%.

In this paper we present the methods we developed to estimate posterior probabilities for HMM states in continuous and discrete HMM-based speech recognition systems and several ways to speed up decoding by using these posterior probability estimates. The proposed pruning techniques are state deactivation pruning (SDP), similar to an approach proposed for hybrid recognition systems, and a novel posteriori-based lookahead technique, posteriori lookahead pruning (PLP), that evaluates future posteriors in order to exclude unlikely HMM states as early as possible during search. By applying the proposed methods we managed to vastly reduce the decoding time consumed by our time-synchronous Viterbi-decoder for recognition systems based on the Verbmobil and the Wall Street Journal database with hardly any additional search error.

This paper introduces a method for regularization of HMM systems that avoids parameter overfitting causedby insufficient training data. Regularization is done by augmenting the EM training method by a penalty term that favors simple and smooth HMM systems. The penalty term is constructed as a mixture model of negative exponential distributions that is assumed to generate the state dependent emission probabilities of the HMMs. This new method is the successful transfer of a well known regularization approachin neural networks to the HMM domain andcan be interpreted as a generalization of traditional state-tying for HMM systems. The effect of regularization is demonstrated for continuous speech recognition tasks by improving overfitted triphone models and by speaker adaptation with limited training data. 1. INTRODUCTION A general problem when constructing statistical pattern recognition systems is to ensure the capability to generalize well, i.e. the system must be able to classify dat...

In this paper, we describe our work on the field of confidence measures for HMM-based speech recognition. Confidence measures are a means of estimating the recognition reliability for single words of the recognizer output. The possible applications of such measures are manifold. We present our experiments with well known approachesand proposesome new ones. Particularly, we propose to combine the mere acoustical measures with language model-based ones for continuous speech recognition that involves a stochastic language model. This slightly improves the acoustical measures and preserves their advantage of being computationally very cheap. Experiments are carried out on a German isolated word recognition system and on continuous speech recognition systems for the Resource Management database and the Wall Street Journal WSJ0 task. 1. INTRODUCTION Word-based confidencemeasures for speechrecognition basedon hidden Markov models (HMMs) have for some years now been an important research top...

This paper deals with the construction and optimization of a hybrid speech recognition system that consists of a combination of a neural vector quantizer (VQ) and discrete HMMs. In our investigations an integration of VQ based classification in the continuous classifier framework is given and some constraints are derived that must hold for the PDFs in the discrete pattern classifier context. Furthermore it is shown that for ML training of the whole system the VQ parameters must be estimated according to the maximum mutual information (MMI) criterion. A novel training method based on gradient search for neural networks that serve as optimal VQ is derived. This allows faster training of arbitrary network topologies compared to the traditional MMI-NN training. An integration of multilayer MMI-NNs as the VQ in the hybrid discrete HMM based speech recognizer leads to a large improvement compared to other supervised and unsupervised single layer VQ systems. For the speaker independent Resource Management database the constructed hybrid MMI-connectionist/HMM system achieves recognition rates that are comparable to traditional sophisticated continuous PDF HMM systems.

A hybrid MMI-connectionist/hidden Markov model (HMM) speech recognition system for the Wall Street Journal (WSJ) database is presented. The HMM part of this system uses discrete probability density functions (PDF). The neural network (NN) is used to replace a classical vector quantizer (VQ) like a k-means or LBG algorithm, which are typically used in discrete HMM systems. The NN is trained on an algorithm, that tries to achieve maximum mutual information (MMI) between the generated output labels and the underlying phonetic description. The system has been trained and tested with the five thousand word speaker independent WSJ task. The error rates of the MMI-connectionist approach are 21% lower than the error rates of a k-means system. The system achieves error rates which have been achieved before only by the best continuous/semi-continuous HMM speech recognizers, with the advantage of a faster recognition algorithm.

The estimation of the HMM parameters has always been a major issue in the design of speech recognition systems. Discriminative objectives like maximum mutual information (MMI) or minimum classification error (MCE) have proved to be superior over the common maximum likelihood estimation (MLE) in cases where a robust estimation of the probabilistic density functions (PDFs) is not possible. The determination of the overall likelihood of an acoustic observation is the most crucial point of the MMI-parameter estimation when applied to continuous speech systems. Contrary to the common approaches that estimate the overall likelihood of the training observations by evaluating the most confusing sentences or by applying global state frequencies, this paper suggests a dictionary analysis in order to get estimates for the dictionary-based risk of mixing two HMM states. These estimates are used to estimate the observations' likelihood and to control the discriminative MMI training procedure. Results on a monophone SCHMM speech recognition system are presented that prove the practicability of the new approach.

In this paper we present a context dependent hybrid MMI-connectionist / Hidden Markov Model (HMM) speech recognition system for the Wall Street Journal (WSJ) database. The hybrid system is build with a neural network, which is used as a vector quantizer (VQ) and an HMM with discrete probablility density functions, which has the advantage of a faster decoding. The neural network is trained on an algorithm, that tries to maximize the mutual information between the classes of the input features (e.g. phones, triphones, etc.) and the neural firing sequence of the network.

We present a hybrid speech recognition system for speaker independent continuous speech recognition. The system combines a novel information theory based neural network (NN) paradigm and discrete Hidden Markov models (HMMs) including state-of-the-art techniques like state clustered triphones. The novel NN type is trained by an algorithm based on principles of self-organization that achieves maximum mutual information between the generated output labels and the basic phonetic classes. The structure of the hybrid system is quite similar to a classical VQ-HMM system but the vector quantizer (VQ) is replaced by the NN. To evaluate the system we use the speaker independent part of the resource management (RM) database. We obtained an important improvement by introducing a novel kind of context dependent basic classes used by the acoustic processor. The average RM recognition result with a word-pair grammar is now 95.2% what is significantly better than a classical VQ-system, slightly better than a different hybrid system with a recurrent network as probability estimator, and very close to the best continuous probability density function (PDF) HMM speech recognizers.

This paper presents the results of the comparison of continuous and discrete density hidden Markov models (HMMs) used for cursive handwriting recognition. For comparison, a subset of a large vocabulary (1000 word), writer-independent online handwriting recognition system for word and sentence recognition was used, which was developed at Duisburg University. This system has some unique features that are rarely found in other HMM-based character recognition systems, such as: (1) option between discrete, continuous, or hybrid modeling of HMM probability density distributions; (2) large vocabulary recognition based on either printed or cursive word or complete sentence input; (3) optimized HMM topology with an unusually large number of HMM states; and (4) use of multiple label streams for coding of handwritten information. Emphasis in this paper is on the comparison between continuous and discrete density HMMs, since this is still an open question in handwriting recognition, and is crucial for the future development of the system. However, in order to give a complete description of the basic system architecture, some of the above mentioned issues are also addressed. The surprising result of our investigation was the fact that discrete density models led to better results than continuous models, although this is generally not the case for HMM-based speech recognition systems. With the optimized system, a 70% word recognition rate was obtained for a challenging large-vocabulary, writer-independent sentence input task.

This paper presents a new hybrid system for speaker independent continuous speech recognition in a large vocabulary task. The hybrid system is a combination of context dependent discrete Hidden Markov Models and artificial neural networks that are trained by an information theory based algorithm. This algorithm maximizes the Mutual Information (MMI) between the network output and the phone descriptions by applying a self-organizing learning approach instead of forcing constrained network outputs. Recognition results have shown that the new hybrid system outperforms a classical k-means-VQ-based HMM-system. For the speaker independent DARPA Resource Management (RM) task (perplexity 60) we report a decrease in word recognition error rate up to 35% (close to the best continuous pdf systems). 1. INTRODUCTION Research during recent years has shown that Hidden Markov Models (HMMs) are the most successful approach for the recognition of speaker-independent continuous speech (e.g. see [1]). B...