To provide an efficient means of communication for those who cannot move muscles of the whole body except eyes due to amyotrophic lateral sclerosis (ALS), we are developing a speech synthesis interface that is based on electrooculogram (EOG) input.EOG is an electrical signal that is observed through electrodes attached on the skin around eyes and reflects eye position.A key component of the system is a continuous recognizer for the EOG signal.In this paper, we propose and investigate a hidden Markov model (HMM) based EOG recognizer applying continuous speech recognition techniques.In the experiments, we evaluate the recognition system both in user dependent and independent conditions.It is shown that 96.1% of recognition accuracy is obtained for five classes of eye actions by a user dependent system using six channels.While it is difficult to obtain good performance by a user independent system, it is shown that maximum likelihood linear regression (MLLR) adaptation helps for EOG recognition.

It is expensive to prepare a sufficient amount of training data for acoustic modeling for developing large vocabulary continuous speech recognition systems. This is a serious problem especially for resource-deficient languages. We propose an active learning method that effectively reduces the amount of training data without any degradation in recognition performance. It is used to design a text corpus for read speech collection. It first estimates phone-error distribution using a small amount of fully transcribed speech data. Second, it constructs a sentence set whose phone-occurrence distribution is close to the phone-error distribution and collects its speech data. It then extends this process to diphones and triphones and collects more speech data. We evaluated our method with simulation experiments using the Corpus of Spontaneous Japanese. It required only 76 h of speech data to achieve word accuracy of 74.7%, while the conventional training method required 152 h of data to achieve the same rate.

In recent years, adaptation techniques have been given special focus in speaker recognition tasks, mainly targeting speaker and session variation disentangling under the Maximum a Posteriori (MAP) criterion. For these techniques, unseen mixtures are usually adapted in a global manner, if ever. In this paper, we explore Structural MAP (SMAP), Maximum a Posteriori adaptation using hierarchical structures of the acoustic space that allow data scarceness issues to be tackled with different precision levels. We explore this approach in a speaker verification system using a Support Vector Machine (SVM) classifier and Gaussian mean supervectors (GMM-SVM). We show that this is an effective approach that considerably outperforms its relevance MAP counterpart in the 2006 NIST Speaker Recognition Evaluation. We also show that using a speaker-adapted Universal Background Model can improve the stability of the clustering algorithm besides obtaining further improvements.

We propose Cross-Channel Spectral Subtraction (CCSS), a source separation method for recognizing meeting speech where one microphone is prepared for each speaker. The method quickly adapts to changes in transfer functions and uses spectral subtraction to suppress the speech of other speakers. Compared with conventional source separation methods based on independent component analysis (ICA) or that use binary masks, it requires less computational costs and the resulting speech signals have less distortion. In a recognition task of computer-simulated, partially-overlapped speech, CCSS improved the word accuracy from 66.5% to 77.7%. It also significantly improved the recognition accuracy of speech data in actual meetings.

In speaker verification, structural maximum-a-posteriori (SMAP) adaptation for Gaussian mixture model (GMM) has been proven effective, especially when the speech segment is very short.In SMAP adaptation, an acoustic tree of Gaussian components is constructed to represent the hierarchical acoustic space.Until now, however, there has been no clear way to automatically find the optimal tree structure for a given speaker.In this paper, we propose using an acoustic forest, which is a set of trees, for SMAP adaptation, instead of a single tree.In this approach, we combine the results of SMAP adaptation systems with different acoustic trees.A key issue is how to combine the trees.We explore three score fusion techniques, and evaluate our approach in the text-independent speaker verification task of the NIST 2006 SRE plan using 10-second speech segments.Our proposed method decreased EER by 3.2% from the relevant MAP adaptation and by 1.6% from the conventional SMAP with a single tree.

Research and development in the field of spoken language depends critically on the existence of software tools.A large range of excellent tools have been developed and are widely used today.Most tools were developed by individuals who recognized the need for a given tool, had the necessary conceptual and programming skills, and were deeply rooted in the application field, namely spoken language.Excellent tools are a prerequisite to research.However, tool developers rarely receive academic recognition for their efforts.Journals, conferences and funding agencies are interested in the results of the work on a research question while the tools developed to achieve these results are of less interest.This makes it difficult to publish articles on tools, and it is next to impossible to obtain funding for their development.The Interspeech 2011 special event on speech processing tools aims to provide a forum for tool developers to improve their academic visibility and thus enhance their motivation to continue developing the software needed by the community.This year, the focus is on interoperability -how can different tools be integrated into the workflow, data be exchanged between tools, queries work across tools, and a consistent user interface be achieved.

A general framework of language model task adaptation is to select documents in a large training set based on a language model estimated on a development data.However, this strategy has a deficiency that the selected documents are biased to the most frequent patterns in the development data.To address this problem, a new task adaptation method is proposed that selects documents in the training set so as to directly reduce the perplexity on the development set.Moreover, a weighting method to modify the perplexity objective function is proposed to improve the generalization to unseen data.The proposed adaptation methods are evaluated by large vocabulary speech recognition experiments.It is shown that the proposed adaptation with the weighting term produces a compact-size model that gives consistently lower word error rates for different tasks.

We propose a committee-based active learning method for large vocabulary continuous speech recognition. In this approach, multiple recognizers are prepared beforehand, and the recognition results obtained from them are used for selecting utterances. Here, a progressive search method is used for aligning sentences, and voting entropy is used as a measure for selecting utterances. We apply our method not only to acoustic models but also to language models and their combination. Our method was evaluated by using 190-hour speech data in the Corpus of Spontaneous Japanese. It proved to be significantly better than random selection. It only required 63 h of data to achieve a word accuracy of 74%, while standard training (i.e., random selection) required 97 h of data. The recognition accuracy of our proposed method was also better than that of the conventional uncertainty sampling method using word posterior probabilities as the confidence measure for selecting sentences.

We have previously proposed unsupervised cross-validation (CV) adaptation that introduces CV into an iterative unsupervised batch mode adaptation framework to suppress the influence of errors in an internally generated recognition hypothesis and have shown that it improves recognition performance. However, a limitation was that the experiments were performed using only a clean speech recognition task with a ML trained initial acoustic model. Another limitation was that only the CV method was investigated while there was a possibility of using other ensemble methods. In this study, we evaluate the CV method using a discriminatively trained baseline and a noisy speech recognition task. As an alternative to CV adaptation, unsupervised aggregated (Ag) adaptation is proposed and investigated that introduces a bagging like idea instead of CV. Experimental results show that CV and Ag adaptations consistently give larger improvements than the conventional batch adaptation but the former is more advantageous in terms of computational cost.

Variations in walking speed have a strong impact on the recognition of gait. We propose a method of recognition of gait that is robust against walking-speed variations. It is established on a combination of Fisher discriminant analysis (FDA)-based cubic higher-order local auto-correlation (CHLAC) and the statistical framework provided by hidden Markov models (HMMs). The HMMs in this method identify the phase of each gait even when walking speed changes nonlinearly, and the CHLAC features capture the within-phase spatio-temporal characteristics of each individual. We compared the performance of our method with other conventional methods in our evaluation using three different databases, i.e., USH, USF-NIST, and Tokyo Tech DB. Ours was equal or better than the others when the speed did not change too much, and was significantly better when the speed varied across and within a gait sequence.

We propose a statistical framework for high-level feature extraction that uses SIFT Gaussian mixture models (GMMs) and audio models. SIFT features were extracted from all the image frames and modeled by a GMM. In addition, we used mel-frequency cepstral coefficients and ergodic hidden Markov models to detect high-level features in audio streams. The best result obtained by using SIFT GMMs in terms of mean average precision on the TRECVID 2009 corpus was 0.150 and was improved to 0.164 by using audio information.

This paper describes an investigation into current browser based runtimes including Adobe’s Flash and Microsoft’s Silverlight as platforms for delivering web based speech interfaces. The key difference here is the browser plugin is used to perform all the computation without any server side processing. The first application is an HMM based text-to-speech engine running in the Adobe Flash plugin. The second application is a WFST based large vocabulary speech recognition decoder written in C# running inside the Silverlight plugin.

In this paper we perform a cross-comparison of the T 3 WFST decoder against three different speech recognition decoders on three separate tasks of variable difficulty. We show that the T 3 decoder performs favorably against several established veterans in the field, including the Juicer WFST decoder, Sphinx3, and HDecode in terms of RTF versus Word Accuracy. In addition to comparing decoder performance, we evaluate both Sphinx and HTK acoustic models on a common footing inside T 3 , and show that the speed benefits that typically accompany the WFST approach increase with the size of the vocabulary and other input knowledge sources. In the case of T 3 , we also show

We previously proposed a decoding method for automatic speech recognition utilizing hypothesis scores weighted by voice activity detection (VAD)-measures. This method uses two Gaussian mixture models (GMMs) to obtain confidence measures: one for speech, the other for non-speech. To achieve good search performance, we need to adapt the GMMs properly for input utterances and environmental noise. We describe a new unsupervised on-line GMM adaptation method based on MAP estimation. The robustness of our method is further improved by weighting updating parameters of GMMs according to the confidence measure for the adaptation data. We also describe an approach to accelerate the adaptation by caching statistical values to adapt GMMs. Experimental results on Drivers’ Japanese Speech Corpus in a Car Environment (DJSC) show that the adaptation with decoding method significantly improves the word accuracy from 54.8% to 59.6%. Moreover, the weighting method improves the robustness of the unsupervised adaptation, and the cache method greatly accelerates the decoding process. Consequently, our adaptive decoding method significantly improves the word accuracy in a noisy environment with only a minor increase in the computational cost. Index Terms: speech recognition, voice activity detection, Gaussian mixture model adaptation

Since speech is highly variable, even if we have a fairly large-scale database, we cannot avoid the data sparseness problem in constructing automatic speech recognition (ASR) systems. How to train and adapt statistical models using limited amounts of data is one of the most important research issues in ASR. This paper summarizes major techniques that have been proposed to solve the generalization problem in acoustic model training and adaptation, that is, how to achieve high recognition accuracy for new utterances. One of the common approaches is controlling the degree of freedom in model training and adaptation. The techniques can be classified by whether a priori knowledge of speech obtained by a speech database such as those spoken by many speakers is used or not. Another approach is maximizing ¿margins¿ between training samples and the decision boundaries. Many of these techniques have also been combined and extended to further improve performance. Although many useful techniques have been developed, we still do not have a golden standard that can be applied to any kind of speech variation and any condition of the speech data available for training and adaptation.

In this paper we present a fast method for computing acoustic likelihoods that makes use of a graphics processing unit (GPU). After enabling the GPU acceleration the main processor runtime dedicated to acoustic scoring tasks is reduced from the largest consumer to just a few percent even when using mixture models with a large number of Gaussian components. The results show a large reduction in decoding time with no change in accuracy and we also show by using a 16bit half precision floating point format for the acoustic model parameters, storage requirements can be halved with no reduction in accuracy.

Independent component analysis (ICA) is not only popular for blind source separation but also for unsupervised learning when the observations can be decomposed into some independent components. These components represent the specific speaker, gender, accent, noise or environment, and act as the basis functions to span the vector space of the human voices in different conditions. Different from eigenvoices built by principal component analysis, the proposed independent voices are estimated by ICA algorithm, and are applied for efficient coding of an adapted acoustic model. Since the information redundancy is significantly reduced in independent voices, we effectively calculate a coordinate vector in independent voice space, and estimate the hidden Markov models (HMMs) for speech recognition. In the experiments, we build independent voices from HMMs under different noise conditions, and find that these voices attain larger redundancy reduction than eigenvoices. The noise adaptive HMMs generated by independent voices achieve better recognition performance than those by eigenvoices.

In the Weighted Finite State Transducer (WFST) framework for speech recognition, we can reduce memory usage and increase flexibility by using on-the-fly composition which generates the search network dynamically during decoding. Methods have also been proposed for optimizing WFSTs in on-the-fly composition, however, these operations place restrictions on the structure of the component WFSTs. We propose extended on-the-fly optimization operations which can operate on WFSTs of arbitrary structure by utilizing a filter composition. The evaluations illustrate the proposed method is able to generate more efficient WFSTs.

An unsupervised cross-validation adaptation algorithm and its variation are proposed that introduce the idea of cross-validation in the unsupervised batch-mode adaptation framework to improve the adaptation performance. The first algorithm is constructed on a general adaptation technique such as MLLR and can be used in combination with any adaptation method. The second algorithm is a modified version of the first algorithm and works with lower computational cost by assuming MLLR. These algorithms are extensions of our previously proposed CV training methods and are useful to suppress the negative effect of the conventional unsupervised batch-mode adaptation process that reinforces the errors included in automatic transcriptions. The proposed algorithms were evaluated in domain adaptation, speaker adaptation, and in their combination for large vocabulary spontaneous speech recognition. When the domain and speaker adaptations were combined using a read speech initial model, the relative word error rate reduction by the proposed method was 29% whereas the reduction by the conventional approach was 23%.

This paper summarizes my 40 years of research on speech and speaker recognition, focusing on selected topics that I have investigated at NTT Laboratories, Bell Laboratories and Tokyo Institute of Technology with my colleagues and students. These topics include: the importance of spectral dynamics in speech perception; speaker recognition methods using statistical features, cepstral features, and HMM/GMM; text-prompted speaker recognition; speech recognition using dynamic features; Japanese LVCSR; robust speech recognition; spontaneous speech corpus construction and analysis; spontaneous speech recognition; automatic speech summarization; and WFST-based decoder development and its applications.

In a speech recognition system a Voice Activity Detector (VAD) is a crucial component for not only maintaining accuracy but also for reducing computational consumption. Front-end approaches which drop non-speech frames typically attempt to detect speech frames by utilizing speech/non-speech classification information such as the zero crossing rate or statistical models. These approaches discard the speech/non-speech classification information after voice detection. This paper proposes an approach that uses the speech/non-speech information to adjust the score of the recognition hypotheses. Experimental results show that our approach can improve the accuracy significantly and reduce computational consumption by combining the frontend method. Index Terms: speech recognition, voice activity detection, decoder

We propose a two-step active learning method for supervised speaker adaptation. In the first step, the initial adaptation data is collected to obtain a phone error distribution. In the second step, those sentences whose phone distributions are close to the error distribution are selected, and their utterances are collected as the additional adaptation data. We evaluated the method us- ing a Japanese speech database and maximum likelihood linear regression (MLLR) as the speaker adaptation algorithm. We confirmed that our method had a significant improvement over a method using randomly chosen sentences for adaptation. Index Terms: speech recognition ,speaker adaptation, active learning nition accuracies in comparison with the conventional adapta- tion frameworks when the initial speaker-independent acoustic model has already been tuned to the given recognition task. In this paper, we propose a two-step active learning method for supervised speaker adaptation. In the first step, our method collects a small amount of utterances from a user to obtain his/her tendency in speech recognition errors. In the second step, it selects those sentences rich in phonetic units in the er- rors from a sentence pool, and it collects their utterances as ad- ditional data for adaptation. Since our method directly aims at decreasing recognition errors, it is expected to be highly dis- criminative. Our method has two critical issues: One is how to relate the recognition errors to the selection criterion of adap- tation sentences. The other is how to set the size of the initial adaptation data in the first step; it should be as small as possi- ble in order to decrease the user's effort, but it must be sufficient enough to estimatethe tendency of errors precisely. Wedescribe our approach to resolving these issues in this paper. This paper is organized as follows. Section 2 explains our method, and Section 3 briefly explains the MLLR speaker adap- tation method. Section 4 reports our evaluation experiments us- ing a Japanese speech database, and Section 5 concludes the paper.

Abstract To improve speech recognition performance in adverse condi-tions, a noise compensation method is proposed that applies atransformation in the spectral domain whose parameters are op-timized based on likelihood of speech GMM modeled on thefeature domain. The idea is that additive and convolutionalnoises have mathematically simple expression in the spectraldomain while speech characteristics are better modeled in thefeature domain such as MFCC. The proposed method worksas a feature extraction front-end that is independent from de-coding engine, and has ability to compensate for non-stationaryadditive and convolutional noises with a short time delay. Itincludes spectral subtraction as a special case when no param-eter optimization is performed. Experiments were performedusing the AURORA-2J database. It has been shown that signif-icantly higher recognition performance is obtained by the pro-posed method than spectral subtraction. Index Terms : noisy speech recognition, spectrum, Gaussianmixture model

Long organization names are often abbreviated in spoken Chinese, and abbreviated utterances cannot be recognized correctly if the abbreviations are not included in the recognition vocabulary. Therefore, it is very important to automatically generate and add abbreviations for organization names to the vocabulary. Generation of Chinese abbreviations is much more complex than English abbreviations which are mostly acronyms and truncations. In this paper, we propose a new hybrid method for automatically generating Chinese abbreviations and we perform vocabulary expansion using output of the abbreviation model for voice search. In our abbreviation modeling, we treat the abbreviation generation problem as a tagging problem and use conditional random fields (CRF) as the tagging tool, the output of which is then re-ranked by a length model and web information. In the vocabulary expansion, considering the multiple abbreviation phenomenon and limited coverage of the top-1 abbreviation candidate, we add top-10 candidates into the vocabulary. In our experiments, for the abbreviation modeling, we achieved a top-10 coverage of 88.3% with the proposed method. For the voice search using abbreviated utterances, we improved the full-name search accuracy from 16.9% to 79.2% by incorporating the top-10 abbreviation candidates to the vocabulary.

Independent component analysis (ICA) is a popular approach for blind source separation (BSS). In this study, we develop a new mutual information measure for BSS and unsupervised learning of acoustic models. The underlying concept of ICA unsupervised learning algorithm is to demix the observations vectors and identify the corresponding mixture sources. These independent sources represent the specific speaker, gender, accent, noise or environment, etc, embedded in acoustic models. The novelty of the proposed ICA is to derive a new metric of mutual information for measuring the dependence among mixture sources. We focus on building this metric based on the Jensen’s inequality, which is illustrated to use smaller number of iterations in finding the demixing matrix compared to other types of mutual information. We present a parametric ICA using the generalized Gaussian distribution to characterize the non-Gaussianity of model parameters. Also, a nonparametric ICA is established by using the Parzen window based distribution. In the experiments on BSS and noisy speech recognition, we demonstrate the effectiveness of the proposed Jensen ICA compared to FastICA and other nonparametric ICA.

We propose a robust spoken term detection method against word recognition errors using a combination of phone-based and word-based recognition.Conventional methods based on similar frameworks are problematic because phone-based recognition produces a large number of insertion errors.In our method, different substitution penalties are assigned for phone pairs to reduce such errors.We evaluated our method using the corpus of spontaneous Japanese.When recall was fixed at 50%, precision improved to 4.4 points above detection using only word-based recognition.We also report here on the effectiveness of optimization of the combination weight for each keyword.

When using Weighted Finite State Transducers (WFSTs) in speech recognition, on-the-fly composition approaches have been proposed as a method of reducing memory consumption and increasing flexibility during decoding.We have recently implemented several fast on-the-fly techniques, namely avoiding dead-end states, dynamic pushing and state sharing in our decoding engine.The goal of this paper is to provide a unified study of how the different on-the-fly techniques and online composition combinations effect speech recognition performance.The evaluations were performed on a large spontaneous speech recognition task and the results show that when using on-the-fly composition with a fully dynamically composed language model component the performance degrades substantially even when avoiding dead-end states.We then show in these cases the recognition performance can be dramatically improved with the addition of dynamic pushing and state sharing.

We have previously proposed a cross-validation (CV) based Gaussian mixture optimization method that efficiently optimizes the model structure based on CV likelihood.In this study, we propose aggregated cross-validation (AgCV) that introduces a bagging-like approach in the CV framework to reinforce the model selection ability.While a single model is used in CV to evaluate a held-out subset, AgCV uses multiple models to reduce the variance in the score estimation.By integrating AgCV instead of CV in the Gaussian mixture optimization algorithm, an AgCV likelihood based Gaussian mixture optimization algorithm is obtained.The algorithm works efficiently by using sufficient statistics and can be applied to large models such as Gaussian mixture HMM.The proposed algorithm is evaluated by speech recognition experiments on oral presentations and it is shown that lower word error rates are obtained by the AgCV optimization method when compared to CV and MDL based methods.

The segmental eigenvoice method has been proposed to provide rapid speaker adaptation with limited amounts of adaptation data.In this method, the speaker-vector space is clustered to several subspaces and PCA is applied to each of the resulting subspaces.In this paper, we propose two new techniques to improve the performance of this segmental eigenvoice approach.First, we propose a soft-clustering method in which each element in a speaker vector can be assigned to more than one cluster.Second, those elements far apart from any of the clusters are removed.Our experiments using the JNAS and S-JNAS databases show that the proposed method outperforms both the original eigenvoice and the segmental eigenvoice methods, e.g., 3.3% average improvement when only 10 utterances are used for adaptation.

In a previous study, we developed an interface using semisynchronous speech and pen input.In this interface, a user speaks while writing, and the pen input complements the speech, enabling a higher recognition performance than with speech alone.When a user inputs speech and pen, there is a time lag between the two modes, and the lag differs among users.We propose a method for adapting to the different time lags of individual users.This method was evaluated in a Japanese continuous speech recognition task with three different pen-input interfaces including a QWERTY keyboard interface.The time-lag adaptation improved recognition accuracies by up to 0.5 point.

In this paper we present evaluations on the large vocabulary speech decoder we are currently developing at Tokyo Institute of Technology. Our goal is to build a fast, scalable, flexible decoder to operate on weighted finite state transducer (WFST) search spaces. Even though the development of the decoder is still in its infancy we have already implemented a impressive feature set and are achieving good accuracy and speed on a large vocabulary spontaneous speech task. We have developed a technique to allow parts of the decoder to be run on the graphics processor, this can lead to a very significant speed up.

Waseda University, Tokyo Institute of Technology, and six companies, Asahi-kasei, Hitachi, Mitsubishi, NEC, Oki and Toshiba, initiated a three year project in 2006 supported by the Ministry of Economy, Industry and Trade (METI), Japan, for jointly developing fundamental automatic speech recognition (ASR) technology. The project focuses on utilizing ASR technology in car and home environments. Seven subtasks are being investigated: speech/non-speech separation using multiple microphones, speech/non-speech separation for a single audio stream, developing a high-performance WFST-based decoder, multi-lingual ASR modeling, higher-order language modeling, developing a system for assisting speech interface development, and overall technology evaluation. This talk will give an overview of the intermediate technological progress achieved by the project.

This paper presents a language modeling approach to sentence retrieval for Question Answering (QA) that we used in Question Answering on speech transcripts (QAst), a pilot task at the Cross Language Evaluation Forum (CLEF) evaluations 2007. A language model (LM) is generated for each sentence and these models are combined with document LMs to take advantage of contextual information. A query expansion technique using class models is proposed and included in our framework. Finally, our method's impact on exact answer extraction is evaluated. We show that combining sentence LMs with document LMs significantly improves sentence retrieval performance, and that this sentence retrieval approach leads to better answer extraction performance.

We focus on the problem of speech recognition in the presence of nonstationary sudden noise, which is very likely to happen in home environments. As a model compensation method for this problem, we investigated the use of factorial hidden Markov model (FHMM) architecture developed from a clean-speech hidden Markov model (HMM) and a sudden-noise HMM. While in conventional studies this architecture is defined only for static features of the observation vector, we extended it to dynamic features. A database recorded by a personal robot called PaPeRo in home environments was used for the evaluation of the proposed method under noisy conditions. While we presented a recognition system using isolated-word FHMMs in our previous work, here we evaluated the effectiveness of the phoneme FHMMs.

This paper proposes a new method to calculate the cepstral coefficients for an HMM-based synthesizer. It consists of a direct maximization of the log-likelihood function of a Gaussian mixture model using a gradient ascent algorithm. The method permits to integrate efficiently the global variance factor with a Gaussian mixture acoustic model. The perceptual experiments confirmed that these two factors produce significant improvements on the speech quality, which are independent from each other. By using the proposed method, it is possible to get the benefits of both factors. This paper also proposes a 2-class model for the global variance that discriminates between consonants and vowels. Such 2-class global variance model produces more stable cepstral coefficients than the single-class one.

Although speech derived from reading texts and similar types of speech, e.g., that from reading newspapers or that from news broadcasts, can be recognized with high accuracy, recognition performance drastically decreases for spontaneous speech. This is due to the fact that spontaneous speech and read speech are significantly different acoustically as well as linguistically. This paper analyzes differences in acoustic features between spontaneous and read speech using a large-scale spontaneous speech database "Corpus of Spontaneous Japanese (CSJ)". Using a linear transformation matrix, experimental results show that spontaneous speech can be characterized by reduced size of spectral space in comparison with that of read speech. These have also clarified that a reduction in the spectral space leads to a reduction in phoneme recognition accuracy. This result indicates that spectral reduction is one major reason for the decrease of recognition accuracy of spontaneous speech.

This paper proposes a new interface method using semi-synchronous speech and pen input for mobile environments. In this interface, a user speaks while writing, where pen input complements speech to achieve higher recognition performance than speech alone. A multimodal recognition algorithm that can handle the asynchronicity of the two modes using a segment-based unification scheme is proposed. This method is evaluated under noisy conditions with four different pen-input interfaces: character, stroke, pen-touch, and point-to-character, each of which is assumed to be given for a phrase unit in speech. It is confirmed that the recognition accuracy is improved by the proposed method in comparison with that by speech alone in all the pen-input conditions.

This paper presents a new Bayes classification rule towards minimizing the predictive Bayes risk for robust speech recognition. Conventionally, the plug-in maximum a posteriori (MAP) classification is constructed by adopting nonparametric loss function and deterministic model parameters. Speech recognition performance is limited due to the environmental mismatch and the ill-posed model. Concerning these issues, we develop the predictive minimum Bayes risk (PMBR) classification where the predictive distributions are inherent in Bayes risk. More specifically, we exploit the Bayes loss function and the predictive word posterior probability for Bayes classification. Model mismatch and randomness are compensated to improve generalization capability in speech recognition. In the experiments on car speech recognition, we estimate the prior densities of hidden Markov model parameters from adaptation data. With the prior knowledge of new environment and model uncertainty, PMBR classification is realized and evaluated to be better than MAP, MBR and Bayesian predictive classification.

One of the difficulties of Thai language modeling is the process of text corpus preparation. Because there is no explicit word boundary marker in written Thai text, word segmentation must be performed prior to training a language model. This paper presents two approaches to language model construction for Thai LVCSR based on pseudo-morpheme merging. The first approach merges pseudo-morphemes using forward and reverse bi-grams. The second approach utilizes the C4.5 decision tree to merge pseudo-morphemes based on multiple features. The performance of ASR systems with language models built using these methods are better than systems which use only pseudo-morpheme or lexicon-based word segmentation. These approaches produce results which are are also comparable to the system which utilizes manual segmentation.

We propose a dynamic language model adaptation method that uses the temporal information from lecture slides for lecture speech recognition. The proposed method consists of two steps. First, the language model is adapted with the text information extracted from all the slides of a given lecture. Next, the text information of a given slide is extracted based on temporal information and used for local adaptation. Hence, the language model, used to recognize speech associated with the given slide changes dynamically from one slide to the next. We evaluated the proposed method with the speech data from four Japanese lecture courses. Our experiments show the effectiveness of our

This paper proposes an automatic stream-weight and threshold estimation method for noise-robust speaker verification using multistream HMMs integrating segmental and prosodic information. The proposed method simultaneously optimizes stream-weights and a decision threshold by combining the linear discriminant analysis (LDA) and Adaboost techniques. Experiments were conducted using Japanese connected digit speech contaminated by white noise with various SNRs. In this experiment, a target ratio of false acceptance rate (FAR) and false rejection rate (FRR) was set by 1:1 so as to adjust them to approach an equal error rate (EER). Experimental results show that the proposed method effectively estimates stream-weights and thresholds so that FARs and FRRs are adjusted to EERs in most of the SNR conditions.

Contemporary approaches to automatic speech summarisation comprise several components, among them a linguistic model (LiM) component, which is unrelated to the language model used during the recognition process. This LiM component assigns a probability to word sequences from the source text according to their likelihood of appearing in the summarised text. In this paper we investigate LiM topic and stylistic adaptation using combinations of LiMs each trained on different adaptation data. Experiments are performed on 9 talks from the TED corpus of Eurospeech conference presentations, as well as 5 news stories from CNN broadcast news data, for all of which human (TRS) and speech recogniser (ASR) transcriptions along with human summaries were used. In all ASR cases, summarisation accuracy (SumACCY) of automatically generated summaries was significantly improved by automatic LiM adaptation, with relative improvements of at least 2.5% in all experiments

This paper presents a new speech recognition framework towards fulfilling optimal Bayes decision theory, which is essential for general pattern recognition. The recognition procedure is developed through minimizing the Bayes risk, or equivalently the expected loss due to classification action. Typically, loss function measures the penalty/evidence of choosing a candidate hypothesis. This function was manually specified or empirically calculated. Here, we exploit a novel Bayes loss function via testing the hypotheses whether the classification action produces loss or not. A Bayes factor is derived to measure loss in a statistical and meaningful way. Attractively, Bayes loss function using predictive distributions is robust to the uncertainty of environments. Also, optimizing this Bayes criterion equals to minimizing classification errors of test data. The relation between the minimum classification error (MCE) classifier and the proposed optimal Bayes classifier (OBC) is bridged. Specifically, the logarithm of Bayes factor in OBC is analogous to the misclassification measure in MCE when using predictive distribution as the discriminant function. We accordingly build a robust and discriminative classification for large vocabulary continuous speech recognition. In the experiments on broadcast news transcription, the new OBC rule significantly outperforms traditional maximum a posteriori classification.

This paper presents an automatic sentence segmentation method for an automatic speech summarization system. The segmentation method is based on combining word- and class-based statistical language models to predict sentence and non-sentence boundaries. We study both the performance of the sentence segmentation system itself and the effect of the segmentation on the summarization accuracy. The sentence segmentation is done by modelling the probability of a sentence boundary given a certain word history with language models trained on transcriptions and texts from several sources. The resulting segmented data is used as the input to an existing automatic summarization system to determine the effect it has on the summarization process. We conduct all our experiments with two types of evaluation data: broadcast news and lecture transcriptions. The automatic summarizations are created with different sentence segmentations and different summarization ratios (30% and 40%) and evaluated by comparing them to human-made summaries. We show that a proper sentence segmentation is essential to achieve good performance with an automatic summarization system

Speech summarization technology, which extracts important information and removes irrelevant information from speech, is expected to play an important role in building speech archives and improving the efficiency of spoken document retrieval. However, speech summarization has a number of significant challenges that distinguish it from general text summarization. Fundamental problems with speech summarization include speech recognition errors, disfluencies, and difficulties of sentence segmentation. Typical speech summarization systems consist of speech recognition, sentence segmentation, sentence extraction, and sentence compaction components. Most of the research has focuses on sentence extraction, using LSA (latent semantic analysis), MMR (maximal marginal relevance), or feature-based approaches, among which no decisive method has yet been found. Proper sentence segmentation is also essential to achieve good summarization performance. How to objectively evaluate speech summarization results is an important issue. Several measures, including families of SumACCY and ROUGE measures, have been proposed, and correlation analyses between subjective and objective evaluation scores have been performed. Although these measures are useful for ranking various summarization methods, they do not correlate well with human evaluations, especially when spontaneous speech is targeted.

In this paper we treat question answering (QA) as a classification problem. Our motivation is to build systems for many languages without the need for highly tuned linguistic modules. Consequently, word tokens and Web data are used extensively but no explicit linguistic knowledge is incorporated. A mathematical model for answer retrieval, answer classification and answer length prediction is derived. The TREC 2002 QA task is used for system development where 33% of questions are answered correctly. Performance is then evaluated on the factoid questions of the TREC 2003 QA task where 23% of questions were answered correctly, which would rank the system in the top 10 of contemporary QA systems on the same task

This paper presents automatic speech summarization techniques and its evaluation metrics, focusing on sentence extraction-based summarization methods for making abstracts from spontaneous presentations. Since humans tend to summarize presentations by extracting important sentences from introduction and conclusion parts, this paper proposes a method using sentence location. Experimental results show that the proposed method significantly improves automatic speech summarization performance for the condition of 10% summarization ratio. Results of correlation analysis between subjective and objective evaluation scores confirm that objective evaluation metrics, including summarization accuracy, sentence F-measure and ROUGE-N, are effective for evaluating summarization techniques.

The paper proposes a new approach to multilingual synthesis based on an HMM synthesis technique. The idea consists of combining data from different monolingual speakers in different languages to create a single polyglot average voice. This average voice is then transformed into any real speaker's voice of one of these languages. The speech synthesized in this way has the same intelligibility and retains the same individuality for all the languages mixed to create the average voice, regardless of the target speaker's own language.

In the field of audio-visual speech recognition, multi-stream HMM are widely used, thus how to automatically and properly determine stream weight factors using a small data set becomes an important research issue. This paper proposes a new stream-weight optimization method based on an output likelihood normalization criterion. In this method, the stream weights are adjusted to equalize the mean values of log likelihood for all HMM based on likelihood-ratio maximization which achieved significant improvement by using a large optimization data set. The new method is evaluated using Japanese connected digit speech recorded in real-world environments. Using 10 seconds speech data for stream-weight optimization, a 10% absolute accuracy improvement is achieved compared to the result before optimization. By additionally applying the MLLR (maximum likelihood linear regression) adaptation, a 23% improvement is obtained over the audio-only scheme.

How to detect speech periods in noisy speech and how to cope with the temporal variation of noise characteristics are challenging problems. This paper proposes a new robust noisy speech recognition method based on robust end-point detection and online model adaptation using tree-structured noisy speech HMMs. The basic algorithm consists of; 1) blind speech segmentation; 2) best matching GMM selection; 3) recognizing the speech with the HMM that corresponds to the GMM; 4) end-point detection based on the recognition results; 5) HMM adaptation based on the recognition results; and 6) re-recognition using the adapted HMM. The processes of 1) through 6) are repeated by shifting the blind segmentation window until the end of the sequence of utterances is detected. The proposed method is evaluated by noisy speech collected by a Japanese dialogue system. Experimental results show that the proposed method is effective in recognizing noisy speech under various noise conditions.

This paper proposes a robust statistical framework to extract highlights from a baseball broadcast video. We applied multi-stream hidden Markov models (HMMs) to control the weights among different features. To achieve robustness against new highlights, we used a common simple structure for all the HMMs. In addition, scene segmentation and unsupervised adaptation were applied to achieve more robustness against the differences of environmental conditions among games. The precision rate of high-light extracting experiments for eight kinds of highlights from 4.5 hours of digest data was 77.4% and was increased to 78.7% by applying scene segmentation. Furthermore, the unsupervised adaptation method improved precision by 2.7 points to 81.4%. These results confirm the effectiveness of our framework.

This paper proposes an automatic stream-weight optimization method for noise-robust speaker verification using multi-stream HMMs integrating spectral and prosodic information. The paper first shows the effectiveness of the multi-stream technique in our speaker verification framework. Next, a stream-weight adaptation method combining the linear discriminant analysis (LDA) and Adaboost techniques is proposed. Experiments were conducted using four-connected-digit utterances of Japanese contaminated by white noise with various SNRs. Experimental results show that 1) the verification performance was improved in all SNR conditions by using stream weights estimated by the LDA and 2) the performance is further improved by using the Adaboost in 10 - 30dB SNR conditions.

In order to realize speech recognition systems that can achieve high recognition accuracy for ubiquitous speech, it is crucial to make the systems flexible enough to cope with a large variability of spontaneous speech. This paper investigates two speech recognition methods that can adapt to speech variation using a large number of models trained based on clustering techniques; one automatically builds a model adapted to input speech using recognition hypotheses and clustered models, and the other directly uses clustered models in parallel. Both methods have been confirmed to be effective by evaluation experiments using presentation speech. Although the latter method needs a large amount of computation, it has an advantage in that it can be applied to online recognition, since it does not need recognition hypotheses. The former method can also be applied to online recognition, if the text of proceedings for the presentation can be used in place of recognition hypotheses. 1.

Text corpus size is an important issue when building a language model (LM). This is a particularly important issue for languages where little data is available. This paper introduces a technique to improve a LM built using a small amount of task dependent text with the help of a machine-translated text corpus. Perplexity experiments were performed using data, machine translated (MT) from English to French on a sentence-by-sentence basis and using dictionary lookup on a word-by-word basis. Then perplexity and word error rate experiments using MT data from English to Icelandic were done on a word-by-word basis. For the latter, the baseline word error rate was 44.0%. LM interpolation reduced word error rate significantly to 39.2%.

In this paper we examine the use of an HMM-based polyglot synthesizer for languages for which very limited or no speech data is available. In a former study, we presented a system that combines monolingual corpora from several languages to create a polyglot synthesizer. With this synthesizer we can synthesize any of the languages included in the training data with the same output voice and speech quality. In this paper, we approximate the sounds of non-included languages, by those available in the polyglot training data. Since the phonetic inventory of a polyglot synthesizer is wider than that of a monolingual one, the approximation of such non-included sounds becomes more accurate and thus the perceptual intelligibility increases. Moreover, the performance of a polyglot synthesizer can be further improved by adding a reduced amount of data from the target language.

This paper surveys a series of model selection approaches and presents a novel predictive information criterion (PIC) for hidden Markov model (HMM) selection. The approximate Bayesian using Viterbi approach is applied for PIC selection of the best HMMs providing the largest prediction information for generalization of future data. When the perturbation of HMM parameters is expressed by a product of conjugate prior densities, the segmental prediction information is derived at the frame level without Laplacian integral approximation. In particular, a multivariate t distribution is attained to characterize the prediction information corresponding to HMM mean vector and precision matrix. When performing model selection in tree structure HMMs, we develop a top-down prior/posterior propagation algorithm for estimation of structural hyperparameters. The prediction information is determined so as to choose the best HMM tree model. Different from maximum likelihood (ML) and minimum description length (MDL) selection criteria, the parameters of PIC chosen HMMs are computed via maximum a posteriori estimation. In the evaluation of continuous speech recognition using decision tree HMMs, the PIC criterion outperforms ML and MDL criteria in building a compact tree structure with moderate tree size and higher recognition rate.

For multi-stream HMM that are widely used in audio-visual speech recognition, it is important to automatically and properly adjust stream weights. This paper proposes a stream-weight optimization technique based on a likelihood-ratio maximization criterion. In our audiovisual speech recognition system, video signals are captured and converted into visual features using HMM-based techniques. Extracted acoustic and visual features are concatenated into an audio-visual vector. A multi-stream HMM is obtained from audio and visual HMM. Experiments are conducted using Japanese connected digit speech recorded in real-world environments. Applying the MLLR (maximum likelihood linear regression) adaptation and our optimization method, we achieve a 29% absolute accuracy improvement and a 76% relative error rate reduction compared with the audio-only scheme.

This paper proposes the application of a tree-structured clustering method that integrates the effects of noise as well as SNR variation in the framework of piecewise-linear transformation (PLT)-based noise adaptation for robust speech recognition. According to the clustering results, a noisy speech HMM is made for each node of the tree structure. An HMM that best matches the input speech is selected based on the likelihood maximization criterion by tracing the tree downward from the top (root), and the selected HMM is further adapted by linear transformation. The proposed method is evaluated by applying it to a Japanese dialogue recognition system. Experimental results confirm that the proposed method is effective in recognizing numerically noise-added speech and actual noisy speech uttered by a wide range of speakers under various noise conditions.

This paper proposes a noise-robust speaker verification method augmented by fundamental frequency (F0).The paper first describes a noise-robust F0 extraction method using the Hough transform.Then, it proposes a robust speaker verification method using multi-stream HMMs which fuse the extracted F0 and cepstral features.Experiments are conducted using fourconnected-digit utterances of Japanese by 37 male speakers recorded at five sessions over a half year period.The utterances are contaminated with white noise at various SNR levels.Experimental results show that the F0 features improve the verification performance in all SNR conditions.

Since spontaneous utterances include many variations, speakerand task-independent general models do not work well.This paper proposes combining cluster-based language and acoustic models based on the framework of Massively Parallel Decoder (MPD).The MPD is a parallel decoder that has a large number of decoding units, in which each unit is assigned to each combination of element models.It runs efficiently on a parallel computer, and thus the turnaround time is comparable to conventional decoders using a single model and a processor.In the experiments conducted using lecture speeches from the Corpus of Spontaneous Japanese, two types of cluster models have been investigated: lecture-based cluster models and utterancebased cluster models.It has been confirmed that utterancebased cluster models give significantly lower recognition error rate than lecture-based cluster models in both language and acoustic modeling.It has also been shown that roughly 100 decoding units are enough in terms of recognition rate, and in the best setting, 12% reduction in word error rate was obtained in comparison with the conventional decoder.

This paper proposes an approach to improve speech understanding based on rescoring of N-best semantic hypotheses.In rescoring, probabilities produced by an understanding component are combined with additional probabilities derived from system beliefs.While a normal rescoring approach is to multiply or linearly interpolate with belief probabilities, this paper shows that probabilities from various sources are better combined using a nonlinear estimator.Using the proposed model together with a dialogue-state dependent semantic model shows a significant improvement when applying to a Thai interactive hotel reservation agent (TIRA), the first spoken dialogue system in Thai language.

This paper uses the AURORA2 task to investigate the performance of our proposed tree-structured piecewise-linear transformation (PLT) noise adaptation.In our proposed method, an HMM that best matches the input speech is selected based on the likelihood maximization criterion by tracing a tree structured HMM space that is prepared in the training step, and the selected HMM is further adapted by linear transformation.Experimental results show that our method achieves a significant improvement for the AURORA2 database.

This paper presents techniques for speech-to-text and speech-to-speech automatic summarization based on speech unit extraction and concatenation. For the former case, a two-stage summarization method consisting of important sentence extraction and word-based sentence compaction is investigated. Sentence and word units which maximize the weighted sum of linguistic likelihood, amount of information, confidence measure, and grammatical likelihood of concatenated units are extracted from the speech recognition results and concatenated for producing summaries. For the latter case, sentences, words, and between-filler units are investigated as units to be extracted from original speech. These methods are applied to the summarization of unrestricted-domain spontaneous presentations and evaluated by objective and subjective measures. It was confirmed that proposed methods are effective in spontaneous speech summarization.

Recently, open-domain question answering (ODQA) systems that extract an exact answer from large text corpora based on text input are intensively being investigated. However, the information in the first question input by a user is not usually enough to yield the desired answer. Interactions for collecting additional information to accomplish QA is needed. This paper proposes an interactive approach for spoken interactive ODQA systems. When the reliabilities for answer hypotheses obtained by an ODQA system are low, the system automatically derives disambiguous queries (DQ) that draw out additional information. The additional information based on the DQ should contribute to distinguishing effectively an exact answer and to supplementing a lack of information by recognition errors. In our spoken interactive ODQA system, SPIQA, spoken questions are recognized by an ASR system, and DQ are automatically generated to disambiguate the transcribed questions. We confirmed the appropriateness of the derived DQ by comparing them with manually prepared ones.

This paper proposes a new automatic speech summarization method having two stages: important sentence extraction and sentence compaction. Relatively important sentences are extracted based on the amount of information and the confidence measures of constituent words, and the set of extracted sentences is compressed by our sentence compaction method. The sentence compaction is performed by selecting a word set that maximizes a summarization score consisting of the amount of information and the confidence measure of each word, the linguistic likelihood of word strings, and the word concatenation probability. The selected words are concatenated to create a summary. Effectiveness of the proposed method was confirmed by summarizing a spontaneous presentation.

This article introduces a novel approach to model morphosyntax in morpheme unit based speech recognizers. The proposed method is evaluated in our recent Hungarian large vocabulary continuous speech recognition (LVCSR) system. The architecture of the recognition system is based on the weighted finite state transducer (WFST) paradigm. The task domain is the recognition of fluently read sentences selected from a major daily newspaper. The vocabulary units used in the system are morpheme based in order to provide sufficient coverage of the large number of word-forms resulting from affixation and compounding. Besides the standard morpheme N-gram language model we evaluate the novel stochastic morphosyntactic language model (SMLM) that describes the valid word-forms (morpheme combinations) of the language. Thanks to the flexible transducer-based architecture of the system the morphosyntactic component is integrated seamlessly with the basic modules with no need to modify the decoder itself. The proposed stochastic morphosyntactic language model decreases the error rate by 17.9% relatively compared to the baseline trigram system. The morpheme error rate of the best configuration is 14.75% in a 1350 morpheme Hungarian dictation task.

This paper proposes an unsupervised, batch-type, class-based language model adaptation method for spontaneous speech recognition. The word classes are automatically determined by maximizing the average mutual information between the classes using a training set. A class-based language model is built based on recognition hypotheses obtained using a general word-based language model, and linearly interpolated with the general language model. All the input utterances are re-recognized using the adapted language model. The proposed method was applied to the recognition of spontaneous presentations and was found to be effective in improving the recognition accuracy for all the presentations. The best condition was found to be using 100 word classes, and in this condition 2.3% of the absolute value improvement in the word accuracy averaged over all the speakers was achieved.

This paper presents a novel predictive information criterion (PIC) for hidden Markov model (HMM) selection.The PIC criterion is exploited to select the best HMMs, which provide the largest prediction information for generalization of future data.When the randomness of HMM parameters is expressed by a product of conjugate prior densities, the prediction information is derived without integral approximation.In particular, a multivariate t distribution is attained to characterize the prediction information corresponding to HMM mean vector and precision matrix.When performing HMM selection in tree structure HMMs, we develop a top-down prior/posterior propagation algorithm for estimation of structural hyperparameters.The prediction information is accordingly determined so as to choose the best HMM tree model.The parameters of chosen HMMs can be rapidly computed via maximum a posteriori (MAP) estimation.In the evaluation of continuous speech recognition using decision tree HMMs, the PIC model selection criterion performs better than conventional maximum likelihood and minimum description length criteria in building a compact tree structure with moderate tree size and higher recognition rate.

This paper overviews robust architecture and modeling techniques for automatic speech recognition and understanding. The topics include robust acoustic and language modeling for spontaneous speech recognition, unsupervised adaptation of acoustic and language models, robust architecture for spoken dialogue systems, multi-modal speech recognition, and speech understanding. This paper also discusses the most important research problems to be solved in order to achieve ultimate robust speech recognition and understanding systems.

We have proposed an automatic speech summarization approach that extracts words from transcription results obtained by automatic speech recognition (ASR) systems. To numerically evaluate this approach, the automatic summarization results are compared with manual summarization generated by humans through word extraction. We have proposed three metrics, weighted word precision, word strings precision and summarization accuracy (SumACCY), based on a word network created by merging manual summarization results. In this paper, we propose a new metric for automatic summarization results, weighted summarization accuracy (WSumACCY). This accuracy is weighted by the posterior probability of the manual summaries in the network to give the reliability of each answer extracted from the network. We clarify the goal of each metric and use these metrics to provide automatic evaluation results of the summarized speech. To compare the performance of each evaluation metric, correlations between the evaluation results using these metrics and subjective evaluation by hand are measured. It is confirmed that WSumACCY is an effective and robust measure for automatic summarization.

Abstract Oneof themostseriousproblemsin spontaneousspeechrecog-nition is the degradation of recognition accuracy due to thespeaking rate fluctuation in an utterance. This paper proposesa method for adjusting mixture weights of an HMM frame byframe depending on the local speaking rate. The proposedmethodisimplementedusingtheBayesiannetworkframework.A hidden variable representing the variation of the “mode” ofthespeakingrateisintroducedanditsvaluecontrolsthemixtureweights of Gaussian mixtures. Model training and maximumprobability assignment of the variablesare conducted usingtheEM/GEMandinferencealgorithmsforBayesiannetworks. TheBayesian network is used to rescore the acoustic likelihood ofthe hypotheses in N-best lists. Experimental results show thatthe proposed method improves word accuracy by 1.6% for theabsolute value on meeting speech given the speaking rate in-formation, whereas improvement by a regression HMM is lesssignificant. 1. Introduction Conventional HMM-based recognizers suffer a lower recogni-tion rate for spontaneous speech. One of the significant factorsreducing the recognition rate is the variable nature of sponta-neousutterances[1]. For example,thespeakingratecanchangeeven within one utterance. A possible strategy to manage thisproblem is first estimating the speaking rate and then adjustingarecognizer basedonthe speakingrate. Thispaperinvestigateshow to use the speaking rate information to control probabilitydensity functions in the HMM.Since estimating the speaking rate is itself a difficult prob-lem and assuming an accurate detection is unrealistic, themethod of adjusting the decoder accordingto the speaking rateshould be probabilistic. This paper proposes a method of ad-justing mixture weights of the HMM at every frame based onthe speaking rate. A hidden variable that represents a “mode”ofthespeakingratecontrolsthemixtureweights. Theproposedmethod can model complex changes of acoustic characteristicswith only asmall increaseof the numberof parameters.The proposed method is implemented using the Bayesiannetwork framework to realize detailed control of HMM param-eters[2]. WeuseGMTK[3]for modelparametertraining usingthe EM/GEM algorithms and for decoding. The network ac-ceptsthespeakingrateinadditiontotheusualacousticfeatures.The proposed method is used to rescore the acoustic likelihoodof N-best hypotheses.This paper is organized as follows. In Section 2, the pro-posed method is formulated as a Bayesian network. In Sec-tion 3, an experimental set up is described. Training processesof the Bayesian network are described in Section 4, and ob-tained parameters related to the speaking rate are analyzed inSection 5. The proposed method is applied to meeting speechrecognition in Section 6 and the results are discussed in Sec-tion 7. It is shown that the proposed method is more effectivein improving the recognition ratethan aregressionHMM givenspeaking rate information. Finally, in Section 8, the paper isconcluded.

This paper proposes a novel approach for spoken language understanding based on a combination of weighted finite state automata and an artificial neural network.The former machine acts as a robust parser, which extracts some semantic information called subframes from an input sentence, then the latter machine interprets a concept of the sentence by considering the existence of subframes and their scores obtained from the automata.With a large number of concepts handled in our mixed-initiative dialogue system, the proposed system achieves a considerable concept interpretation result on either a typed-in test set or a spoken test set.A high subframe recall rate also verifies an applicability of the proposed system.

This paper proposes the application of tree-structured clustering to various noise samples or noisy speech in the framework of piecewise-linear transformation (PLT)-based noise adaptation.According to the clustering results, a noisy speech HMM is made for each node of the tree structure.Based on the likelihood maximization criterion, the HMM that best matches the input speech is selected by tracing the tree from top to bottom, and the selected HMM is further adapted by linear transformation.The proposed method is evaluated by applying it to a Japanese dialogue recognition system.The results confirm that the proposed method is effective in recognizing noise-added speech under various noise conditions.

This paper proposes a new automatic speech summarization method. In this method, a set of words maximizing a summarization score is extracted from automatically transcribed speech. This extraction is performed according to a target compression ratio using a dynamic programming (DP) technique. The extracted set of words is then connected to build a summarization sentence. The summarization score consists of a word significance measure, a confidence measure, linguistic likelihood, and a word concatenation probability. The word concatenation score is determined by a dependency structure in the original speech given by stochastic dependency context free grammar (SDCFG). Japanese broadcast news speech transcribed using a large-vocabulary continuous-speech recognition (LVCSR) system is summarized using our proposed method and compared with manual summarization by human subjects. The manual summarization results are combined to build a word network. This word network is used to calculate the word accuracy of each automatic summarization result using the most similar word string in the network. Experimental results show that the proposed method effectively extracts relatively important information by removing redundant and irrelevant information.

This paper reports an automatic speech summarization method and experimental results using English broadcast news speech. In our proposed method, a set of words maximizing a summarization score indicating an appropriateness of summarization is extracted from automatically transcribed speech. This extraction is performed using a Dynamic Programming (DP) technique according to a target compression ratio. We have previously tested the performance of our method using Japanese broadcast news speech. Since our method is based on a statistical approach, it could be applied to any language. In this paper, English broadcast news speech transcribed using a speech recognizer is automatically summarized. In order to apply our method to English, the model of estimating word concatenation probabilities based on a dependency structure in the original speech given by a Stochastic Dependency Context Free Grammar (SDCFG) is modified. A summarization method for multiple utterances using two-level DP technique is also proposed.

This paper reports an analysis of individual differences in spontaneous presentation speech recognition performances. Ten minutes from each presentation given by 50 male speakers, for a total of 500 minutes, has been automatically recognized for the analysis. Correlation and regression analyses were applied to the word recognition accuracy and various speaker attributes. A restricted set of the speaker attributes comprising the speaking rate, the out of vocabulary rate and the repair rate was found to be most significant to yield individual differences in the word accuracy. Unsupervised MLLR speaker adaptation worked well for improving the word accuracy but did not change the structure of the individual differences. Approximately half of the variance in the word accuracy was explained by a regression model using the limited set of three attributes.

This paper describes a new method of implementing mixed-initiative spoken dialogue systems based on parallel computing architecture. In a mixed-initiative dialogue, the user as well as the system needs to be capable of controlling the dialogue sequence. In our implementation, various language models corresponding to different dialogue contents, such as requests for information or replies to the system, are built and multiple recognizers using these language models are driven under a parallel computing architecture. The dialogue content of the user is automatically detected based on likelihood scores given by the recognizers, and the content is used to build the dialogue. A transitional probability from one dialogue state uttering a kind of content to another state uttering a different content is incorporated into the likelihood score. A flexible dialogue structure that gives users the initiative to control the dialogue is implemented by this architecture. Real-time dialogue systems for retrieving information about restaurants and food stores are built and evaluated in terms of dialogue content identification rate and keyword accuracy. The proposed architecture has the advantage that the dialogue system can be easily modified without remaking the whole language model.

This paper proposes a noise robust speech recognition method using prosodic information.In Japanese, fundamental frequency (F0) contour represents phrase intonation and word accent information.Consequently, it conveys information about prosodic phrase and word boundaries.This paper first proposes a noise robust F0 extraction method using Hough transform, which achieves high extraction rates under various noise environments.Then it proposes a robust speech recognition method using syllable HMMs which model both segmental spectral features and F0 contours.Speaker-independent experiments are conducted using connected digits uttered by 11 male speakers in various kinds of noise and SNR conditions.The recognition accuracy is improved in all noise conditions, and the best absolute improvement of digit accuracy is about 4.7%.This improvement is achieved due to the more precise digit boundary detection by the robust prosodic information.

This paper proposes a new lexicon optimization method to improve recognition rate of large scale spontaneous speech recognition.Occurrence count and length of a word has strong correlation with difficulty of recognizing the word.First, we investigate the relation and make a word correctness probability model.The proposed method optimizes the lexicon by making compound words or phrases step by step based on the word correctness probability model so as to improve the estimated recognition rate of the system.The optimization method is applied to a large scale Japanese spontaneous speech corpus.Experimental results show that the language model using the optimized lexicon improves the recognition rate.

This article describes the design and the experimental evaluation of the first Hungarian large vocabulary continuous speech recognition (LVCSR) system.The architecture of the recognition system is based on the recently proposed weighted finite state transducer (WFST) paradigm.The task domain is the recognition of fluently read sentences selected from a major daily newspaper.Recognition performance is evaluated using both monophone and triphone gender independent acoustic models.The vocabulary units used in the system are morpheme based in order to provide sufficient coverage of the large number of word-forms resulting from affixation and compounding in Hungarian.The language model is a statistical morpheme bigram model.Besides the basic list style pronunciation dictionary model we evaluate a novel phonology modeling component that describes the phonological changes prevalent in fluent Hungarian.Thanks to the flexible transducerbased architecture of the system the phonological component is integrated seamlessly with the basic modules with no need to modify the decoder itself.The proposed phonological model decreases the error rate by 8.32% relatively compared to the baseline triphone system.The morpheme error rate of the best configuration is 17.74% in a 1200 morpheme task with test set perplexity 70.

This paper proposes a new method, using neural networks, of adapting phone HMMs to noisy speech. The neural networks are designed to map clean speech HMMs to noise-adapted HMMs, using noise HMMs and signal-to-noise ratios (SNRs) as inputs, and are trained to minimize the mean square error between the output HMMs and the target noise-adapted HMMs. In evaluation, the proposed method was used to recognize noisy broadcast-news speech in speaker-dependent and -independent modes. The trained networks were confirmed to be effective in recognizing new speakers under new noise and various SNR conditions.

In the ubiquitous (pervasive) computing era, it is expected that everybody will access information services anytime anywhere, and these services are expected to augment various human intelligent activities. Speech recognition technology can play an important role in this era by providing: (a) conversational systems for accessing information services and (b) systems for transcribing, understanding and summarizing ubiquitous speech documents such as meetings, lectures, presentations and voicemails. In the former systems, robust conversation using wireless handheld/hands-free devices in the real mobile computing environment will be crucial as will multimodal speech recognition technology. To create the latter systems, the ability to understand and summarize speech documents is one of the key requirements. The paper presents technological perspectives and introduces several research activities being conducted from these standpoints in our research group.

This paper reports recent advances in automatic speech summarization method. In our proposed method, a set of words maximizing a summarization score is extracted from automatically transcribed speech. This extraction is performed according to a target compression ratio using a dynamic programming technique. The extracted set of words is then connected to build a summarized sentence. The summarization score consists of a word significance measure, a confidence measure, linguistic likelihood, and a word concatenation probability which is determined by a dependency structure in the original speech given by Stochastic Dependency Context Free Grammar (SDCFG) . Japanese broadcast news speech transcribed using a large vocabulary continuous speech recognition (LVCSR) system is summarized using our proposed method and evaluated in comparison with manual summarization by human subjects. The manual summarization results are combined to build a word network, and word accuracy of each automatic summarization result is calculated comparing with the most similar word string in the network.

This paper reports various investigations on recognizing spontaneous presentation speech in connection with the "Spontaneous Speech" national project started in 1999.Presentation speech uttered by 10 male speakers of approximately 4.5 hours duration has been recognized.Experimental results show that acoustic and language modeling based on an actual spontaneous speech corpus is far more effective than conventional modeling based on read speech.The recognition accuracy has a wide speaker-tospeaker variability according to the speaking rate, the number of fillers, the number of repairs, etc.It was confirmed that unsupervised speaker adaptation of acoustic models was effective to improve the recognition accuracy.The recognition accuracy for spontaneous speech is, however, still rather low, and there remains a large number of research issues.

Due principally to the technology of making computers smaller, more powerful and cheaper, the ubiquitous and wearable computing era is expected to come into being in the beginning of 21st century. In such an environment, speech recognition will be widely used as one of the principal methods of human-computer interaction, and in most of cases it will be performed using a microphone and a networked computer worn and tailored to each person. Accordingly, speech recognition systems will comprise very different structures than do present systems. Promptly retrieving environmental information, such as noise, which is regularly monitored by computers, and task-dependent information transmitted according to each application to the personalized recognizers will significantly improve system adaptability to the new tasks and robustness. Nonetheless, a number of salient issues remain to be resolved to generate the ubiquitous/wearable computing environment.

This paper proposes a new method of automatically summarizing speech by extracting a limited number of relatively important words from its automatic transcription according to a target compression ratio for the number of characters. To determine a word set to be extracted, we define a summarization score consisting of a topic score (significance measure) of words and a linguistic score (likelihood) of the word concatenation. A set of words maximizing the score is efficiently selected using a dynamic programming (DP) technique. Japanese broadcast news speech transcribed using a large vocabulary continuous speech recognition system was summarized. As a result 86% of important words in the original speech were correctly included in the summarizing sentences and 72% of the summarizing sentences could maintain the meanings of the original speech under the 60-70% summarization condition.

In order to improve the performance of speech recognition systems when speakers change frequently and each of them utters a series of several sentences, a new unsupervised, online and incremental speaker adaptation technique combined with automatic detection of speaker changes is proposed. The speaker change is detected by comparing likelihoods using speaker-independent and speaker-adaptive Gaussian mixture models (GMMs). Both the phone HMM and GMM are adapted by MLLR transformation. In a broadcast news transcription task, this method reduces the word error rate by 10.0%. In comparison with the conventional method that uses HMMs for the speaker change detection, the GMM-based method requires a significantly less number of computations at the cost of only a slightly lower word recognition rate.

Although high recognition accuracy can be obtained for speech in the form of reading a written text or similar by using state-of-the art speech recognition technology, the accuracy is quite poor for freely spoken spontaneous speech. From this perspective, a five-year national project for raising the technological level of speech recognition and understanding commenced in Japan in 1999. The project focuses on building a large-scale spontaneous speech corpus and acoustic and linguistic modeling for spontaneous speech recognition and summarization. This paper reports some results of preliminary experiments which have been conducted at Tokyo Institute of Technology. Experimental results show that acoustic and language modeling based on the actual spontaneous speech corpus is far more effective than modeling based on read speech. It was also shown that our proposed automatic speech summarization method could effectively extract relatively important information and remove redundant and irrelevant information. 1.

This paper proposes an improved method of summarizing speech in which a con dence measure of a word hypothesis is incorporated in the summarization score and also proposes a new method for evaluating the summarized sentences.The automatically summarized sentences were evaluated based on the precision of extracted keywords and each word string with a certain length in the manual summarizations by human subjects.Japanese broadcast-news speech transcribed using a large-vocabulary continuousspeech recognition (LVCSR) system was summarized using our proposed method.Experimental results show that a con dence score giving a penalty for acoustically as well as linguistically unreliable hypotheses can reduce the meaning alteration of summarizations caused by recognition errors especially when the speech recognition rate is relatively low.

Robust speech recognition via hidden Markov modeling of spectral vectors is studied in this paper.The hidden Markov model (HMM) mixture components are assumed complex Gaussian with zero mean, diagonal covariance, and with incorporating an unknown scalar gain term.The gain term is associated with each spectral vector and it models the varying energy of speech signals.It is estimated by applying the maximum likelihood (ML) criterion.On an isolated digit database, in clean conditions, the spectral modeling with ML gain estimation approach a c hieved similar performance to cepstral modeling of speech.Two additive noise compensation approaches for the spectral modeling scheme are also considered.The rst approach requires a full noise HMM.This HMM is combined with the clean speech HMM to yield a noisy speech HMM.The second approach requires only the spectral shape of the noise.A term dependent on the spectral shape, together with an unknown magnitude term, is incorporated into the clean speech HMM to yield a noisy speech HMM.The unknown magnitude of the noise is estimated via the ML criterion.The performance of these two approaches for isolated digit recognition in noise is demonstrated and compared to a robust cepstral modeling approach from the literature.

We previously proposed an incremental speaker adaptation method combined with automatic speaker-change detection for broadcast news transcription where speakers change frequently and each of them utters a series of several sentences.In this method, the speaker change is detected using speaker-independent and speaker-adaptive Gaussian mixture models (GMMs).Both phone HMMs and GMMs are incrementally adapted to each speaker by the combination of MLLR, MAP and VFS methods using speaker-independent (SI) models as initial models.This paper proposes its improvement in which an initial model for speaker adaptation is selected from a set of models made by speaker clustering.Either cluster-dependent phone HMMs or GMMs are used to calculate the likelihood for selecting the best initial model.In a broadcast news transcription task, the proposed method significantly reduces word error rate compared with the method using SI-HMM as an initial model.Online incremental speaker adaptation results show that word error rate is reduced by 11.6% relative to the baseline system with no speaker adaptation.The method using GMMs for cluster selection requires a significantly less number of computations than that using HMMs.

The promise of a powerful computing device to help people in productivity as well as in recreation can only be realized with proper human-machine communication. Automatic recognition and understanding of spoken language is the first step toward natural human-machine interaction. Research in this field has produced remarkable results, leading to many exciting expectations and new challenges. We summarize the development of the spoken language technology from both a vertical (chronology) and a horizontal (spectrum of technical approaches) perspective. We highlight the introduction of statistical methods in dealing with language-related problems, as this represents a paradigm shift in the research field of spoken language processing. Statistical methods are designed to allow the machine to learn structural regularities in the speech signal, directly from data, for the purpose of automatic speech recognition and understanding. Research results in spoken language processing have led to a number of successful applications, ranging from dictation software for personal computers and telephone-call processing systems for automatic call routing, to automatic sub-captioning for television broadcasts. We analyze the technical successes that support these applications. Along with an assessment of the state of the art in this broad technical field, we also discuss the limitations of the current technology, and point out the challenges that are ahead. This paper presents an accurate overview of spoken language technology as a basis to inspire future advances.

This paper describes recent progress and the author's perspectives of speech recognition technology. Applications of speech recognition technology can be classified into two main areas, dictation and human-computer dialogue systems. In the dictation domain, the automatic broadcast news transcription is now actively investigated, especially under the DARPA project. The broadcast news dictation technology has recently been integrated with information extraction and retrieval technology and many application systems, such as automatic voice document indexing and retrieval systems, are under development. In the human-computer interaction domain, a variety of experimental systems for information retrieval through spoken dialogue are being investigated. In spite of the remarkable recent progress, we are still behind our ultimate goal of understanding free conversational speech uttered by any speaker under any environment. This paper also describes the most important research issues that we should attack in order to advance to our ultimate goal of fluent speech recognition.

This paper proposes a new formulation for speech recognition/understanding systems. In which the posteriori probability of a speaker's message that the speaker intends to address given an observed acoustic sequence is maximized. This is an extension of the current criterion that maximizes the probability of a word sequence. Among the various possible representations, we employ a co-occurrence score of words measured by mutual information as the conditional probability of a word sequence occurring in a given message. The word sequence hypotheses obtained by bigram and trigram language models are rescored using the co-occurrence score. Experimental results show that the word accuracy is improved by this method. Topic-words which represent the content of a speech signal are then extracted from speech recognition results based on the significance score of each word. When five topic-words are extracted for each broadcast-news article, 82.8% of them are correct in average. This paper also proposes a verbalization-dependent language model which is useful for Japanese dictation systems.

In this paper, we report on language modeling and acoustic modeling studies for Japanese broadcast news speech recognition. We constructed a language model that reduces recognition errors by utilizing context-dependent readings of Japanese characters. We also introduced filled-pause modeling into the language model. To improve the model’s performance for a series of sentences spoken by one speaker, an on-line incremental speaker adaptation was combined with automatic detection of speaker changes. By incorporating all the above methods, we achieved a 25.1% reduction in word error rate over the baseline results. This paper also reports on our preliminary studies on topic extraction and summarization of broadcast-news speech.

This paper reports on topic extraction in Japanese broadcast-news speech. We studied, using continuous speech recognition, the extraction of several topic-words from broadcast-news. A combination of multiple topic-words represents the content of the news. This is a more detailed and more flexible approach than using a single word or a single category. A topic extraction model shows the degree of relevance between each topic-word and each word in the article. For all words in an article, topic-words which have high total relevance score are extracted. We trained the topic extraction model with five years of newspapers, using the frequency of topic-words taken from headlines and words in articles. The degree of relevance between topic-words and words in articles is calculated on the basis of statistical measures, i.e., mutual information or the /spl chi//sup 2/-value. In topic extraction experiments for recognized broadcast-news speech, we extracted five topic-words from the 10-best hypotheses using a /spl chi//sup 2/-based model and found that 76.6% of them agreed with the topic-words chosen by subjects.

This paper introduces a paradigm for designing multimodal dialogue systems. An example system task of the system is to retrieve particular information about different shops in the Tokyo Metropolitan area, such as their names, addresses and phone numbers. The system accepts speech and screen touching as input, and presents retrieved information on a screen display. The speech recognition part is modeled by the FSN (finite state network) consisting of keywords and fillers, both of which are implemented by the DAWG (directed acyclic word-graph) structure. The number of keywords is 306, consisting of district names and business names. The fillers accept roughly 100,000 non-keywords/phrases occuring in spontaneous speech. A variety of dialogue strategies are designed and evaluated based on an objective cost function having a set of actions and states as parameters. Expected dialogue cost is calculated for each strategy, and the best strategy is selected according to the keyword recognition accuracy.

This paper proposes an efficient method for large-vocabulary continuous-speech recognition, using a compact data structure and an efficient search algorithm. We introduce a very compact data structure DAWG as a lexicon to reduce the search space. We also propose a search algorithm to obtain the N-best hypotheses using the DAWG structure. This search algorithm is composed of two phases: "forward search" and "traceback". Forward search, which basically uses the time-synchronous Viterbi algorithm, merges candidates and stores the information about them in DAWG structures to create phoneme graphs. Traceback traces the phoneme graphs to obtain the N-best hypotheses. An evaluation of this method's performance using a speech-recognition-based telephone-directory-assistance system having a 4000-word vocabulary confirmed that our strategy improves speech recognition in terms of time and recognition rate.

Smoothed estimation and utterance verification are introduced into the N-best-based speaker adaptation method. That method is effective even for speakers whose decodings using speaker-independent (SI) models are error-prone, that is, for speakers for whom adaptation techniques are truly needed. The smoothed estimation improves the performance for such speakers, and the utterance verification reduces the required amount of calculation. Performance evaluation using connected-digit (four-digit strings) recognition experiments performed over actual telephone lines showed a reduction of 36.4% in the error rates for speakers whose decodings using SI models are error-prone. To try and find an effective model-transformation for speaker adaptation, we discuss replacing mixture-mean bias estimation by the widely used mixture-mean linear-regression-matrix estimation.

A large-vocabulary continuous-speech recognition (LVCSR) system was developed and evaluated. To evaluate the system, a Japanese business-newspaper speech corpus was designed and recorded. The corpus was designed so that is can be used for Japanese LVCSR research in the same way that the Wall Street Journal (WSJ) corpus, for example, is used for English LVCSR research. Since Japanese sentences are written without spaces between words, a morphological analysis was introduced to segment sentences into words so that word n-gram language models could be used. To enable the use of detailed word n-gram (n/spl ges/3) language models, a two-pass decoding strategy was applied. Context-dependent (CD) phone models and word trigram language models reduced the word error rate from 80.2% to 10.1% (an error reduction of about 88%). This result shows that CD phoneme modeling and word trigram language models can be used effectively in Japanese LVCSR.

2. BASELINE SYSTEM We performed simple speech recognition experiments for 4-digit strings to analyze the major errors in spontaneous speech . 2.1 Recognition system •Start-point and end-point detection The input of a realistic recognition system, being a continuous sequence of speech and background events, requires an efficient algorithm to distinguish the speech utterances from the surrounding background. In this study, we implemented a start-point and end-point detector based on HMM likelihoods. For every input speech frame we compared the probability between two HMMs: a background model and a composite all-digit model, whose purpose is to match any digit event. This model was trained using a large database containing samples of all digits. When the difference between the probabilities rises above an experimentally determined threshold, the start-point is set and recognition is started. The comparison is then carried on until the end-point is decided, using the same likelihood-based criterion. The thresholds were first determined so that the largest pause between two digits in a string did not exceed 320ms. •HMMs A feature vector consists of the 16-dimension mean normalized cepstrum vector and 16-dimension delta ABSTRACT In this paper, we describe a new recognition system for 4-digit-strings in Japanese under fluent speech conditions. In particular, we introduce several methods to solve the problems related to the spontaneity of speech: discrimination of speech and background noise, out-ofvocabulary words, pauses between digits, etc. These methods led to an error rate reduction of 76%, compared to a simple startand end-point detection based recognizer using non-refined models.

In this paper, we report on the automatic recognition of Japanese broadcast-news speech. We have been working on largevocabulary continuous speech recognition (LVCSR) for Japanese newspaper speech transcription and have achieved good performance. We have recently applied our LVCSR system to transcribing Japanese broadcast-news speech. We extended the vocabulary from 7k words to 20k words and trained the language models using newspaper texts and broadcast-news manuscripts. These two language models were applied to our evaluation speech sets. The language model trained using broadcast-news manuscripts achieved better results for broadcast-news speech than the language model trained using newspaper texts, which achieved better results for newspaper speech. We achieved a word error rate of 19.7% for anchor-speaker’s speech by using a bigram language model and a trigram language model both trained using broadcast-news manuscripts.

We describe a method of updating a hidden Markov model (HMM) for speaker verification using a small amount of new data for each speaker. The HMM is updated by adapting the model parameters to the new data by maximum a posteriori (MAP) estimation. The initial values of the a priori parameters in MAP estimation are set using training speech used for first creating a speaker HMM. We also present a method of resetting the a priori threshold as the updating of the model proceeds. Evaluation of the performance of the two methods using 10 male speakers showed that the verification error rate was about 42% of that without updating.

Speech understanding can be viewed as a problem of translating the input natural language of speech recognition results into an output semantic language. This paper describes automatic acquisition of a language model for translating natural language into semantic language from a text corpus using a stochastic method. The method estimates the co-occurrence probabilities of input and output grammar rules as a translation language model. Since the amount of text is limited, estimating a reliable language model is difficult. Therefore, we propose a method of concisely modeling input and output grammars in order to estimate a reliable translation model. Our method is shown to be effective by experiments using the ARPA ATIS task.

A method for adapting HMMs to additive noise and multiplicative distortion at the same time is proposed. This method first creates a noise HMM for additive noise, then composes HMMs for noisy and distorted speech data from this HMM and speech HMMs so that these composed HMMs become the functions of signal-to-noise (S/N) ratio and multiplicative distortion. S/N ratio and multiplicative distortion are estimated by maximizing the likelihood of the HMMs to the input speech. To achieve this, we propose a new method that divides the maximization process into estimation of S/N ratio and estimation of cepstrum bias. The S/N ratio is estimated using the parallel model method. The cepstrum bias is estimated using the EM algorithm. To evaluate this method, two experiments in terms of phoneme recognition and connected digit recognition are performed. The guarantee of convergence of this algorithm is also discussed.

An instantaneous speaker adaptation method is proposed that uses N-best decoding for continuous mixture-density hidden-Markov-model based speech recognition systems.An N-best paradigm of multiple-pass search strategies is used that makes this method eective even for speakers whose decodings using speaker-independent models are error-prone.To cope with an insucient amount of data, our method uses constrained maximum a posteriori estimation, in which the parameter vector space is clustered, and a mixturemean bias is estimated for each cluster.Moreover, to maintain continuity between clusters, a bias for each mixturemean is calculated as the weighted sum of the estimated biases.Performance evaluation using connected-digit (fourdigit strings) recognition experiments performed over actual telephone lines showed more than a 20% reduction in the error rates, even for speakers whose decodings using speakerindependent models were error-prone.

This paper proposes an adaptation method for universal noise (additive noise and multiplicative distortion) based on the HMM composition (compensation) technique. Although the original HMM composition can be applied only to additive noise, our new method can estimate multiplicative distortion by maximizing the likelihood value. The signal-to-noise ratio is automatically estimated as part of the estimation of multiplicative distortion. Phoneme recognition experiments show that this method improves recognition accuracy for noisy and distorted speech.

Abstract This paper investigates a speaker recognition method that is robust against background noise. In noisy environments, one important issue is how to create a model for each speaker so as to compensate for noise. The method described here is based on hidden Markov model (HMM) composition, which combines a speaker HMM and a noise-source HMM into a noise-added speaker HMM with a particular signal-to-noise ratio (SNR). Since it is difficult to measure the SNR of input speech with non-stationary noise exactly, this method creates several noise-added speaker HMMs with various SNRs. The HMM that has the highest likelihood value for the input speech is selected, and a speaker decision is made using this likelihood value. Experimental application of this method to text-independent speaker identification and verification in various kinds of noisy environments demonstrated considerable improvement in speaker recognition for speech utterances of male speakers.

Speaker adaptation methods for tied-mixture-based phoneme models are investigated for text-prompted speaker recognition. For this type of speaker recognition, speaker-specific phoneme models are essential for verifying both the key text and the speaker. This paper proposes a new method of creating speaker-specific phoneme models. This uses speaker-independent (universal) phoneme models consisting of tied-mixture HMMs and adapts the feature space of the tied-mixtures to that of the speaker through phoneme-dependent/independent iterative training. Therefore, it can adapt models of phonemes that have a small amount of training data to the speaker. The proposed method was tested using 15 speakers' voices recorded over 10 months and achieved a speaker and text verification rate of 99.4% even when both the voices of different speakers and different texts uttered by the true speaker were to be rejected.>

This paper describes a phonetic typewriter and a dictation machine that utilize the underlying statistical structure of phoneme or character sequences. The approach of using syllable or character trigrams is applied to language source modeling. The language source models are obtained by calculating trigram probabilities from a large text database. These models are combined with the HMM-LR continuous speech recognition system.3,6 The phonetic typewriter is tested using 274 phrases uttered by one male speaker. The syllable source model achieves a 94.9% phoneme recognition rate with the test-set phoneme perplexity of 3.9. Without the syllable source model, the phoneme recognition rate is only 73.2%. A trigram model based on characters is also evaluated. This character source model can reduce the syllable perplexity significantly to 7.7, compared with 10.5 of the syllable source model. The character source model achieves a 78.5% character transcription rate for the 274 phrase utterances. The experimental results show that a syllable source model and a character source model are very effective for realizing a Japanese dictation machine.

This paper compares a VQ (vector quantization)-distortion-based speaker recognition method and discrete/continuous ergodic HMM (hidden Markov model)-based ones, especially from the viewpoint of robustness against utterance variations. The authors show that a continuous ergodic HMM is as robust as a VQ-distortion method when enough data is available and that a continuous ergodic HMM is far superior to a discrete ergodic HMM. They also show that the information on transitions between different states is ineffective for text-independent speaker recognition. Therefore, the speaker recognition rates using a continuous ergodic HMM are strongly correlated with the total number of mixtures irrespective of the number of states.>

A VQ (vector quantization)-distortion-based speaker recognition method and discrete/continuous ergodic HMM (hidden Markov model)-based ones are compared, especially from the viewpoint of robustness against utterance variations. It is shown that a continuous ergodic HMM is far superior to a discrete ergodic HMM. It is also shown that the information on transitions between different states is ineffective for text-independent speaker recognition. Therefore, the speaker identification rates using a continuous ergodic HMM are strongly correlated with the total number of mixtures irrespective of the number of states. It is also found that, for continuous ergodic HMM-based speaker recognition, the distortion-intersection measure (DIM), which was introduced as a VQ-distortion measure to increase the robustness against utterance variations, is effective.>

The authors describe a VQ (vector-quantization)-based text-independent speaker recognition method which is robust against utterance variations. Three techniques are introduced to cope with temporal and text-dependent spectral variations. First, either an ergodic hidden Markov model or a voiced/unvoiced decision is used to classify input speech into broad phonetic classes. Second, a new distance measure, the distortion-intersection measure (DIM), is introduced for calculating VQ distortion of input speech compared to speaker-independent codebooks. Third, a normalization method, talker variability normalization (TVN), is introduced. TVN normalizes parameter variation taking both inter- and intra-speaker variability into consideration. The system was tested using utterances of nine speakers recorded over three years. The combination of the three techniques achieves high speaker identification accuracies of 98.5% using only vocal tract information and 99.0% using both vocal tract and pitch information.>

A vector quantization (VQ)-based recognition method which uses feature vector codebooks containing hierarchical spectral dynamics is proposed. This method is highly effective for reducing the number of candidates in word recognition and achieving a high recognition accuracy in /b/,/d/ and /g/ recognition. Since this method does not need time alignment, it has the advantage of a small amount of computation and ease of parallel processing. Experimental results comparing the performances of the multiple-codebook and single-codebook methods indicate that, when the codebook size is small, the multiple-codebook method is better than the single-codebook method. However, if the codebook size is reasonably large, the single-codebook method displays better performance than the multiple-codebook method.>

A Japanese continuous-speech recognition system is described which is based on phonetic hidden Markov models (HMMs) combined with two levels of grammatical representations: an intraphase transition network grammar and an interphase dependency grammar. A joint score, combining acoustic likelihood and linguistic certainty factors derived from phonetic HMMs and two levels of grammar, is maximized to obtain the optimal recognition results of sentences. Two efficient algorithms, bidirectional network parsing and breadth-first dependency parsing, are deviced to optimize the joint score globally. The system attains a phrase recognition rate of 80.8% with the intraphase parser only and 86.8% with both the intraphase and interphase parsers, where the perplexity of the phrase syntax is 40. This result shows the effectiveness of the two-level grammar approach.>

An automatic speaker adaptation method is proposed for speech recognition in which a small amount of training material of unspecified text can be used. This method is easily applicable to vector-quantization-based speech recognition systems where each word is represented as multiple sequences of codebook entries. In the adaptation algorithm, either the codebook is modified for each new speaker or input speech spectra are adapted to the codebook, thereby using codebook sequences universally for all speakers. The important feature of this algorithm is that a set of spectra in training frames and the codebook entries are clustered hierarchically. Based on the deviation vectors between centroids of the training frame clusters and the corresponding codebook clusters, adaptation is performed hierarchically from small to large numbers of clusters. Results of recognition experiments indicate that the proposed adaptation method is highly effective. Possible variations using this method are presented.>

The authors propose a new mathematical method for extracting spectral movement in a time-sequence of speech spectrum. The spectral movement is characterized by the time and frequency derivative of a time-sequence of log spectrum envelopes. Spectral movement direction, the movement toward a higher or lower frequency region, can be identified by the sign of the proposed function. A parameter which can be used for speech segmentation is derived form this function A distance measure for speech recognition is also derived as the Euclidean distance between two spectral movement patterns extracted by the proposed function. This distance is easily calculated using cepstrum coefficients. Speech recognition results using dynamic time warping template matching with this new distance measure indicate that recognition error rate can be reduced to less than half compared with the conventional Euclidean cepstrum distance measure.>

This paper proposes a new VQ (Vector Quantization)- based preprocessor for use in a method which reduces the amount of computation necessary in speaker-independent large vocabulary isolated word recognition. A speech wave is analyzed by time functions of instantaneous cepstrum coefficients and short-time regression coefficients for both cepstrum coefficients and logarithmic energy. A universal VQ codebook for these time functions is constructed based on a multi-speaker, multi-word database. Next, a separate codebook is designed as a subset of the universal codebook for each word in the vocabulary. These word-specific codebooks are used for front-end preprocessing to eliminate word candidates whose distance scores are large. A dynamic time-warping processor based on a word dictionary, in which each word is represented as a time-sequence of the universal codebook elements (SPLIT method), then resolves the choice among the remaining word candidates. Effectiveness of this method has been ascertained by recognition experiments using a database consisting of words from a vocabulary of 100 Japanese city names uttered by 20 male speakers.

A new speech analysis technique applicable to speech recognition is proposed considering the auditory mechanism of speech perception which emphasizes spectral dynamics and which compensates for the spectral undershoot associated with coarticulation. A speech wave is represented by the LPC cepstrum and logarithmic energy sequences, and the time sequences over short periods are expanded by the first- and second-order polynomial functions at every frame period. The dynamics of the cepstrum sequences are then emphasized by the linear combination of their polynomial expansion coefficients, that is, derivatives, and their instantaneous values. Speaker-independent word recognition experiments using time functions of the dynamics-emphasized cepstrum and the polynomial coefficient for energy indicate that the error rate can be largely reduced by this method.

This paper describes new technique for use in a word recognition system. This recognition system is especially efffective in speaker-dependent large vocabulary word recognition based on multiple reference templates. In this system, word templates are represented as sequences of descrete phoneme-like (pseudo-phoneme) templates which are automatically determined from a training set of word utterances by a clustering technique. In speaker-dependent 641 city names word recognition experiments, 96.3% recognition accuracy was obtained using 256 phoneme-like templates.

New techniques for automatic speaker verification are described which are based on cepstral analysis of fixed sentence-long utterances. Utterances are represented by time functions of cepstral coefficients expanded by an orthogonal polynomial. Sample utterance representations are compared with similarly constituted reference patterns to decide whether to accept or reject an identity claim. A dynamic programming technique is used to bring sample utterance representations into time registration with reference patterns. A local distance measurement is summed along the optimal path associated with the time registration to provide an overall distance which is compared with a threshold to effect the decision. Results of an evaluation using prerecorded utterance sets including telephone speech are presented. Verification error rates less 1% are obtained.