Automatic Speech Verification (ASV) systems are highly vulnerable to spoofing attacks, and replay attack poses the greatest threat among various spoofing attacks. In this paper, we propose a novel multi-channel feature extraction method with attention-based adaptive filters (AAF). Original phase information, discarded by conventional feature extraction techniques after Fast Fourier Transform (FFT), is promising in distinguishing genuine from replay spoofed speech. Accordingly, phase and magnitude information are respectively extracted as phase channel and magnitude channel complementary features in our system. First, we make discriminative ability analysis on full frequency bands with F-ratio methods. Then attention-based adaptive filters are implemented to maximize capturing of high discriminative information on frequency bands, and the results on ASVspoof 2017 challenge indicate that our proposed approach achieved relative error reduction rates of 78.7% and 59.8% on development and evaluation dataset than the baseline method.

Deep neural networks (DNN) have achieved significant success in the field of automatic speech recognition. Previously, we proposed a filterbank-incorporated DNN which takes power spectra as input features. This method has a function of VTLN (Vocal tract length normalization) and fMLLR (feature-space maximum likelihood linear regression). The filterbank layer can be implemented by using a small number of parameters and is optimized under a framework of backpropagation. Therefore, it is advantageous in adaptation under limited available data. In this paper, speaker adaptation is applied to the filterbank-incorporated DNN. By applying speaker adaptation using 15 utterances, the adapted model gave a 7.4% relative improvement in WER over the baseline DNN at a significance level of 0.005 on CSJ task. Adaptation of filterbank layer also showed better performance than the other adaptation methods; singular value decomposition (SVD) based adaptation and learning hidden unit contributions (LHUC).

One of the difficulties in sung speech recognition is the small distance in an acoustic space between phonemes in sung speech. Therefore we considered clustering the speech based on a pitch (fundamental frequency F0) and creating a larger distance between the phonemes. In addition, we considered a two-stage training method of DNN-HMM: the first stage is trained by using conventional acoustic features like MFCCs, and the second stage is re-trained by augmented features with a pitch feature. We expected to train pitch information more explicitly in the second stage of the training and obtained a relative improvement of 9% as expected.

Deep neural networks (DNN) have achieved significant success in the field of speech recognition. One of the main advantages of the DNN is automatic feature extraction without human intervention. Therefore, we incorporate a pseudo-filterbank layer to the bottom of DNN and train the whole filterbank layer and the following networks jointly, while most systems take pre-defined mel-scale filterbanks as acoustic features to DNN. In the experiment, we use Gaussian functions instead of triangular mel-scale filterbanks. This technique enables a filterbank layer to maintain the functionality of frequency domain smoothing. The proposed method provides an 8.0% relative improvement in clean condition on ASJ+JNAS corpus and a 2.7% relative improvement on noise-corrupted ASJ+JNAS corpus compared with traditional fully-connected DNN. Experimental results show that the frame-level transformation of filterbank layer constrains flexibility and promotes learning efficiency in acoustic modeling.

Phase information is ignored for almost all voice activity detection (VAD). To exploit full information in the original signal, this paper proposes a deep neural network (DNN) using magnitude and phase information (that is, phase aware DNN) to achieve better VAD performance. Mel-frequency cepstral coefficient (MFCC), power-normalized cepstral coefficients (PNCC), instantaneous frequency derivative (IF), baseband phase difference (BPD) and modified group delay cepstral coefficient (MGDCC) are used as magnitude and phase information. The proposed methods were evaluated using CENSREC-1-C database under noise condition. The results show that the phase aware DNN significantly outperforms the DNN using only magnitude information. For DNN-based classifier, the equal error rate (EER) was reduced from 23.70% of MFCC, to 20.43% of joint dual magnitude and single phase features (augmenting PNCC, MGDCC and IF), to 19.92% of joint dual phase and single magnitude feature features (augmenting PNCC, MGDCC and BPD). By combining joint dual magnitude and single phase features with joint dual phase and single magnitude features, the EER was reduced to 19.44%.

Lyric recognition in singing is challenging because of a number of problems, including a lack of singing databases, superposed musical instruments and different spectral variations. First of all, we investigated the difference of spectral variations among read speech, spontaneous speech and sung speech and we found that sung speech recognition was the most difficult. Next, we consider Japanese lyric recognition in monophonic singing that contains no musical instruments. To express singing well, we use an n-gram language model with a lyrics corpus, singing-adapted acoustic models, and plural pronunciation lexicons for vowel-lengthening. We also compare GMM-HMM and DNN-HMM acoustic models. We obtained a remarkable improvement on lyric recognition in comparison with the baseline system for spontaneous speech recognition.

This paper presents a Japanese spoken term detection method for spoken queries using a combination of word-based search and syllable-based N-gram search with in-vocabulary/out-of-vocabulary (IV/OOV) term classification. The N-gram index in a recognized syllable-based lattice for OOV terms, which assumes recognition errors such as substitution, insertion and deletion errors, incorporates a distance metric as a confidence score. To address spoken queries, we propose an automatic method for discriminating IV and OOV terms by using the confidence scores of spoken queries through large-vocabulary/syllable continuous speech recognition. Evaluation on an academic lecture presentation database with 44 hours of data shows that the combination of word search and syllable-based N-gram search yields significant improvement and outperforms the baseline syllable-based DTW approach.

Recently, acoustic models based on deep neural notworks (DNNs) have been introduced and showed dramatic improvements over acoustic models based on GMM in a variety of tasks. In this paper, we considered the improvement of noise robustness of DNN. Inspired by Missing Feature Theory and static noise aware training, we proposed an approach that uses a noise-suppressed acoustic feature and estimated noise information as input of DNN. We used simple Spectral Subtraction as noise-suppression. As noise estimation, we used estimation per utterance or frame. In noisy speech recognition experiments, we compared the proposed method with other methods and the proposed method showed the superior performance than the other approaches. For noise estimation per utterance with log Mel Filterbank, we obtained 28.6% word error rate reduction compared with multi condition training, 5.9% reduction compared with noise adaptive training. Index Terms: speech recognition, deep neural network, noise robustness, adaptive training, noise aware training, Spectral Subtraction, Missing Feature Theory

The detection of human and spoofed (synthetic/converted) speech has started to receive more attention. In this study, relative phase information extracted from a Fourier spectrum is used to detect human and spoofed speech. Because original/natural phase information is almost entirely lost in spoofed speech using current synthesis/conversion techniques, a modified group delay based feature, the frequency derivative of the phase spectrum, has been shown effective for detecting human speech and spoofed speech. The modified group delay based phase contains both the magnitude spectrum and phase information. Therefore, the relative phase information, which contains only phase information, is expected to achieve a better spoofing detection performance. In this study, the relative phase information is also combined with the Mel-Frequency Cepstral Coefficient (MFCC) and modified group delay. The proposed method was evaluated using the “ASVspoof 2015: Automatic Speaker Verification Spoofing and Countermeasures Challenge” dataset. The results show that the proposed relative phase information significantly outperforms the MFCC and modified group delay. The equal error rate (EER) was reduced from 1.74% of MFCC, 0.83% of modified group delay to 0.013% of relative phase. By combining the relative phase with MFCC and modified group delay, the EER was reduced to 0.002%. Index Terms: Spoofing detection, relative phase information, group delay, GMM, countermeasures

We considered a speech recognition method for mixed sound, which is composed of both speech and music, that only removes music based on non-negative matrix factorization (NMF). We used Itakura-Saito divergence instead of Kullback-Leibler divergence to compare the cost function, and the dynamics and sparseness constraints of a weight matrix to improve speech recognition. For isolated word recognition using the matched condition model, we reduced the word error rate of 52:1% relative from the case that didn’t remove music (on average, from 69.3% to 85.3%). Index Terms: speech recognition, mixed sound, music removal, vector quantization, non-negative matrix factorization

Almost all current spoken dialog systems treat dialog as that where a single user talks to an agent. We, on the other hand, set out to investigate a multiparty dialog system that deals with two agents and a single user. We developed a three person (one user and two agents) and a two person (one user and one agent) dialog system to consider the same dialog task, that is, “Which do you prefer, udon or ramen (Japanese noodle or Chinese noodle)?” and compared them with respect to user behavior and satisfaction. According to the results of the experiments, the three person dialog system performed better in terms of lively conversation, and user can talk with the agents more like chatting.

A method for semi-supervised language modeling, which was designed to improve the robustness of a language model (LM) obtained from manually transcribed (labeled) data, is proposed. The LM is implemented as a log-linear model, which employs a set of linguistic features derived from word or phoneme n-grams. The proposed method is formulated as a multi-objective optimization programming problem (MOP), which consists of two objective functions based on expected risks for labeled lattices and automatic speech recognition (ASR) lattices as unlabeled training data. The model is trained in a discriminative manner and acquired as a solution to the problem. In transcribing Japanese broadcast programs, the proposed method reduced word error rate by 6.3% compared with that achieved by a conventional trigram LM.

In conventional speaker recognition methods based on Mel-frequency cepstral coefficients (MFCCs), phase information has hitherto been ignored. In this paper, we propose a phase information extraction method that normalizes the change variation in the phase according to the frame position of the input speech and combines the phase information with MFCCs in text-independent speaker identification and verification methods. There is a problem with the original phase information extraction method when comparing two phase values. For example, the difference in the two values of π-\mathtildeθ1and \mathtildeθ2=-π+\mathtildeθ1is 2π-2\mathtildeθ1. If \mathtildeθ1≈ 0, then the difference ≈ 2π, despite the two phases being very similar to one another. To address this problem, we map the phase into coordinates on a unit circle. Speaker identification and verification experiments are performed using the NTT database which consists of sentences uttered by 35 (22 male and 13 female) Japanese speakers with normal, fast and slow speaking modes during five sessions. Although the phase information-based method performs worse than the MFCC-based method, it augments the MFCC and the combination is useful for speaker recognition. The proposed modified phase information is more robust than the original phase information for all speaking modes. By integrating the modified phase information with the MFCCs, the speaker identification rate was improved to 98.8% from 97.4% (MFCC), and equal error rate for speaker verification was reduced to 0.45% from 0.72% (MFCC), respectively. We also conducted the speaker identification and verification experiments on a large-scale Japanese Newspaper Article Sentences (JNAS) database, a similar trend as NTT database was obtained.

A topic-dependent-class (TDC)-basedn-gram language model (LM) is a topic-based LM that employs a semantic extraction method to reveal latent topic information extracted from noun-noun relations. A topic of a given word sequence is decided on the basis of most frequently occuring (weighted) noun classes in the context history through voting. Our previous work (W. Naptali, M. Tsuchiya, and S. Seiichi, “Topic-dependent language model with voting on noun history,”ACM Trans. Asian Language Information Processing (TALIP), vol. 9, no. 2, pp. 1-31, 2010) has shown that in terms of perplexity, TDCs outperform several state-of-the-art baselines, i.e., a word-based or class-basedn-gram LM and their interpolation, a cache-based LM, ann-gram-based topic-dependent LM, and a Latent Dirichlet Allocation (LDA)-based topic-dependent LM. This study is a follow up of our previous work and there are three key differences. First, we improve TDCs by employing soft-clustering and/or soft-voting techniques, which solve data shrinking problems and make TDCs independent of the word-basedn-gram, in the training and/or test phases. Second, for further improvement, we incorporate a cache-based LM through unigram scaling, because the TDC and cache-based LM capture different properties of the language. Finally, we provide an evaluation in terms of the word error rate (WER) and an analysis of the automatic speech recognition (ASR) rescoring task. Experiments performed on the Wall Street Journal and the Mainichi Shimbun (a Japanese newspaper) demonstrate that the TDC LM improves both perplexity and the WER. The perplexity reduction is up to 25.1% relative on the English corpus and 25.7% relative on the Japanese corpus. Furthermore, the greatest reduction in the WER is 15.2% relative to the English ASR and 24.3 relative to the Japanese ASR, as compared to the baseline.

Although large-scale spontaneous speech corpora are crucial resource for various domains of spoken language processing, they are usually limited due to their construction cost especially in transcribing precisely. On the other hand, inexact transcribed corpora like shorthand notes, meeting records and closed captions are widely available. Unfortunately, it is difficult to use them directly as speech corpora for learning acoustic models, because they contain two kinds of text, precisely transcribed parts and edited parts. In order to resolve this problem, this paper proposes an automatic detection method of precisely transcribed parts from inexact transcribed corpora. Our method consists of two steps: the first step is an automatic alignment between the inexact transcription and its corresponding utterance, and the second step is a support vector machine based detector of precisely transcribed parts using several features obtained by the first step. Experiments using the Japanese National Diet Record shows that automatic detection of precise parts is effective for lightly supervised speaker adaptation, and shows that it achieves reasonable performance to reduce the converting cost from inexact transcribed corpora into precisely transcribed ones.

Hidden Conditional Random Fields(HCRF) is a very promising approach to model speech. However, because HCRF computes the score of a hypothesis by summing up linearly weighted features, it cannot consider non-linearity among features that will be crucial for speech recognition. In this pa per, we extend HCRF by incorporating gate function used in neural networks and propose a new model called Hidden Conditional Neural Fields(HCNF). Differently with conventional approaches, HCNF can be trained without any initial model and incorporate any kinds of features. Experimental results of continuous phoneme recognition on TIMIT core test set and Japanese read speach recognition task using monophone showed that HCNF was superior to HCRF and HMM trained in MPE manner.

For spoken document retrieval, it is very important to con sider Out-of-Vocabulary (OOV) and mis-recognition of spoken words. Therefore, sub-word unit based recognition and retrieval methods have been proposed. This paper describes a Japanese spoken document retrieval system that is robust for considering OOV words and mis-recognition of sub-units. We used individual syllables as sub-word unit in continuous speech recognition and an n-gram sequence of syllables in a recognized syllable-based lattice. We propose an n-gram indexing/retrieval method with distance in the syllable lattice for attacking OOV, recognition errors, and high speed retrieval. We applied this method to academic lecture presentation database of 44 hours, and 0.58(F-value) of the OOV words were detected in less than 2.5 milliseconds.

We have investigated automatic speech recognition using Hidden Conditional Neural Fields (HCNF). In this paper, we propose a new objective function, Hidden Boosted MMI (HBMMI) that considers the number of errors in the training data even if the correct state sequence is not known for training the HCNF. The experimental results show that HB-MMI can improve recognition accuracy if overfitting does not occur. We also present an automatic speech recognition method using a hierarchical state posterior feature where the output from the first stage HCNF is used as input for the second stage HCNF. The experimental results show that the feature improves recognition accuracy. By combining both of the proposed methods, we obtain further improvements. Index Terms: hidden conditional neural fields, automatic speech recognition, hidden boosted MMI, state posterior feature

We have previously proposed a statistical method for estimating the pronunciation proficiency and intelligibility of presentations made in English by non-native speakers. To investigate the relationship between various acoustic measures and the pronunciation score and intelligibility, we statistically analyzed the speaker’s actual utterances to find combinations of acoustic features with a high correlation between the score estimated by a linear regression model and the score perceived by native English teachers. In this paper, we examined the quality of new acoustic features that are useful when used in combination with the system’s estimates of pronunciation score and intelligibility. Results showed that the best combination of acoustic features produced correlation coefficients of 0.929 and 0.753 for pronunciation and intelligibility, respectively, using open data for speakers at the 10-sentence level. Index Terms: pronunciation evaluation, intelligibility evaluation, English, HMM, phoneme pair, perplexity

This paper describes a lattice-based risk minimization training method for unsupervised language model (LM) adaptation. In a broadcast archiving system, unsupervised LM adaptation us-ing transcriptions generated by speech recognition is consid-ered to be useful for improving the performance. However, conventional linear interpolation methods occasionally degrade the performance because of incorrect words in the training tran-scriptions. Accordingly, we propose a new adaptation method aiming to reflect error information among training lattices. The method minimizes the whole risk of training lattices to yield a log-linear model, which consists of a set of linguistic fea-tures. The advantage of the method is that the model parameters can be obtained efficiently in an unsupervised manner. Experi-mental results obtained in transcribing Japanese broadcast news showed significant word error rate reduction for those of con-ventional mixture LMs. Index Terms: unsupervised learning, Bayes risk minimization, language model adaptation, speech recognition

In conventional speaker recognition methods based on MFCC, the phase information has been ignored. Recently, we proposed a method that integrated MFCC with the phase information on a speaker recognition method. Using the phase information, the speaker identification error rate was reduced by 78% for clean speech. In this paper, we describe the effectiveness of phase information for noisy environments on speaker identification. Integrationg MFCC with phase information, the speaker error identification rates were reduced by 20% ~70% in comparison with using only MFCC in noisy environments.

This paper presents a method for improving the readability of Automatic Speech Recognition (ASR) results for classroom lectures. Most of the previous research on improving the readability of recognition results focused mainly on manually transcribed texts, and not ASR results. Due to the presence of a large number of domain-dependent words and the casual presentation style, even state-of-the-art recognizers yield a 30-50% word error rate for speech in classroom lectures. Thus, a method for improving the readability of ASR results needs to be robust against recognition errors. In this paper, we propose a novel method for improving the readability based on a machine translation model that uses a confusion network representing multiple hypotheses of the ASR results to achieve robustness against recognition errors. Experimental results show that the proposed method outperforms the baselines in both automatic and manual evaluations. Index Terms: improving readability, confusion network, automatic speech recognition, classroom lecture speech

In this paper, we propose a method for estimating a score for English pronunciation. Scores estimated by the proposed method were evaluated by correlating them with the learner’s pronunciation score which was scored by native English teachers. The average correlation between the estimated pronunciation scores and the learner’s pronunciation scores over 1, 5, and 10 sentences was 0.807, 0.873, and 0.921, respectively. When a text of spoken sentence was unknown, we obtained a correlation of 0.878 for 10 utterances. For English phonetic evaluation, we classified English phoneme pairs that are difficult for Japanese speakers to pronounce, using SVM, NN, and HMM classifiers. The correct classification ratios for native English and Japanese English phonemes were 94.6% and 92.3% for SVM, 96.5% and 87.4% for NN, 85.0% and 69.2% for HMM, respectively. We then investigated the relationship between the classification rate and a proficiency score of non-native learner’s English pronunciation, and obtained a high correlation of 0.6 ∼ 0.7. Index Terms: pronunciation evaluation, English, Japanese, HMM, SVM, Neural Network

Spoken Document Retrieval (SDR) and Spoken Term Detection (STD) have been two of the most intensively investigated topics in spoken document processing research according to the establishment of the SDR and STD test collections by the Text REtrieval Conference (TREC) and NIST. Because Japanese spoken document processing researchers also requires such test collections for SDR and STD, we have established a working group to develop these collections in Special Interest Group-Spoken Language Processing (SIG-SLP) of the Information Processing Society of Japan. The working group has constructed and made available a test collection for SDR, and is now constructing new test collections for STD that will be open to researchers. The present paper introduces the policies, outline, and schedule of the new test collections. Then, the new test collections are compared with the NIST STD test collections. Index Terms: spoken term detection, test collection 1.

We developed a supporting system for creation of educational video contents. The system automatically segments a lecture video material into subtopics based on speech signals by a statistical model for text segmentation. In this paper, we reports on the result of retrieving the lecture subtopics by keyword expansion using the knowledge of the dictionary, and so on. The keyword expansion using the subordinate concept improved the average reciprocal order (MRR:Mean Reciprocal Rank) from 0.51 to 0.55 when subtopics are retrieved by a set of three search keywords for the lecture voice text recognized by automatic speech recognition. Index Terms: subtopic retrieval, keyword expansion, automatic speech recognition, lecture video

A topic dependent class (TDC) [1] language model (LM) is a topic-based LM that uses a semantic extraction method to re-veal latent topic information from nouns relation. Then a clus-tering for a given context is performed to define topics. Finally, a fixed window of word history is observed to decide the topic of the current event through voting in online manner. Previously, we have shown that TDC overperforms several state-of-the-art baselines. There are two separate points that we would like to introduce in this paper. First, we improves the TDC further by incorporating cache-based LM through unigram scaling. The combination is possible since TDC only tried to capture top-ical words, and does not models re-occurring words, such as functional words, very well. Experiments on Wall Street Jour-nal (WSJ) and Japanese newspaper (Mainichi Shimbun) corpora show that this combination improves the model significantly in terms of perplexity. Second, TDC stand-alone model suffers from shrinking training corpus size when the number of top-ics is increased. We solved this problem by performing soft-clustering and soft-voting on the training and test phase. Exper-iments result on WSJ corpus shows that TDC performance over perform the baseline without being needed to be interpolated with the word-based n-gram. Index Terms: topic dependent, language model, latent seman-tic analysis, soft voting, soft clustering, cache

Current speech recognition systems use mainly amplitude spectrum-based features such as MFFC for acoustic feature parameters, while discarding phase spectral information. The results of perceptual experiments, however, suggested that phase spectral information based on long-term analysis includes certain linguistic information. In this paper, we propose the use of phase features based on long-term analysis for speech recognition. We use two types of parameters: the delta phase parameter as a group delay and analytic group delay features. Isolated word and continuous digit recognition experiments were performed, resulting in a greater than 90% word or digit accuracy for each of the experiments. The experimental results confirmed that a long-term phase spectrum includes sufficient information for recognizing speech. Furthermore, combining likelihoods of MFCC and long-term group delay cepstrum outperformed the baseline MFCC relatively 20% for clean speech. Index Terms: speech recognition, phase information, longterm analysis, group delay

For spoken document retrieval, it is very important to consider Out-of-Vocabulary (OOV) and mis-recognition of spoken words. Therefore, sub-word unit based recognition and retrieval methods have been proposed. This paper describes a Japanese spoken document retrieval system that is robust for considering OOV words and mis-recognition of sub-units. To solve the problem of OOV keywords and mis-recognized words, we used individual syllables as sub-word unit in continuous speech recognition and an n-gram sequence of syllables as a retrieval unit. We propose an n-gram indexing/retrieval method with distance in a syllable lattice for attacking OOV, recognition errors, and high speed retrieval. We applied this method to academic lecture presentation database of 44 hours, and 60% of the OOV words were detected in less than 2.5 milliseconds.

A topic dependent class (TDC) language model (LM) is a topic-based LM that uses a semantic extraction method to reveal latent topic information from noun-document relation. Then a clustering for a given context is performed to define topics. Finally, a fixed window of word history is observed to decide the topic of the current event through voting in online manner. Previously, we have shown that TDC overperforms several state-of-the-art baselines in terms of perplexity. In this paper we evaluate TDC on automatic speech recognition experiment (ASR) for rescoring task. Experiments on read speech Wall Street Journal (English ASR system) and Mainichi Shimbun (Japanese ASR system) show that TDC LM improves both perplexity and word-error-rate (WER). The result shows that the proposed model gives improvements 3.0% relative on perplexity and 15.2% relative on WER for English ASR system, and 16.4% relative on perplexity and 24.3% relative on WER for Japanese ASR system.

Language models (LMs) are an important field of study in automatic speech recognition (ASR) systems. LM helps acoustic models find the corresponding word sequence of a given speech signal. Without it, ASR systems would not understand the language and it would be hard to find the correct word sequence. During the past few years, researchers have tried to incorporate long-range dependencies into statistical word-based n -gram LMs. One of these long-range dependencies is topic. Unlike words, topic is unobservable. Thus, it is required to find the meanings behind the words to get into the topic. This research is based on the belief that nouns contain topic information. We propose a new approach for a topic-dependent LM, where the topic is decided in an unsupervised manner. Latent Semantic Analysis (LSA) is employed to reveal hidden (latent) relations among nouns in the context words. To decide the topic of an event, a fixed size word history sequence (window) is observed, and voting is then carried out based on noun class occurrences weighted by a confidence measure. Experiments were conducted on an English corpus and a Japanese corpus: The Wall Street Journal corpus and Mainichi Shimbun (Japanese newspaper) corpus. The results show that our proposed method gives better perplexity than the comparative baselines, including a word-based/class-based n -gram LM, their interpolated LM, a cache-based LM, a topic-dependent LM based on n -gram, and a topic-dependent LM based on Latent Dirichlet Allocation (LDA). The n -best list rescoring was conducted to validate its application in ASR systems.

Ubiquitous network society will be achieved soon. In the society, all electronic equipments including "sensors" are connected to the network and communicate each other to share information. Sensor information is very important for the network, especially for virtual reality systems which give feeling of begin there. Since the sensor information, however, includes a lot of privacy information, it is not preferred to send raw privacy information through the network. In this paper, we describe about privacy protection for speech information. We think that the privacy information in speech is "voice characteristics" and "linguistic privacy information." We try to protect these privacy information by using "voice conversion" and "deletion of privacy linguistic information for speech recognition result." Since, however, speech recognition technology is not robust in real environment still now, "elimination of only speech in noisy speech" technique is also considered.

If a dialog system can respond to a user as naturally as a human, the interaction will appear smoother. In this research, we aim to develop a dialog system that emulates human behavior in a chat-like dialog. The proposed system makes use of a decision tree to generate chat-like responses at the appropriate times. These responses include ¿aizuchi¿ (back-channel), ¿repetition¿, ¿collaborative completion¿, etc. The system also reacts robustly to the user's overlapping utterances (barge-in) and disfluencies. The subjective evaluation shows that there is a high degree of naturalness in the timing of ordinary responses, overlap, and aizuchi, and that the dialog system exhibits user-friendly behavior. The recorded voices system was preferred, and almost all subjects felt familiarity with aizuchi, and the barge-in was also useful.

In conventional speaker recognition methods based on MFCC, phase information has been ignored. We proposed a method that integrated the phase information with MFCC on a speaker identification method, and a preliminary experiment was performed. In this paper, we propose a new modified feature parameter (that is, coordidates on an unit circle) obtained from the original phase information, and evaluated it by using speech database consisting of normal, fast and slow speaking modes. The speaker identification experiments were performed using NTT database which consists of sentences uttered by 35 Japanese speakers (22 males and 13 females) on five sessions over ten months. Each speaker uttered only 5 training utterances at a normal speaking mode (about 20 seconds in total). The proposed new phase information was more robust than the original phase information for all speaking modes. By integrating the new phase information with the MFCC, the speaker identification error rate was remarkably reduced for normal, fast and slow speaking rates in comparison with a standard MFCC-based method. In this paper, speaker verification experiments were also evaluated using the phase information. The experiments show that the phase information is also very useful for the speaker verification.

We propose a method that finds the position and orientation of an acoustic source in an enclosed environment. For each of eight T-shaped arrays forming a microphone array network, the time delay of arrival (TDOA) of signals from microphone pairs, a source position candidate, and energy related features are estimated. These form the input for artificial neural networks (ANNs), the purpose of which is to provide indirectly a more precise position of the source and, additionally, to estimate the source’s orientation using various combinations of the estimated parameters. The best combination of parameters (TDOAs and microphone positions) yields a 21.8% reduction in the mean average position error compared to baselines, and a correct orientation ratio higher than 99.0%. The position estimation baselines include two estimation methods: a TDOAbased method that finds the source position geometrically, and the SRP-PHATthat finds the most likely source position by spatial exploration. Index Terms: microphone array network, position and orientation estimation, artificial neural network

Language models (LMs) are important in automatic speech recognition systems. In this paper, we propose a new approach to a topic dependent LM, where the topic is decided in an unsupervised manner. Latent Semantic Analysis (LSA) is employed to reveal hidden (latent) relations among nouns in the context words. To decide the topic of an event, a fixed size word history sequence (window) is observed, and voting is then carried out based on noun class occurrences weighted by a confidence measure. Experiments on the Wall Street Journal corpus and Mainichi Shimbun (Japanese newspaper) corpus show that our proposed method gives better perplexity than the comparative baselines, including a word-based/class-based n-gram LM, their interpolated LM, a cache-based LM, and the Latent Dirichlet Allocation (LDA)-based topic dependent LM. Index Terms: topic dependent, language model, latent semantic analysis, topic voting

This paper addresses mismatch between speech processing units used by a speech recognizer and sentences of corpora. A standard speech recognizer divides an input speech into speech processing units based on its power information. On the other hand, training corpora of language models are divided into sentences based on punctuations. There is inevitable mismatch between speech processing units and sentences, and both of them are not optimal for a spontaneous speech recognition task. This paper presents two sub issues to address this problem. At first, the words of the preceding units are utilized to predict the words of the succeeding units, in order to address the mismatch between speech processing units and optimal units. Secondly, we propose a method to build a language model including short pause from a corpus with no short pause to address the mismatch between speech processing units and sentences. Their combination achieved a 4.5% relative improvement over the conventional method in the meeting speech recognition task. Index Terms: speech recognition, language model, pause information, processing unit for speech recognition

This paper presents a novel sentence extraction framework that takes into account the consecutiveness of important sentences using a Support Vector Machine (SVM). Generally, most ex-tractive summarizers do not take context information into ac-count, but do take into account the redundancy over the entire summarization. However, there must exist relationships among the extracted sentences. Actually, we can observe these rela-tionships as consecutiveness among the sentences. We deal with this consecutiveness by using dynamic and difference features to decide if a sentence needs to be extracted or not. Since impor-tant sentences tend to be extracted consecutively, we just used the decision made for the previous sentence as the dynamic fea-ture. We used the differences between the current and previ-ous feature values for the difference feature, since adjacent sen-

This paper discusses the speech recognition of Japanese classroom lecture speech. In particular, we mention the influences of microphone differences and the language model differences on the speech recognition performance of classroom lectures. First, we collected actual classroom lecture contents from several universities in Japan. In this paper, we recorded the lecture speech using lapel microphones because lapel microphones are more commonly used to record lectures. LVCSR is one of the essential technologies for adding tag information to such lecture speech. Next, therefore, we researched the influence of the differences between microphones used for recording lecture on speech recognition performance. Finally, seven types of language models that were trained using three types of corpora were compared on the basis of their ability to lecture speech.

If a dialog system could respond to a user as naturally as a human, the interaction would be smoother. Imitating human prosodic characteristics of utterances is important in computer-to-human natural interaction. To develop a cooperative/friendly spoken dialog system, we analyzed the correlation between the fundamental frequency’s synchrony tendency, or overlap fre-quency, and subjective measures of “liveliness”, “familiarity”, and “informality ” in human-to-human dialogs. We also mod-eled the properties of these features to realize chat-like conver-sations in our spoken dialog system. Index Terms: spoken dialog system, prosody control, response timing, chat-like conversation

We propose a method that uses a filler prediction model for building a language model that includes fillers from a cor-pus without fillers. In our method, a filler prediction model is trained from a corpus that does not cover domain-relevant topics. It recovers fillers in inexact transcribed corpora in the target domain, and then a language model that includes fillers is built from the corpora. The results of an evaluation of the Japanese National Diet Record showed that a model using our method achieves higher recognition performance than conven-tional ones.

Developing technologies to summarize and retrieve huge quantities of spoken documents, recorded during classroom lectures, for the purpose of e-Learning or self-learning are important. In this paper, we describe an adaptation method of a language model to recognize keywords in given slides. Next, we propose a summarization method for spoken classroom lectures using prosodic features and linguistic information. Last, we describe our “Classroom Lecture Browsing System”, which enables more efficient access to lecture video by using the results of keyword-indexing and automatic summarization.

We proposed a blind dereverberation method based on spectral subtraction by Multi-Channel Least Mean Square (MCLMS) al-gorithm for distant-talking speech recognition in our previous study [1]. In this paper, we discuss the problems of the pro-posed method and present some solutions. In a distant-talking environment, the length of channel impulse response is longer than the short-term spectral analysis window. By treating the late reverberation as additive noise, a noise reduction technique based on spectral subtraction was proposed to estimate power spectrum of the clean speech using power spectra of the dis-torted speech and the unknown impulse responses. To estimate the power spectra of the impulse responses, a Variable Step-Size Unconstrained MCLMS (VSS-UMCLMS) algorithm for iden-tifying the impulse responses in a time domain was extended to a frequency domain. To reduce the effect of the estimation error of channel impulse response, we normalize the early re-verberation by CMN instead of the spectral subtraction used by the estimated impulse response in this paper. Furthermore, our proposed method is combined with a conventional delay-and-sum beamforming. We conducted the experiments on distorted speech signal simulated by convolving multi-channel impulse responses with clean speech. The modified proposed method achieved a relative error reduction rate of 22.7 % from conven-tional CMN and 12.0 % from the original proposed method, re-spectively. By combining the modified proposed method with the beamforming, a furthermore improvement (relative error re-duction rate of 23.3%) was achieved. Index Terms: distant-talking speech recognition, blind dere-verberation, Multi-channel LMS, spectral subtraction, CMN. 1.

Voice activity detection (VAD) plays an important role in speech processing including speech recognition, speech enhancement, and speech coding in noisy environments. We developed an evaluation framework for VAD in such environments, called corpus and environment for noisy speech recognition 1 concatenated (CENSREC-1-C). This framework consists of noisy continuous digit utterances and evaluation tools for VAD results. By adoptiong two evaluation measures, one for frame-level detection performance and the other for utterance-level detection performance, we provide the evaluation results of a power-based VAD method as a baseline. When using VAD in speech recognizer, the detected speech segments are extended to avoid the loss of speech frames and the pause segments are then absorbed by a pause model. We investigate the balance of an explicit segmentation by VAD and an implicit segmentation by a pause model using an experimental simulation of segment extension and show that a small extension improves speech recognition.

To precisely model the time dependency of features is one of the important issues for speech recognition. Segmental unit input HMM with a dimensionality reduction method is widely used to address this issue. Linear discriminant analysis (LDA) and heteroscedastic discriminant analysis (HDA) are classical and popular approaches to reduce dimensionality. However, it is difficult to find one particular criterion suitable for any kind of data set in carrying out dimensionality reduction while preserving discriminative information. In this paper, we propose a new framework which we call power linear discriminant analysis (PLDA). PLDA can describe various criteria including LDA and HDA with one parameter. Experimental results show that the PLDA is more effective than PCA, LDA, and HDA for various data sets.

We proposed an environmentally robust speech recognition method based on position-dependent cepstral mean normalization (PD-CMN) to compensate for channel distortion depending on speaker position. PDCMN can efficiently compensate for the channel transmission characteristics while it cannot normalize speaker variation because position-dependent cepstral mean does not contain speaker characteristics. Conventional CMN can compensate for the speaker variation while it cannot obtain good recognition performance for short utterances. In this paper, we propose a robust distant speech recognition by combining position-dependent CMN with the conventional CMN to address the above problems. The position-dependent cepstral mean is linearly combined with conventional cepstral mean with following two types of processing. The first method is to use a fixed weighting coefficient over whole test data to obtain the combinational CMN, which is called fixed-weight combinational CMN. The second method is to calculate the output probability of multiple features compensated by a variable weighting coefficient at each frame, and a single decoder using these output probabilities is used to perform speech recognition, which is called variable-weight combinational CMN. We conducted the experiments of our proposed method using small vocabulary (100 words) distant isolated word recognition in a real environment. The proposed variable-weight combinational CMN method achieved a relative error reduction rate of 56.3% from conventional CMN and 22.2% from PDCMN, respectively.

We automatically extract the summaries of spoken class lectures. This paper presents a novel method for sentence extraction-based automatic speech summarization. We propose a technique that extracts “cue phrases for im-portant sentences (CPs) ” that often appear in important sen-tences. We formulate CP extraction as a labeling problem of word sequences and use Conditional Random Fields (CRF) [1] for labeling. Automatic summarization using CP extraction re-sults as features yields precisions of 0.603 and 0.556 when us-ing manual transcriptions and Automatic Speech Recognition (ASR) results, respectively. Combining the features derived from the CPs and tradi-tional features (including repeated words, words repeated in a slide text, and term frequency (tf), which are surface linguistic information, and speech power and duration, which are prosodic features) [2, 3], we obtained better summarization performance with a κ-value of 0.380, a F-measure of 0.539, and a Rouge-4 of 0.709. Index Terms: automatic speech summarization, sentence ex-traction, speech synthesis

In conventional speaker recognition method based on MFCC, the phase information has been ignored. In this paper, we proposed a method that integrates the phase in-formation on a speaker recognition method. The speaker identification experiments were performed using NTT database which consists of sentences uttered at normal speed mode by 35 Japanese speakers (22 males and 13 females) on five sessions over ten months. Each speaker uttered only 5 training utterances (about 20 seconds in to-tal). Using the phase information, the speaker recognition error rate was reduced by about 44%. Index Terms: speaker identification, MFCC, phase in-formation, GMM, combination method

In this paper, we propose a statistical method of evaluating the pronunciation proficiency for presentation in English. We sta-tistically analyze the utterances to find a combination that has a high correlation between an English teacher’s score and some acoustic features. We found that ratio of the likelihoods of free phoneme recognition by using native English HMMs and non-native English HMMs was the best measure of pronunciation proficiency. The combination of likelihood ratio between na-tive English HMMs and non-native English HMMs (concate-nation of correct phone HMMs), likelihood ratio between na-

If a dialog system were to respond to a user as naturally as a human, interaction would be smoother. Imitating the human prosodic behavior of utterances is important in computer-human natural conversations. In this paper, to develop a coopera-tive/friendly spoken dialog system, we analyzed the correlations between F0 synchrony tendency or overlap frequency and sub-jective measures: “liveliness, ” “familiarity, ” and “informality” in human-human dialogs. We also modeled the properties of these features and implemented the model on our dialog system that generated the response timing of aizuchi (back-channel), turn-taking based on a decision tree in real time, and dynamical F0 changes to realize chat-like conversations. Index Terms: spoken dialog system, prosody control, response timing

This paper proposes a novel method to construct a spoken language model including fillers from a corpus including no fillers using a filler prediction model. It consists of two submodels: a filler insertion model which predicts places where fillers should be inserted, and a filler selection model which predicts appropriate fillers for given places. It converts a corpus that covers domain-relevant topics but includes no fillers into a corpus that contains fillers as well as domain-relevant topics. The experiment against the corpus of spontaneous Japanese shows that language models constructed by the proposed method achieve quite near performance of the traditional trigram language model constructed from the real spontaneous corpus including fillers.

To precisely model the time dependency of features, segmental unit input HMM with a dimensionality reduction method has been widely used for speech recognition. Linear discriminant analysis (LDA) and heteroscedastic discriminant analysis (HDA) are popular approaches to reduce the dimen-sionality. We have proposed another dimensionality reduction method called power linear discriminant analysis (PLDA) to se-lect the best dimensionality reduction method that yields the highest recognition performance. This selection process on the basis of trial and error requires much time to train HMMs and to test the recognition performance for each dimensionality re-duction method. In this paper we propose a performance comparison method without training or testing. We show that the proposed method using the Chernoff bound can rapidly and accurately evaluate the relative recognition performance. Index Terms: speech recognitoin, feature extraction, multidi-mensional signal processing

To achieve high recognition performance for a wide variety of noise and for a wide range of signal-to-noise ratio, this paper presents integration methods of four noise reduction algorithms: spectral subtraction with smoothing of time direction, temporal domain SVD-based speech enhancement, GMM-based speech estimation and KLT-based comb-filtering. In this paper, we proposed two types of combination methods of noise suppression algorithms: selection of front-end processor and combination of results from multiple recognition processes. Recognition results on the AURORA-2J task showed the effectiveness of our proposed methods. Intex Terms: Noisy speech recongition, noise suppression method selection, AURORA-2J

We proposed a novel dialog strategy which can recover from mis- recognition through a spoken dialog. To recover from the misrecog- nition without confirmation, our system kept multiple understanding hypotheses at each turn and 'searched' for a globally optimal hypothesis across user's utterances in a whole dialog. As for a dialog strategy, we introduced a new criterion based on 'efficiency for convergence' and 'consistency with understanding hypotheses' to select an appropriate system response. Using such criterion, the system removes the ambiguity without making the user feel unnatural in relation to the response conflicting with actual user intent. We also proposed to adopt the repetition utterance detection to update the understanding hypotheses. We developed a spoken dialog system using these techniques and showed some dialog examples in which misrecognition was naturally corrected. We also showed that our strategy was efficient in terms of the number of turns.

We aim to extract automatically the summarization of spoken lectures for conferences and classes. For this purpose, at first we compared results of summarization extracted by human subjects. We found large differences with every subject. Then we investigated the relations between linguistic surface information and human results, and we obtained useful linguistic surface information. Next, we summarized spoken lectures on conferences and classes using the linguistic information. Additionally, we also focused on prosodic features; F0 and power. We conducted the same experiments on them. Lastly, we combined linguistic surface information and prosodic information. As a result, the proposed automatic summarization produced a better F- measure (0.599), k-value (0.420) and Rouge metric (0.758) comparable with human results.

We investigate a multimodal interface for organization name input to forms in WWW. The user first utters an organization name in an open vocabulary domain to the system. The system recognizes it with a combination method of isolated word recognition and continuous “base-word ” recognition. Word candidates and a base-word lattice obtained by this recognition procedure are displayed on a touch panel. Then the user chooses a sequence or base-words from the candidates by pen touch to construct the desired name and the name is sent to the client. This recognition method performs well in the organization name recognition task, in which a very large vocabulary size and its successive update is needed when using isolated word recognition. Our interface design also elevates users ’ input ability of organization names. 1.

In a distant environment, channel distortion may drastically degrade speaker recognition performance. In this paper, we propose a robust speaker recognition method based on posi-tion dependent Cepstral Mean Normalization (CMN) to com-pensate the channel distortion depending on the speaker posi-tion. It is shown in [1] that the position dependent CMN is robust for speech recognition in a distant environment. We ex-tend this method to the speaker recognition and show that this method is much effective to speaker recognition. In the train-ing stage, the system measures the transmission characteristics according to the speaker positions from some grid points to the microphone in the room and estimated the compensation pa-rameters a priori. In the recognition stage, the system esti-mates the speaker position and adopts the estimated compen-sation parameters corresponding to the estimated position, and then the system applies the CMN to the speech and performs speaker recognition. In our past study, we proposed a new text-independent speaker recognition method by combining speaker-specific Gaussian Mixture Models (GMMs) with syllable-based HMMs adapted to the speakers by MAP [2]. The robustness of this speaker recognition method for the change of the speaking style in close-talking environment was evaluated in [2]. We in-tegrated this method to the proposed position dependent CMN for distant speaker recognition. Our experiments showed that the proposed method improved the speaker recognition perfor-mance remarkably in a distant environment. 1.

In a distant environment, channel distortion may drastically degrade speech recognition performances. In this paper, we propose a robust multiple microphone speech processing approach based on position dependent Cepstral Mean Normalization (CMN). In the training stage, the system measures the transmission characteristics according to the speaker positions from some grid points in the room and estimated the compensation parameters a priori. In the recognition stage, the system estimates the speaker position and adopts the estimated compensation parameters corresponding to the estimated position, and then the system applies the CMN to the speech and performs speech recognition for each microphone. Finally, the maximum vote or the maximum summation likelihood of whole channels (that is, multiple microphones) is used to obtain the final result. In our proposed method, we use utterances emitted from a loudspeaker located at various positions to estimate compensation parameters for a convenient sake, and we also compensate the mismatch between the cepstral means of utterances spoken by human and those emitted from the loudspeaker. Our experiments showed that the proposed method improved the performances of speech recognition system in a distant environment efficiently and it could also compensate the mismatch between voices from human and loudspeaker well.

We presented a new text-independent speaker recognition method by combining a speaker-specific Gaussian mixture model (GMM) with a syllable-based HMM adapted by MLLR or MAP (S. Nakagawa et al., Proc. Eurospeech, p.3017-3020, 2003). The robustness of this speaker recognition method for speaking style changes was evaluated in this paper. A speaker identification experiment, using an NTT database, which consists of sentences of data uttered at three speed modes (normal, fast and slow) by 35 Japanese speakers (22 males and 13 females) on five sessions over ten months, was conducted. Each speaker uttered only 5 training utterances (about 20 seconds in total). We obtained an accuracy of 98.8% for text-independent speaker identification for three speaking style modes (normal, fast, slow) by using a short test utterance (about 4 seconds). This result was superior to conventional methods for the same database. We show that the attractive result was brought from the compensational effect between speaker specific GMM and speaker adapted syllable based HMM.

This paper presents speech-driven Web retrieval models which accepts spoken search topics (queries) in the NTCIR-3 Web retrieval task. We experimentally evaluate the techniques of combining outputs of multiple LVCSR models with a language model(LM) with a 60,000 vocabulary size in recognition of spoken queries. As model combination techniques, we use the SVM learning. We show that the techniques of multiple LVCSR model combination can achieve improvement both in speech recognition and retrieval accuracies in speech-driven text retrieval. Comparing with the retrieval accuracies when a LM with a 20,000/60,000 vocabulary size is used in LVCSRs, the LM that has larger size of the vocabulary improves also retrieval accuracies.

For many practical applications of speech recognition systems, it is quite desirable to have an estimate of confidence for each hypothesized word. Unlike previous works on confidence measures, we have proposed features for confidence measures that are extracted from outputs of more than one LVCSR models. For further analysis of the proposed confidence measure, this paper examines the correlation between each word's confidence and the word's features such as its part-of-speech and syllable length. We then apply SVM learning technique to the task of combining outputs of multiple LVCSR models, where, as features of SVM learning, information such as the pairs of the models which output the hypothesized word are useful for improving the word recognition rate. Experimental results show that the combination results achieve a relative word error reduction of up to 72 % against the best performing single model and that of up to 36 % against ROVER.

Recently, the performance of speech recognition was drastically improved, and the products with the interface based on speech recognition have been realized. However, when we communicate with computers through a speech interface, misrecognition is inevitable, and it is difficult to recover from it because of the immaturity of the interface. Users try to recover from misrecognition by a repetition of the same content. So, the detection of user’s repetition is helpful for a system to detect its misunderstanding, and to recover from the misrecognition. In this paper, we assume the utterance which includes repetitions a correction and propose a method to detect correction utterances in spontaneously spoken dialog using a word spotting based on DTW (dynamic time warping) and N -best hypotheses overlapping measure. As a result, we achieved recall rate of 92.7% and precision of 89.1%. Moreover, we tried to improve recognition accuracy using the detection. Using the choice of vocabulary and grammar setup based on the detection, we achieved improvement in recognition performance from 42.7% to 50.0% for correction utterance and from 70.5% to 77.9% for non-correction utterance.

This paper studies speech-driven Web retrieval models which accepts spoken search topics (queries) in the NTCIR-3 Web retrieval task. The major focus of this paper is on improving speech recognition accuracy of spoken queries and then improving retrieval accuracy in speech-driven Web retrieval. We experimentally evaluate the techniques of combining outputs of multiple LVCSR models in recognition of spoken queries. As model combination techniques, we compare the SVM learning technique and conventional voting schemes such as ROVER. We show that the techniques of multiple LVCSR model combination can achieve improvement both in speech recognition and retrieval accuracies in speech-driven text retrieval. We also show that model combination by SVM learning outperforms conventional voting schemes both in speech recognition and retrieval accuracies.

For many practical applications of speech recognition systems, it is quite desirable to have an estimate of confidence for each hypothesized word. Unlike previous works on confidence measures, this paper studies features for confidence measures that are extracted from outputs of more than one LVCSR models. More specifically, this paper experimentally evaluates the agreement among the outputs of multiple Japanese LVCSR models, with respect to whether it is effective as an estimate of confidence for each hypothesized word. The results of experimental evaluation show that the agreement between the outputs with two LVCSR models with different decoders and acoustic models can achieve quite reliable confidence. Furthermore, among various features of acoustic models based on Gaussian mixture HMMs, it is concluded that ones such as whether or not to have short pause models, as well as different units in HMMs (e.g., triphone model or syllable model) are the most effective in achieving highly reliable confidence.

We present the maximum normalized likelihood estimation (MNLE) algorithm and its application for discriminative training of hidden Markov models (HMMs) for continuous speech recognition. The objective of this algorithm is to maximize the normalized frame likelihood of training data. Instead of gradient descent techniques usually applied for objective function optimization in other discriminative algorithms such as the minimum classification error (MCE) and maximum mutual information (MMI), we used a modified expectation-maximization (EM) algorithm which greatly simplifies and speeds up the training procedure. Evaluation experiments showed better recognition rates compared, to both the maximum likelihood (ML) training method and MCE/GPD discriminative method. In addition, the MNLE algorithm showed better generalization abilities and was faster than MCE/GPD.

Addresses the problem of the detection of speaker changes and clustering speakers when no information is available regarding speaker classes or even the total number of classes. We assume that no previous information on speakers is available (no speaker model, no training phase) and that people do not speak simultaneously. The aim is to apply speaker grouping information to speaker adaptation for speech recognition. We use vector quantization (VQ) distortion as the criterion. A speaker model is created from successive utterances as a codebook by a VQ algorithm, and the VQ distortion is calculated between the model and an utterance. A result was obtained by the experiment on speaker detection and speaker clustering. The speaker change detection experiment was compared with results by generalized likelihood ratio and Bayesian information criterion. We show the superiority of our proposed method.

For many practical applications of speech recognition systems, it is quite desirable to have an estimate of confidence for each hypothesized word. Unlike previous works on confidence measures, this paper studies features for confidence measures that are extracted from outputs of more than one LVCSR models. More specifically, this paper experimentally evaluates the agreement among the outputs of multiple Japanese LVCSR models, with respect to whether it is effective as an estimate of confidence for each hypothesized word. The results of experimental evaluation show that the agreement between the outputs with two acoustic models which have different units in HMMs, such as phonemes and syllables, can achieve quite reliable confidence. 1.

More and more eorts have been recently made to apply speech technologies to language learning[1]{ [4].The authors have been especially focusing on Japanese manners of generating English word stress.This is because accentuation habits inevitable to Japanese learners can be easily found in their stress generation.In our previous studies, a stressed syllable detector and an accentuation habit estimator were developed[5]{ [7], where the estimated habits of individual learners accorded well with their English pronunciation prociency rated by English teachers.However, the estimation methods in our previous studies required several dozens of word utterances or a relatively large amount of computation even when a single word utterance enabled the estimation.In this paper, we i n v estigated a method which required only a single word utterance with a small computation cost.Results showed that similar tendencies can be found between the habits estimated in our previous study and those in the current one.

We have been focusing on applying speech technologies to pronunciation learning. In our previous study[1], a stressed syllable detector was implemented by using stressed syllable HMMs and unstressed ones. And using the detector internally, several systems were implemented[2]. However, their development did not necessarily require the use of HMMs as an acoustic modeling method. In this paper, an HMM-based method, a DTW-based method, and a human strategy only with visual inspection were compared in terms of their performance in judging whether two utterances of a word have the same stress pattern, e.g. récord and recórd. Here, one utterance was given by a Japanese learner and the other one was done by a native speaker. Experiments showed that HMMs gave us the higher performance

This paper discusses two issues of the quality improvement of F0 modified speech based upon PSOLA analysissynthesis. Previous studies[1][2] pointed out that the location of a window of PSOLA influences the quality of synthesized speech and one of them claimed that the center of a window should be located at a pitch pulse in source waveforms. However, pitch pulse detection sometimes fails due to undesired acoustic events. In this paper, several methods are experimentally examined to reduce pitch pulse detection errors. Even when the detection is done correctly, F0 modified re-synthesized speech sometimes causes “echoes ” in the re-arranged waveforms. This is mainly caused by a pitch pulse with small sharpness or by that with two relatively high pulses, not pitch pulses, before and after it. To suppress the echoes with little loss of naturalness, partial zero/π-phase conversion is proposed here. Experiments show the high validity of the proposed methods in improving the quality of re-synthesized speech. 1.

More and more efforts have been recently made to apply speech technologies to language learning and develop CALL systems[1]–[4]. The authors have been focusing on Japanese manners of generating English word stress. This is because pronunciation habits which are inevitable to Japanese learners can be easily found in the stress generation. In our previous study, a stressed syllable detector and a pronunciation habit estimator were developed[5], where the estimated habits of individual learners accorded well with their English pronunciation proficiency rated by English teachers. However, the habit could be estimated only after a learner pronounced several dozens of words because the habit estimator referred to stressed syllable detection rates. In this paper, using another criterion, a method of instantaneous estimation was proposed which required only a single word utterance. Results showed that an average pattern of the instantaneously estimated habits accorded well with the habits obtained in our previous study.

In this paper, we describe a semantic interpreter and a cooperative response generator for a multimodal dialogue system which consists of speech input, touch screen input, speech output and graphical output. The dialogue system understands spontaneous speech which has many ambiguous phenomena such as interjections, ellipses, inversions, repairs, unknown words and so on, and responses to user for the utterance. But some utterances fail to be analyzed. This is caused due to "misrecognition" with the speech recognizer, "incompleteness" with the semantic interpreter and "lack of database" with the response generator. Therefore, we improved the semantic interpreter to build a more robust one. If a user's query does not have enough conditions/information to answer the question with the system, the dialogue manager should query the user to get the necessary conditions or to select the candidate. Further, if the system cannot retrieve any information related to the user's question, the generator should propose an alternative plan. Based on these considerations, we developed a cooperative response generator in the dialogue system. We report the evaluation results of the semantic interpreter and cooperative response generator in our dialogue system.

In this paper, the correlation between spectral variations and F0 changes in a vowel sound is rstly analyzed, where the variations are also compared to VQ distortions calculated in a ve-vowel space. It is shown that the F0 change approximately by a half octave produces the spectral variation comparable to the averaged VQ distortion when the codebook size is the number of the vowels. Next, a model to predict the cepstral coe cients' variations caused by the F0 changes is built based on the multivariate regression analysis. Experiments show that the generated frame by the model has a remarkably small distance to the target frame and that the distance is almost the same as the VQ distortion with the codebook size being 10 to 20. Furthermore, the model is evaluated separately in terms of a spectral envelope predictor with a given F0 and a mapping function of feature sub-spaces. It is indicated that, while the models should be built dependently on phonemes and speakers as a spectrum predictor, adequate selection of parameters can enable the speaker/phoneme-independent models to work e ectively as a mapping function.

We propose a new method, where the likelihood normalization technique is applied at both the frame and utterance levels. In this method based on Gaussian mixture models (GMM), every frame of the test utterance is inputed to the claimed and all background speaker models in parallel. In this procedure, for each frame, likelihoods from all the background models are available, hence they can be used for normalization of the claimed speaker likelihood at every frame. A special kind of likelihood normalization, called weighting models rank, is also proposed. We have evaluated our method using two databases-TIMIT and NTT. Results show that the combination of frame and utterance level likelihood normalization in some cases reduces the equal error rate (EER) more than twice.

The standard HMM cannot fully express the time variant features while staying at the same state. So as not to ignore the dynamic changes of the speech characteristics, various methods have been studied. In this paper, we compare a segmental unit input HMM where several successive frames are combined and become an input vector, with conditional density HMM or the use of regression coefficients and evaluate them. Using segmental statistics, since the dimension of the parameters increases, results in a lesser precision in estimation of the covariance matrix. Therefore we used methods for compressing dimension and reducing computation by K-L expansion and MQDF. By segmental unit inputting for the basic structure HMM, we got a better recognition rate than by traditional methods and the combination of a segmental unit of successive mel-cepstrum frames and regression coefficients showed the best recognition rate.

In this paper we propose a new speaker identication system, where the likelihood normalization technique, widely used for speaker verication, is introduced.In the new system, which is based on Gaussian Mixture Models, every frame of the test utterance is inputed to all the reference models in parallel.In this procedure, for each frame, likelihoods from all the models are available, hence they can be normalized at every frame.A special kind of likelihood normalization, called Weighting Models Rank, is also proposed.Experiments were performed using two databases -TIMIT and NTT.Evaluation results clearly show that frame level likelihood normalization technique is superior to the standard accumulated likelihood approach.

A new scheme is proposed to incorporate prosodic processing into speech recognition, where the accent n uclei at the head of words are detected automatically and used to limit the searching space in speech recognition, that is, to preselect candidate words.Especially in this paper, the proposed method for the automatic detection of the accent n uclei and its performance are described.Using this scheme, it is expected that the recognition speed is improved.This scheme is derived from a nding by perceptual experiments conducted previously by the rst author.Results of the experiments indicated that the accent n ucleus at the rst mora has acceleration eect on perceiving the word.This effect can be explained by the earlier identication of the word accent t ype as type 1 b y its nucleus at the rst mora.In other words, the accent n ucleus at the head of a word can limit the searching space eectively in the mental lexicon.This mechanism was implemented using HMMs and examined for isolated words on a machine, where the vowel detection by broad segmental features and the rejection of words with a devoiced vowel at the rst or second mora were introduced at the same time.Evaluation experiments showed 94.7% and 90.0% as recall factor and precision factor of the accent n ucleus detection respectively.

An evaluation technique is very important for developing a successful continuous speech recognition system. The branching factor and the perplexity have been used to measure the complexity of speech recognition task. The authors describe their evaluation method, which is based on such a measure. They found the relationship among perplexity (V/sub p/) on word-unit (or phoneme-unit), sentence length (L), word (or phoneme) recognition rate (R/sub w/), and sentence recognition rate. From this relationship, they can predict the sentence recognition rate, if the word (or phoneme) recognition performance and task definition are given. The approximate equation is as follows: sentence recognition rate=(f(V/sub p/, R/sub w/))/sup L/, where f(V/sub p/,R/sub w/) denotes the word recognition rate for the vocabulary size V/sub p/ obtained by using this recognizer (R/sub w/) and this is estimated from the relationship between the number of categories and recognition rate.>

A continuous-parameter hidden Markov model (HMM) is proposed and compared with a discrete-parameter HMM, a time delay neural network (TDNN), and LVQ2 by using the same training and testing database. It is found that the proposed model's performance is comparable to that of TDNN or LVQ2. Higher performance (96 approximately 97%) is obtained for all Japanese phonemes in isolated words. The HMM approach is superior to others for the recognition of time-sequential patterns like continuous speech.>

A statistical method of segmentation using a hidden Markov model (HMM) and a Bayesian classifier is described. The main features of this method are the use of feature parameters which are independent of each category in vowels of consonants, and the use of only one HMM which commonly represents all syllable patterns. The segmentation strategy is to find the optimal HMM sequence. The optimal/best sequence is found by using the O(n) DP matching based on Viterbi algorithm. The concatenated number and boundaries of the best HMM sequence are regarded as the segmentation result. The experimental result on Japanese spoken sentences shows that the rate of segmentation is more than 92% for two male speakers, and the rate is improved to 97.5% by using a duration control mechanism based on a discrete probability distribution.>

This paper proposes a new continuous speech recognition method by phoneme-based word spotting and time-synchronous context-free parsing. The word pattern is composed of the concatenation of phoneme patterns. The knowledge of syntax is given in Backus Normal Form. Therefore, our method is task-independent in terms of reference patterns and task language. The system first spots word candidates in an input sentence, and then generates a word lattice. The word spotting is performed by a dynamic time warping method. Secondly, it selects the best word sequences found in the word lattice from all possible sentences which are defined by a context-free grammar.

This paper describes 2 efficiency enhancements for speaker-independent connected spoken word spotting. The first enhancement, LESS COST, uses phoneme concatenation to reduce the cost of computing the local distance between each reference and input pattern point. The second enhancement, Coarse DP refinement, reduces the cost of dynamic time warping for only a small error rate penalty. An experiment confirmed these techniques.

In this paper, we present an approach to isolated and connected word recognition by using dynamic time warping algorithm which referes as a hidden Markov model. The classification consists of computing the a posteriori probability for each word model and choosing the word model that gives the highest probability. The probability is calculated by two different ways: One is the exact algorithm and the other is the approximate (Viterbi) algorithm. In our system, first, an input speech is recognized as a string of monosyllables by the syllable-based O(n) DP matching. Second, the recognized string is matched with a mono-syllable string of each lexical model, and the word or word sequence with the highest probability is recognized as the input speech by using O(n) DP matching based on a hidden Markov model. Reference patterns consist of 68 mono-syllables, and test patterns consists of 90 isolated words, two connected words and three connected words. We conclude from the results of the experiments that: (1) The results by using 3 candidates are much better than those by using only best candidate for each segment. (2) The approximate algorithm has almost the same performance as the exact algorithm. (3) The extended algorithm for connected word recognition works well.

An O(n) dynamic programming (time warping) matching algorithm is proposed for connected spoken word recognition. The algorithm is based on the same principle of Two Level DP, Level Building DP, and Clock-Wise DP matching methods. The O(n) DP method is computationally more efficient than the conventional methods and also does not require many memory storages. This is able to recognize connected spoken words by one level (one pass) DP matching for every reference patterns. The amount of computation is the same as that of isolated spoken word recognition by DP matching. Of course, this algorithm gives the same results as Two Level DP, Level Building DP and Clock-Wise DP methods. Further, the flexibility of this algorithm is increased by embedding the unconstrained endpoints DP matching on both reference patterns and test pattern, and weighted DP matching. We show from the experiment on connected spoken digit recognition that the augmented algorithm improves the results.

In this paper, we describe an optimum matching method between a classified phoneme string and a phoneme string of a lexical entry in a word dictionary. This method is performed by using a phoneme similarity matrix and a dynamic programming technique. A classified segment consists of the first candidate, second candidate, reliability and duration. The effect of coarticulation is normalized in this matching procedure. We also describe a word spotting method in continuous speech by modifying this method.