Several speech emotion recognition systems have been proposed, mostly targeting a single language; however, the recognition performance is significantly degraded for languages other than the target language. To solve this problem, a method of normalizing features using synthesized speech was proposed by Okabe et al.; however, this method requires a search for the optimal normalization operation. In this paper, we propose a multilingual emotion recognition method that automatically normalizes acoustic features and recognizes emotions using the “recognition rate maximization” criterion. Synthesized speech with the same content as emotional speech was generated and input to a convolutional recurrent neural network with time axis aligned. Feature normalization by comparison with synthesized speech and emotion recognition were performed in a single network, and efficient training was expected. In speech emotion recognition experiments with four languages, normalization improved the recognition accuracy from 47.5% to 49.0%. Because normalization was not effective for utterances in which the speaking rate of the synthesized and emotional speeches differed significantly, we proposed a method to adjust the speaking rate of the synthesized speech and further improved the accuracy by up to 50.4%.

Spoken dialog system is one of the most useful interface between human and machine. These systems need scenario data, which is represented by Finite State Transducer (FST). It is usually written by hand, but making it is a time-consuming job. In order to make a dialog scenario file easily, a new dialog scenario editor is developed. In dialog systems, it is important for natural communication that an agent speaks with natural gesture. However, it is difficult to make an appropriate motion for every utterance in a dialog scenario. In this paper, automatic motion assignment method is proposed. Most of motions have some meanings, and these meanings correspond to some keywords. Relationship between motions and keywords are investigated in advance, and an appropriate motion is selected based on corresponding keyword list. Moreover, the proposed motion selection method also considers sentence styles (interrogative, negative, and so on) of the main phrase in the sentence. Naturalness evaluation experiment was carried out for combinations between motions and utterances, and motion selection rules (keyword list and conditions) were made based on the results. Totally 83 keywords were listed for 31 motions. The performance of the motion selection method was evaluated. From this experiment, only 13.3% of input sentences were given an appropriate motion. However, many errors could be corrected by modifying the motion selection rules, and finally 80.0% of assigned motions were evaluated as “appropriate.” The dialog scenario editor with automatic motion assignment was developed. It deals with FST-based dialog system. In particular, it outputs a dialog scenario file for MMDAgent. This editor operated correctly, and a scenario file could be created easily.

Emotion recognition from speech signals is one of the most important technologies for natural conversation between humans and robots. Most emotion recognizers extract prosodic features from an input speech in order to use emotion recognition. However, prosodic features changes drastically depending on the uttered text.

We previously proposed a new segment quantization method called Lempel-Ziv segment quantization (LZSQ), which is a modified version of Lempel-Ziv (LZ) coding that can be applied to a continuous information source. In the present paper, LZSQ is applied to the mixed excitation linear prediction (MELP) coder, which is a standardized vocoder-type speech coder that operates at 2.4 kbit/s, in order to reduce the bit rate to below 2.4 kbit/s, while preserving the quality of the coded speech. LZSQ is applied to six coding parameters of the MELP coder to reduce the total bit rate as much as possible. As a result, the total bit rate of the modified MELP coder was reduced to 1.57 kbit/s, while the subjective quality of the modified MELP coder is equivalent to that of the previous MELP coder.

We propose an aspect-model-based reference speaker weighting. The main idea of the approach is that the adapted model is a linear combination of a set of reference speakers like reference speaker weighting (RSW) and eigenvoices. The aspect model is the mixture model of speaker-dependent (SD) models. In this paper, aspect model weighting (AMW) is proposed for finding an optimal weighting of a set of reference speakers unlike RSW and the aspect model which is a kind of cluster models is trained based on likelihood maximization with respect to the training data. The number of adaptation parameters can also be reduced using aspect model approach. For evaluation, we carried out an isolated word recognition experiment on Korean database (KLE452). The results were compared to those of conventional MAP, MLLR, RSW, and eigenvoice. Even though we use only 0.5s of adaptation data, 27.24% relative error rate reduction in comparison with speaker-independent (SI) baseline performance was achieved.

A new segment quantization method using the Lempel-Ziv algorithm is proposed, and it is applied to quantize line spectral frequency parameters in speech codec. The proposed segment quantizer can save four bits per frame, compared with the ITU-T G.729 speech codec (18 bits/frame), without degradation of subjective or objective speech quality

This paper describes a music information retrieval system that uses humming as the key for retrieval. Humming is an easy way for a user to input a melody. However, there are several problems with humming that degrade the retrieval of information. One problem is the human factor. Sometimes, people do not sing accurately, especially if they are inexperienced or unaccompanied. Another problem arises from signal processing. Therefore, a music information retrieval method should be sufficiently robust to surmount various humming errors and signal processing problems. A retrieval system has to extract the pitch from the user's humming. However, pitch extraction is not perfect. It often captures half or double pitches, which are harmonic frequencies of the true pitch, even if the extraction algorithms take the continuity of the pitch into account. Considering these problems, we propose a system that takes multiple pitch candidates into account. In addition to the frequencies of the pitch candidates, the confidence measures obtained from their powers are taken into consideration as well. We also propose the use of an algorithm with three dimensions that is an extension of the conventional Dynamic Programming (DP)algorithm, so that multiple pitch candidates can be treated. Moreover, in the proposed algorithm, DP paths are changed dynamically to take deltaPitches and IOIratios (inter-onset-interval) of input and reference notes into account in order to treat notes being split or unified. We carried out an evaluation experiment to compare the proposed system with a conventional system . When using three-pitch candidates with conference measure and IOI features, the top-ten retrieval accuracy was 94.1%. Thus, the proposed method gave a better retrieval performance than the conventional system.

This paper describes a method to detect smiles and laughter sounds from the video of natural dialogue. A smile is the most common facial expression observed in a dialogue. Detecting a user's smiles and laughter sounds can be useful for estimating the mental state of the user of a spoken-dialogue-based user interface. In addition, detecting laughter sound can be utilized to prevent the speech recognizer from wrongly recognizing the laughter sound as meaningful words. In this paper, a method to detect smile expression and laughter sound robustly by combining an image-based facial expression recognition method and an audio-based laughter sound recognition method. The image-based method uses a feature vector based on feature point detection from face images. The method could detect smile faces by more than 80% recall and precision rate. A method to combine a GMM-based laughter sound recognizer and the image-based method could improve the accuracy of detection of laughter sounds compared with methods that use image or sound only. As a result, more than 70% recall and precision rate of laughter sound detection was obtained from the natural conversation videos

This paper presents a new segment quantization method, which is based upon the Lempel-Ziv algorithm, and it is applied to quantize LSF segments of speech signals. The proposed quantizer is compared with other practical coders such as the G.729 and the MELP, by substituting those original LSF quantizers with the proposed segment quantizer. As a result, at the same cepstral distortion, nine bits per frame were saved compared with the scalar quantizer of the MELP (25 bits/frame), and three bits per frame were saved compared with the MA prediction VQ of the G.729 (18 bits/frame), respectively.

Speech recognition by a small robot is difficult because the robot makes noise itself. In this paper, two new methods are proposed that suppresses internal noise of the small robots. These methods are based on spectral subtraction (SS). The difference of the proposed methods from the original SS is that the proposed methods use the estimated noise spectrum dependent on the motion of the robot. One method, called MDSS, prepares the noise spectrums for all motions. Another method, called NPSS, predicts the noise spectrum from angular velocities of all joints of the robot using a neural network. From the results of the comparison between the original SS and the proposed methods, the proposed methods outperformed the conventional SS. The NPSS worked well even when the noise of the motion was unstable, while the MDSS method gave good result when the noise in one motion was stable.

Speech recognition under noisy environment is one of the hottest topic in the speech recognition research. In this paper, we propose a method to improve accuracy of spoken dialog system from a dialog strategy point of view. In the proposed method, the dialog system automatically changes its dialog strategy according to the estimated recognition accuracy in noisy environment in order to keep the performance of the system constant. In a noisy environment, the system restricts its grammar and vocabulary to improve recognition accuracy. On the other hand, the system accepts any utterance from a user in a noise-free environment. To realize this strategy, we investigated a method to avoid user’s out-of-grammar utterances through an instruction given by the system to a user. Furthermore, we developed a method to estimate recognition accuracy from features extracted from noise signal. Finally, we constructed a proposed dialog system and confirmed its effectiveness.

We have been developing a spoken dialog system. Conventional spoken dialog systems need grammar descriptions and scripts of a dialog, that are difficult to develop. The system proposed in this paper is based on semantic frames, and the system generates the recognition grammar from the frames automatically. As the system requires only a frame-based description for a task of dialog, the system can be easily applied to different kinds of tasks. Moreover, the recognition accuracy is improved by sentence weighting based on phrase class template. We evaluated the system by experiments. The system reached the goal with 2.44 user’s utterances in average.

This paper describes a dialogue-based CALL (Computer Assisted Language Learning) system. One of the major problems in CALL systems is that learners are usually assigned a passive role. Learners have no practices in composing their own utterances. The other major problem is that lots of conventional CALL systems are pronunciation exercise systems. However, pronunciation exercise is only a part of exercise needed to increase a learner’s communication skill. In this paper, we propose a dialoguebased CALL system of new concept that enables exercise of composition, grammar and conversation in addition to pronunciation.

Several CALL systems have two acoustic models to evaluate a learner’s pronunciation. In order to achieve high performance for evaluation, speaker adaptation method is introduced in CALL system. It requires adaptation data of a target language, however, a learner cannot pronounce correctly. In this paper, we proposed two types of new speaker adaptation methods for CALL system. The new methods only require learner’s utterance of the native language for adaptation. The first method is an algorithm to adapt acoustic models using bilingual’s utterances. The speakerindependent acoustic models of native and target languages are adapted to a bilingual speaker once, then they are adapted to the learner again using the learner’s speech of the native language. Phoneme recognition accuracy is about 5% higher than the baseline method. The second method is a training algorithm of an acoustic model. It can robustly train bilinguals’ model from a few bilinguals’ utterances. Phoneme recognition accuracy is about 10% higher than the baseline method.

Statistic language models obtained from a large number of training samples play an important role in speech recognition. In order to obtain higher recognition performance, we should introduce long distance correlations between words. However, traditional statistic language models such as word n-grams and ergodic HMMs are insufficient for expressing long distance correlations between words. We propose an acquisition method for a language model based on HMnet taking into consideration long distance correlations and word location.

The tree-structured speaker clustering was proposed as a highspeed speaker adaptation method. It can select the model which is most similar to a target speaker. However, this method does not consider speaker difference dependent on phoneme class. In this paper, we propose a speaker adaptation method based on speaker clustering by taking speaker difference dependent on phoneme class into account. The experimental results showed that the new method gave a better performance than the original method. Furthermore, we propose the improved method which use a tree-structure of a similar phoneme as the substitute for the phoneme which does not appear in the adaptation data. From the experimental results, the improved method gave a better performance than the method previously proposed.