Speech-to-Speech Translation (S2ST) refers to the conversion of speech in one language into semantically equivalent speech in another language, facilitating communication between speakers of different languages. Speech-to-Discrete Unit Translation (S2UT), a mainstream approach for end-to-end S2ST, addresses challenges such as error propagation across modules and slow inference speed often encountered in traditional cascade systems. However, as discrete units primarily capture content information, conventional S2UT methods fail to retain speaker-specific characteristics from the source. Our previous work, Speaker Consistent S2UT (SC-S2UT), introduced a speaker adapter and a unit-to-mel structure, enabling the preservation of speaker information and non-autoregressive speech generation. Based on this foundation, this study proposes a self-supervised pre-training method to enrich the information extracted by both the speaker adapter and the unit-to-mel structure. Additionally, we investigate different feature fusion strategies to further improve the integration of speaker and content features. Experiments conducted in the CVSS-T dataset for ES-EN, FR-EN and DE-EN tasks demonstrate that our proposed method achieves a BLEU score improvement of 1.14 compared to SC-S2UT, along with significant improvements in UTMOS and speaker similarity. Furthermore, our approach achieves translation quality comparable to traditional S2UT, with a minimal increase of 0.04s per utterance in inference time, while maintaining high speaker similarity. These results validate the effectiveness of the proposed method. • Developed a novel approach to preserve speaker identity in end-to-end S2UT systems. • Leveraged self-supervised pre-training to enhance translation quality and performance. • Investigated feature fusion strategies to better integrate speaker and content features. • Demonstrated effective speaker preservation with no significant inference time increase.

There are high expectations for multimodal dialog systems that can make natural small talk with facial expressions, gestures, and gaze actions as next-generation dialog-based systems. Two important roles of the chat-talk system are keeping the user engaged and establishing rapport. Many studies have conducted user evaluations of such systems, some of which reported that considering the relationship with the user is an effective way to improve the subjective evaluation. To facilitate research of such dialog systems, we are currently constructing a large-scale multimodal dialog corpus focusing on the relationship between speakers. In this paper, we describe the data collection and annotation process, and analysis of the corpus collected in the early stage of the project. This corpus contains 19,303 utterances (10 hours) from 19 pairs of participants. A dialog act tag is annotated to each utterance by two annotators. We compare the frequency and the transition probability of the tags between different closeness levels to help construct a dialog system for establishing a relationship with the user.

This paper explores the effect of emotional speech synthesis on a spoken dialogue system when the dialogue is non-task-oriented.Although the use of emotional speech responses has been shown to be effective in a limited domain, e.g., scenario-based and counseling dialogue, the effect is still not clear in the non-task-oriented dialogue such as voice chat.For this purpose, we constructed a simple dialogue system with example-and rule-based dialogue management.In the system, two types of emotion labeling with emotion estimation are adopted, i.e., system-driven and user-cooperative emotion labeling.We conducted a dialogue experiment where subjects evaluate the subjective quality of the system and the dialogue from multiple aspects such as richness of the dialogue and impression of the agent.We then analyze and discuss the results and show the advantage of using appropriate emotions for expressive speech responses in the non-task-oriented system.

This paper examines a method to improve the user impression of a spoken dialog system by introducing a mechanism that gradually changes form of utterances every time the user uses the system.In some languages, including Japanese, the form of utterances changes corresponding to social relationship between the talker and the listener.Thus, this mechanism can be effective to express the system's intention to make social distance to the user closer; however, an actual effect of this method is not investigated enough when introduced to the dialog system.In this paper, we conduct dialog experiments and show that controlling the form of system utterances can improve the users' impression.

This paper focuses on language modeling with adequate robustness to support different domain tasks.To this end, we propose a hierarchical latent word language model (h-LWLM).The proposed model can be regarded as a generalized form of the standard LWLMs.The key advance is introducing a multiple latent variable space with hierarchical structure.The structure can flexibly take account of linguistic phenomena not present in the training data.This paper details the definition as well as a training method based on layer-wise inference and a practical usage in natural language processing tasks with an approximation technique.Experiments on speech recognition show the effectiveness of h-LWLM in out-of domain tasks.

In this paper, we propose a unique robot that moves on a handrail installed at corridors of buildings like hospitals. The robot guides the guests of the building. Because it moves on the handrail along the corridor, it naturally moves along the corridor without conflicting other people and needs less cost for estimating its position compared with human-sized mobile robots. We propose a method to control this robot so that it precedes a few steps before the person to be guided. We examine two methods: one is a method to keep a fixed distance from the person, and the other one is a method where the robot keeps moving slowly even when the robot and the person is distant. From the experiment, the latter method was more efficient than the former method.

This paper analyzes the problem of the spectral enhancement technique using global variance (GV) in HMM-based speech synthesis. In the conventional GV-based parameter generation, spectral enhancement with variance compensation is achieved by considering a GV pdf with fixed parameters for every output utterances through the generation process. Although the spectral peaks of the generated trajectory are clearly emphasized and subjective clarity is improved, the use of the fixed GV parameters results in a much smaller variation of GVs among the synthesized utterances than that of the natural speech, which sometimes causes undesirable effect. In this paper, we examine the above problem in terms of multiple objective measures such as variance characteristics, spectral and GV distortions, and GV correlations and discuss the result. We propose a simple alternative technique based on an affine transformation that provides a closer GV distribution to the original speech and improves the correlation of GVs of generated parameter sequences. The experimental results show that the proposed spectral enhancement outperforms the conventional GV-based parameter generation in the objective measures. Index Terms: HMM-based speech synthesis, global variance, parameter generation, over-smoothing, variance compensation

When using spoken dialog systems in ac-tual environments, users sometimes aban-don the dialog without making any in-put utterance. To help these users before they give up, the system should know why they could not make an utterance. Thus, we have examined a method to estimate the state of a dialog user by capturing the user’s non-verbal behavior even when the user’s utterance is not observed. The pro-posed method is based on vector quan-tization of multi-modal features such as non-verbal speech, feature points of the face, and gaze. The histogram of the VQ code is used as a feature for determining the state. We call this feature “the Bag-of-Behaviors. ” According to the experi-mental results, we prove that the proposed method surpassed the results of conven-tional approaches and discriminated the target user’s states with an accuracy of more than 70%. 1

Introduction of user models (e.g. models of a user's belief, skill and familiarity to the system) is believed to increase flexibility of response of a dialogue system. Conventionally, the internal state is estimated based on linguistic information of the previous utterance, but this approach cannot applied to the user who did not make an input utterance in the first place. Thus, we are developing a method to estimate an internal state of a spoken dialogue system's user before his/her input utterance. In a previous report, we used three acoustic features and a visual feature based on manual labels. In this paper, we introduced new features for the estimation: length of filled pause and face orientation angles. Then, we examined effectiveness of the proposed features by experiments. As a result, we obtained a three-class discrimination accuracy of 85.6% in an open test, which was 1.5 point higher than the result obtained using the previous feature set.

We investigated the effect of height of a robot on comfortableness of spoken dialog with the robot. We created a robot that could change the height continuously, and carried out dialog experiment with 18 subjects changing the robot's height. From the experimental result revealed the two observations: the "comfortable height" of a robot was lower than the eye height of a subject, and the comfortableness of dialog reduced when the height of the robot changed by 200 mm from the comfortable height.

This paper proposes discriminative modeling in a high dimensional feature space for spoken document retrieval (SDR). To estimate the parameters of a high dimensional model properly, a large quantity of data is necessary, but there is no such large corpus for document retrieval. This paper employs two approaches to overcome this problem. One is a reranking approach. A baseline system first gives each document a score and then the score is compensated by employing a high dimensional model. The other approach is automatic query generation. A large number of queries are automatically generated and used for parameter estimation. Our experimental result shows that our proposed method can greatly improve SDR performance.

Discriminative training has received a lot of attention from both the machine learning and speech recognition communities. The idea behind the discriminative approach is to construct a model that distinguishes correct samples from incorrect samples, while the conventional generative approach estimates the distributions of correct samples. We propose a novel discriminative training method and apply it to a language model for reranking speech recognition hypotheses. Our proposed method has round-robin duel discrimination (R2D2) criteria in which all the pairs of sentence hypotheses including pairs of incorrect sentences are distinguished from each other, taking their error rate into account. Since the objective function is convex, the global optimum can be found through a normal parameter estimation method such as the quasi-Newton method. Furthermore, the proposed method is an expansion of the global conditional log-linear model whose objective function corresponds to the conditional random fields. Our experimental results show that R2D2 outperforms conventional methods in many situations, including different languages, different feature constructions and different difficulties.

We previously proposed a new segment quantization method called Lempel-Ziv segment quantization (LZSQ), which is a modified version of Lempel-Ziv (LZ) coding that can be applied to a continuous information source. In the present paper, LZSQ is applied to the mixed excitation linear prediction (MELP) coder, which is a standardized vocoder-type speech coder that operates at 2.4 kbit/s, in order to reduce the bit rate to below 2.4 kbit/s, while preserving the quality of the coded speech. LZSQ is applied to six coding parameters of the MELP coder to reduce the total bit rate as much as possible. As a result, the total bit rate of the modified MELP coder was reduced to 1.57 kbit/s, while the subjective quality of the modified MELP coder is equivalent to that of the previous MELP coder.

This paper focuses on discriminative n-gram language models for large vocabulary speech recognition. We have proposed a novel training method called the round-robin duel discrimination (R2D2) method. Our previous report showed that R2D2 outperforms conventional methods on word n-gram based discriminative language models (DLMs). In this paper, we achieve additional error reduction and one-pass decoding at the same time. The keys to achieving this are the use of morphological features and the on-the-fly composition of weighted finite-state transducers (WFSTs) that represent both word and morphological discriminative features. Our experimental results show that R2D2 can reduce recognition errors more effectively than conventional methods in the reranking of n-best hypotheses and one-pass decoding can be accomplished with an equivalent accuracy.

In recent years, there has been increasing demand for ad hoc retrieval of spoken documents. We can use existing text retrieval methods by transcribing spoken documents into text data using a Large Vocabulary Continuous Speech Recognizer (LVCSR). However, retrieval performance is severely deteriorated by recognition errors and out-of-vocabulary (OOV) words. To solve these problems, we previously proposed an expansion method that compensates the transcription by using text data downloaded from the Web. In this paper, we introduce two improvements to the existing document expansion framework. First, we use a large-scale sample database of webdata as the source of relevant documents, thus avoiding the bias introduced by choosing keywords in the existing methods. Next, we use a document retrieval method based on a statistical language model (SLM), which is a popular framework in information retrieval, and also propose a new smoothing method considering recognition errors and missing keywords. Retrieval experiments show that the proposed methods yield a good results. Index Terms: Spoken document retrieval, statistical language models, World Wide Web

We propose an aspect-model-based reference speaker weighting. The main idea of the approach is that the adapted model is a linear combination of a set of reference speakers like reference speaker weighting (RSW) and eigenvoices. The aspect model is the mixture model of speaker-dependent (SD) models. In this paper, aspect model weighting (AMW) is proposed for finding an optimal weighting of a set of reference speakers unlike RSW and the aspect model which is a kind of cluster models is trained based on likelihood maximization with respect to the training data. The number of adaptation parameters can also be reduced using aspect model approach. For evaluation, we carried out an isolated word recognition experiment on Korean database (KLE452). The results were compared to those of conventional MAP, MLLR, RSW, and eigenvoice. Even though we use only 0.5s of adaptation data, 27.24% relative error rate reduction in comparison with speaker-independent (SI) baseline performance was achieved.

We developed a system that detects abnormal sound from sound signal observed by a surveillance microphone. Our system learns the ldquonormal soundrdquo from observation of the microphone, and then detects sounds never observed before as ldquoabnormal sounds.rdquo To this end, we developed a technique that uses multiple GMMs for modeling different levels of sound events efficiently. We also consider how to determine thresholds of GMM switching and event detection. As a result, we obtained almost same detection performance using the percentile method to the manually optimized GMMs. Besides, we exploited the segment-based feature, which gave the best result among all methods.

In this paper, we propose a novel data hiding technique for G.711 speech based on the LSB substitution method. The novel feature of the proposed method is that a low-bitrate encoder, G.726 ADPCM, is used as a reference for deciding how many bits can be embedded in a sample. Experiments showed that the method outperformed the simple LSB substitution method and the selective embedding method proposed by Aoki. We achieved 4-kbit/s embedding with almost no subjective degradation of speech quality, and 10 kbit/s while keeping good quality.

A new segment quantization method using the Lempel-Ziv algorithm is proposed, and it is applied to quantize line spectral frequency parameters in speech codec. The proposed segment quantizer can save four bits per frame, compared with the ITU-T G.729 speech codec (18 bits/frame), without degradation of subjective or objective speech quality

This paper describes a music information retrieval system that uses humming as the key for retrieval. Humming is an easy way for a user to input a melody. However, there are several problems with humming that degrade the retrieval of information. One problem is the human factor. Sometimes, people do not sing accurately, especially if they are inexperienced or unaccompanied. Another problem arises from signal processing. Therefore, a music information retrieval method should be sufficiently robust to surmount various humming errors and signal processing problems. A retrieval system has to extract the pitch from the user's humming. However, pitch extraction is not perfect. It often captures half or double pitches, which are harmonic frequencies of the true pitch, even if the extraction algorithms take the continuity of the pitch into account. Considering these problems, we propose a system that takes multiple pitch candidates into account. In addition to the frequencies of the pitch candidates, the confidence measures obtained from their powers are taken into consideration as well. We also propose the use of an algorithm with three dimensions that is an extension of the conventional Dynamic Programming (DP)algorithm, so that multiple pitch candidates can be treated. Moreover, in the proposed algorithm, DP paths are changed dynamically to take deltaPitches and IOIratios (inter-onset-interval) of input and reference notes into account in order to treat notes being split or unified. We carried out an evaluation experiment to compare the proposed system with a conventional system . When using three-pitch candidates with conference measure and IOI features, the top-ten retrieval accuracy was 94.1%. Thus, the proposed method gave a better retrieval performance than the conventional system.

This paper describes a method to detect smiles and laughter sounds from the video of natural dialogue. A smile is the most common facial expression observed in a dialogue. Detecting a user's smiles and laughter sounds can be useful for estimating the mental state of the user of a spoken-dialogue-based user interface. In addition, detecting laughter sound can be utilized to prevent the speech recognizer from wrongly recognizing the laughter sound as meaningful words. In this paper, a method to detect smile expression and laughter sound robustly by combining an image-based facial expression recognition method and an audio-based laughter sound recognition method. The image-based method uses a feature vector based on feature point detection from face images. The method could detect smile faces by more than 80% recall and precision rate. A method to combine a GMM-based laughter sound recognizer and the image-based method could improve the accuracy of detection of laughter sounds compared with methods that use image or sound only. As a result, more than 70% recall and precision rate of laughter sound detection was obtained from the natural conversation videos

Speech recognition by a small robot is difficult because the robot makes noise itself. In this paper, two new methods are proposed that suppresses internal noise of the small robots. These methods are based on spectral subtraction (SS). The difference of the proposed methods from the original SS is that the proposed methods use the estimated noise spectrum dependent on the motion of the robot. One method, called MDSS, prepares the noise spectrums for all motions. Another method, called NPSS, predicts the noise spectrum from angular velocities of all joints of the robot using a neural network. From the results of the comparison between the original SS and the proposed methods, the proposed methods outperformed the conventional SS. The NPSS worked well even when the noise of the motion was unstable, while the MDSS method gave good result when the noise in one motion was stable.

Speech recognition under noisy environment is one of the hottest topic in the speech recognition research. In this paper, we propose a method to improve accuracy of spoken dialog system from a dialog strategy point of view. In the proposed method, the dialog system automatically changes its dialog strategy according to the estimated recognition accuracy in noisy environment in order to keep the performance of the system constant. In a noisy environment, the system restricts its grammar and vocabulary to improve recognition accuracy. On the other hand, the system accepts any utterance from a user in a noise-free environment. To realize this strategy, we investigated a method to avoid user’s out-of-grammar utterances through an instruction given by the system to a user. Furthermore, we developed a method to estimate recognition accuracy from features extracted from noise signal. Finally, we constructed a proposed dialog system and confirmed its effectiveness.

We have been developing a spoken dialog system. Conventional spoken dialog systems need grammar descriptions and scripts of a dialog, that are difficult to develop. The system proposed in this paper is based on semantic frames, and the system generates the recognition grammar from the frames automatically. As the system requires only a frame-based description for a task of dialog, the system can be easily applied to different kinds of tasks. Moreover, the recognition accuracy is improved by sentence weighting based on phrase class template. We evaluated the system by experiments. The system reached the goal with 2.44 user’s utterances in average.

This paper describes a dialogue-based CALL (Computer Assisted Language Learning) system. One of the major problems in CALL systems is that learners are usually assigned a passive role. Learners have no practices in composing their own utterances. The other major problem is that lots of conventional CALL systems are pronunciation exercise systems. However, pronunciation exercise is only a part of exercise needed to increase a learner’s communication skill. In this paper, we propose a dialoguebased CALL system of new concept that enables exercise of composition, grammar and conversation in addition to pronunciation.

Several CALL systems have two acoustic models to evaluate a learner’s pronunciation. In order to achieve high performance for evaluation, speaker adaptation method is introduced in CALL system. It requires adaptation data of a target language, however, a learner cannot pronounce correctly. In this paper, we proposed two types of new speaker adaptation methods for CALL system. The new methods only require learner’s utterance of the native language for adaptation. The first method is an algorithm to adapt acoustic models using bilingual’s utterances. The speakerindependent acoustic models of native and target languages are adapted to a bilingual speaker once, then they are adapted to the learner again using the learner’s speech of the native language. Phoneme recognition accuracy is about 5% higher than the baseline method. The second method is a training algorithm of an acoustic model. It can robustly train bilinguals’ model from a few bilinguals’ utterances. Phoneme recognition accuracy is about 10% higher than the baseline method.

We adopted the Korean phonological rules to state clustering of contextual domain for representing the unknown contexts and tying the model parameters of new states in state clustering of SSS (successive state splitting). We used the decision tree-based successive state splitting (DT-SSS) algorithm, which splits the state of contexts based on phonetic knowledge. The SSS algorithm proposed by Sagayama (1992) is a powerful technique, which designed topologies of tied-state HMMs automatically, but it does not generate unknown contexts adequately. In addition it has some problem in the contextual splits procedure. In this paper, speaker independent Korean isolated word and sentence recognition experiments are carried out. In word recognition experiments, this method shows an average of 6.3% higher word recognition accuracy than the conventional HMMs, and in sentence recognition experiments, it shows an average of 90.9% recognition accuracy.

A sharable software repository for Japanese LVCSR (Large Vocabulary Continuous Speech Recognition) is introduced. It is designed as a baseline platform for research and developed by researchers of different academic institutes under a governmental support. The repository consists of a recognition engine (Julius), Japanese acoustic models and statistical language models as well as Japanese morphological analysis tools. These modules can be easily integrated and replaced under a plug-and-play framework, which makes it possible to fairly evaluate components and to develop specific application systems. Assessment of these modules and systems in a 20000-word dictation task is reported. The software repository is freely available to the public.

Though Perplexity shows good correlation with word error rate within simple n-gram framework like Wall Street Journal task, it has been reported that perplexity have poor correlation with WER when more complicated LM is used. In this paper, a global measure for language model evaluation is proposed which achieveshigher correlation between word accuracy. The metric is based on difference of LM score between a word in the evaluation text and the word that gives the maximum score at that context. Two experiments were carried out to investigate the correlation between word accuracy and the proposed measure. In the first experiment, LMs in this paper were created using n-gram adaptation by n-gram count mixture. 47 LMs were created for the experiments by changing mixture weight and vocabulary cut-off threshold. Correlation betwen perplexity and word accuracy was very poor (correlation coefficient -0.36). On the other hand, the proposed metric gave much higher correlation (correlation coefficient 0.82). In the second experiment, a simple mixture trigram model was employed to recognize Switchboard task data. The highest correlation between word accuracy and the proposed method was 0.81, which was much higher than the correlation between PP and accucary 0.59.

This paper describes a new powerful statistical language model based on N-gram model for Japanese speech recognition.In English, a sentence is written word-by-word.On the other hand, a sentence in Japanese has no word boundary character.Therefore, a Japanese sentence requires word segmentation by morphemic analysis before the construction of word N-gram.We propose an N-gram based language model which requires no word segmentation.This model uses character string patterns as units of N-gram.The string patterns are chosen from the training text according to a statistical criterion.We carried out several experiments to compare perplexities of the proposed and the conventional models, which showed the advantage of our model.For many of the readers' interest, we applied this method to English text.As the result of a preliminary experiment, the proposed method got better performance than conventional word trigram.

This paper presents the performance evaluation method on sentence recognition system which uses a finite state automaton. The relationship between word recognition score and sentence recognition score can be predicted using the number of word sequences at a short distance from the sentence. But it is not clear that how we get this number when the finite state automaton is used as linguistic information. Therefore, we propose the algorithm to calculate this number in polynomial time. Then we carry out the prediction using this method and the simulation to compare with the prediction, and it is shown that our method is usable when the quality of the word lattice is good.>

This paper describes an overview of a continuous speech recognition system composed of an acoustic processor and a linguistic processor. The system deals with 843 conceptual words and 431 functional words. We have constructed an acoustic processor using a modified learning vector quantization method (MLVQ2) for phoneme recognition. The phoneme recognition score was 85.5% for 226 sentences uttered by two male speakers. The linguistic processor is composed of a processor for spotting bunsetsu units (i.e. units similar to a “phrase” in English) and a syntactic processor. The structure of the bunsetsu unit is effectively described by a finite-state automaton, the test-set word-perplexity of which is 230. In the processor for spotting bunsetsu units, using a syntax-driven continuous-DP matching algorithm, the bunsetsu units are spotted from a recognized phoneme sequence and then a bunsetsu unit lattice is generated. In the syntactic processor, the bunsetsu unit lattice is parsed based on the dependency grammar, which is expressed as the correspondence between a FEATURE marker in a modifier-bunsetsu and a SLOT-FILLER marker in a head-bunsetsu. The recognition scores of the bunsetsu units and conceptual words were 75.2% and 88.9% respectively for 226 sentences uttered by the two male speakers.

A prototype of a Japanese text dictation system has been developed. It is composed of an acoustic processor, a Bunsetsu-unit spotting processor, and a syntactic processor with semantic constraints. The acoustic processor is constructed using the modified LVQ2 method. The modified LVQ2 method achieves a high phoneme recognition performance of 86.1%. The syntax driven continuous DP matching algorithm is used for spotting Bunsetsu-units. This method greatly reduces the amount of computation and the storage capacity necessary for spotting the Bunsetsu units. Analysis of the dependency structure among the Bunsetsu-unit candidates is effectively carried out using the syntactic and semantic information.>