In this paper, we propose a musical-noise-free blind speech extraction method using a microphone array for application to nonstationary noise. In our previous study, it was found that optimized iterative spectral subtraction (SS) results in speech enhancement with almost no musical noise generation, but this method is valid only for stationary noise. The proposed method consists of iterative blind dynamic noise estimation by, e.g., independent component analysis (ICA) or multichannel Wiener filtering, and musical-noise-free speech extraction by modified iterative SS, where multiple iterative SS is applied to each channel while maintaining the multichannel property reused for the dynamic noise estimators. Also, in relation to the proposed method, we discuss the justification of applying ICA to signals nonlinearly distorted by SS. From objective and subjective evaluations simulating a real-world hands-free speech communication system, we reveal that the proposed method outperforms the conventional methods.

In this paper, we present novel speaking-aid systems based on one-to-many eigenvoice conversion (EVC) to enhance three types of alaryngeal speech: esophageal speech, electrolaryngeal speech, and body-conducted silent electrolaryngeal speech. Although alaryngeal speech allows laryngectomees to utter speech sounds, it suffers from the lack of speech quality and speaker individuality. To improve the speech quality of alaryngeal speech, alaryngeal-speech-to-speech (AL-to-Speech) methods based on statistical voice conversion have been proposed. In this paper, one-to-many EVC capable of flexibly controlling the converted voice quality by adapting the conversion model to given target natural voices is further implemented for the AL-to-Speech methods to effectively recover speaker individuality of each type of alaryngeal speech. These proposed systems are compared with each other from various perspectives. The experimental results demonstrate that our proposed systems are capable of effectively addressing the issues of alaryngeal speech, e.g., yielding significant improvements in speech quality of each type of alaryngeal speech.

In this paper, we propose a new theory of nonlinear noise reduction with a perfectly musical-noise-free property, where no musical noise is generated even for a high signal-to-noise ratio. To achieve high-quality noise reduction with low musical noise, an iterative spectral subtraction method, i.e., recursively applied weak nonlinear signal processing, has been proposed. Although evaluation experiments indicated the existence of an appropriate parameter setting that gives a musical-noise-free state, no theoretical studies have been carried out. Therefore, in this paper, we theoretically derive pairs of internal parameters that satisfy the musical-noise-free condition by analysis based on higher-order statistics. It is clarified that finding a fixed point in the kurtosis of noise spectra enables the reproduction of the musical-noise-free state, and comparative experiments with commonly used noise reduction methods show the efficacy of the proposed method.

In this paper, we propose a new method for stable estimation of the kurtosis of a speech power spectrum. Speech kurtosis can be used for the prediction of speech recognition accuracy as reported in recent studies. However, the conventional estimation method is very unstable owing to the high sensitivity of higher-order statistics. To overcome this problem, we introduce the generalized Gaussian distribution prior in order to avoid the calculation of higher-order statistics, and construct a kurtosis table that directly represents the relationship among the kurtosis of speech, noise, and their mixture in the power spectrum domain. Speech kurtosis can be estimated stably from observable data by looking up values in the table. An experimental evaluation confirms the efficacy of the proposed method.

This paper describes a voice quality control method in statistical esophageal speech enhancement. Esophageal speech is produced by one of the alternative speaking methods for laryngectomees. Its naturalness and intelligibility are much lower than those of natural voices and its voice quality sounds similar even if uttered by different laryngectomees. These issues are alleviated by a statistical voice conversion method from esophageal speech into normal speech (ES-to-Speech) based on eigenvoices. This method is capable of determining converted voice quality using a few target voice samples. In this paper, we propose ES-to-Speech using regression techniques to make it possible to manually control the converted voice quality by manipulating a few intuitively controllable parameters even if no target voice sample is available. The effectiveness of the proposed method is confirmed by experimental evaluations.

The need for robust pronunciation annotation over out-of-vocabulary (OOV) words has been increasing with the development of an application that deals with proper nouns and brand-new words, such as Voice Search. In robust pronunciation annotation over OOV words, the alignment between graphemes and phonemes is vital data. For a many-to-many alignment algorithm between graphemes and phonemes, we describe its problems and methods to overcome them. An evaluation experiment of a many-to-many alignment by automatic pronunciation annotation using Web text mining is also performed. That experimental result shows that the proposed many-to-many alignment produces an alignment that has the high generalization ability for OOV words while avoiding degradation of the accuracy of the pronunciation annotation compared with the conventional approach.

INTERSPEECH 2012: The 13th Annual Conference of the International Speech Communication Association, September 9-13, 2012, Portland, Oregon, USA.

In this paper, we provide a theoretical analysis of the amount of musical noise in iterative spectral subtraction, and its optimization method for the least musical noise generation. To achieve high-quality noise reduction with low musical noise, iterative spectral subtraction, i.e., iteratively applied weak nonlinear signal processing, has been proposed. Although the effectiveness of the method has been reported experimentally, there have been no theoretical studies. Therefore, in this paper, we formulate the generation process of musical noise by tracing the change in kurtosis of noise spectra, and conduct a comparison of the amount of musical noise for different parameter settings but the same achieved level of noise attenuation. Furthermore, we theoretically derive the optimal internal parameters that generate no musical noise. It is clarified that to find a fixed point in kurtosis yields the no-musical-noise property. Comparative experiments with commonly used noise reduction methods show the proposed method's efficacy.

In this paper, we present statistical approaches to enhance body-conducted unvoiced speech for silent speech communication. A body-conductive microphone called nonaudible murmur (NAM) microphone is effectively used to detect very soft unvoiced speech such as NAM or a whispered voice while keeping speech sounds emitted outside almost inaudible. However, body-conducted unvoiced speech is difficult to use in human-to-human speech communication because it sounds unnatural and less intelligible owing to the acoustic change caused by body conduction. To address this issue, voice conversion (VC) methods from NAM to normal speech (NAM-to-Speech) and to a whispered voice (NAM-to-Whisper) are proposed, where the acoustic features of body-conducted unvoiced speech are converted into those of natural voices in a probabilistic manner using Gaussian mixture models (GMMs). Moreover, these methods are extended to convert not only NAM but also a body-conducted whispered voice (BCW) as another type of body-conducted unvoiced speech. Several experimental evaluations are conducted to demonstrate the effectiveness of the proposed methods. The experimental results show that 1) NAM-to-Speech effectively improves intelligibility but it causes degradation of naturalness owing to the difficulty of estimating natural fundamental frequency contours from unvoiced speech; 2) NAM-to-Whisper significantly outperforms NAM-to-Speech in terms of both intelligibility and naturalness; and 3) a single conversion model capable of converting both NAM and BCW is effectively developed in our proposed VC methods.

In this paper we present a novel approach to acoustic model training for non-audible murmur (NAM) recognition using normal speech data transformed into NAM data. NAM is extremely soft murmur, that is so quiet that people around the speaker can hardly hear it. It is detected directly through the soft tissue of the head using a special body-conductive microphone, NAM microphone, placed on the neck below the ear. NAM recognition is one of the promising silent speech interfaces for man-machine speech communication. We have previously shown the effectiveness of speaker adaptive training (SAT) based on constrained maximum likelihood linear regression (CMLLR) in NAM acoustic model training. However, since the amount of available NAM data is still small, the effect of SAT is limited. In this paper we propose modified SAT methods capable of using a larger amount of normal speech data by transforming them into NAM data. The experimental results demonstrate that the pro posed methods yield an absolute increase of approximately 2% in word accuracy compared with the conventional method.

In this study, we evaluate our proposed methods for enhancing alaryngeal speech based on statistical voice conversion techniques. Voice conversion based on a Gaussian mixture model has been applied to the conversion of alaryngeal speech into normal speech (AL-to-Speech). Moreover, one-to-many eigenvoice conversion (EVC) has also been applied to AL-to-Speech to enable the recovery of the original voice quality of laryngectomees even if only one arbitrary utterance of the original voice is available. VC/EVC-based AL-to-Speech systems have been developed for several types of alaryngeal speech, such as esophageal speech (ES), electrolaryngeal speech (EL), and body-conducted silent electrolaryngeal speech (silent EL). These proposed systems are compared with each other from various perspectives. The experimental results demonstrate that our proposed systems yield significant enhancement effects on each type of alaryngeal speech.

Recently, one of the authors has reported that the amount of generated musical noise is strongly correlated with higher-order statistics of the power spectra. On the basis of this finding, in this paper, we provide a new theoretical analysis of the amount of musical noise generated via the Wiener filtering family. Our theoretical analysis allows the universal performance description from the viewpoint of the amount of musical noise generation and that of noise reduction, enabling reasonable sound quality comparison under the same noise reduction performance. From a mathematical analysis and evaluation experiments, we also clarify which parameter settings result in less musical noise generated in the Wiener filtering family.

For a reproduced sound field, the competing goals between the listening area and reproduction accuracy in an actual environment is one of the most important problems in sound field reproduction using loudspeakers. In this paper, we propose a new method of balancing these goals with absolute accuracy using an inverse filter of the room acoustics: the null space of a generalized inverse matrix given by a compensation filter of the wave field outside the control points. To develop an expression for the compensation filter, we use the loudspeaker driving function of wave field synthesis (WFS) in stead of the filter used in conventional studies. By using WFS, the proposed method overcomes the compensation limitation of auditory distance and azimuth perception outside the control points. The results of computer simulations revealed that the proposed method balances the competing goals and has wide applicability in a spatial domain with high accuracy of reproduction both under free-field conditions and in a simulation model with room reflection.

In this paper, to automatically generate musical thumbnails that con tain the main part of the original tune, we propose a new estimation method for identifying structure changes in stereo tunes based on localization information. The proposed method can estimate the main parts of a musical tune by analyzing the specific timing when localization information changes under the assumption that the changing time of the localization approximately corresponds to the timing of the musical structure change. We evaluate the effectiveness of the proposed method by objective and subjective assessments. The experimental results show that the proposed method is effective in automating musical structure analysis for generating musical thumb nails.

This paper describes a novel approach based on voice conversion (VC) to speaker-adaptive speech synthesis for speech-tospeech translation. Voice quality of translated speech in an output language is usually different from that of an input speaker of the translation system since a text-to-speech system is developed with another speaker’s voices in the output language. To render the input speaker’s voice quality in the translated speech, we propose a voice quality control method based on one-tomany eigenvoice conversion (EVC) and language-dependent prosodic conversion. Spectral parameters of the translated speech are effectively converted by one-to-many EVC enabling unsupervised speaker adaptation. Moreover, prosodic parameters are modified considering their global differences between the input and output languages. The effectiveness of the proposed method is confirmed by experimental evaluations on cross-lingual VC among Japanese, English, and Chinese. Index Terms: speech-to-speech translation, speech synthesis,

INTERSPEECH 2011: 12th Annual Conference of the International Speech Communication Association, 28-31 August, 2011, Florence, Italy.

In this paper, to achieve high-quality speech enhancement, we introduce the generalized minimum mean-square error short-time spectral amplitude estimator with a new blind prior estimation of the speech probability density function (p.d.f.). To deal with various types of speech signals with different p.d.f., we propose an algorithm of speech kurtosis estimation based on moment-cumulant transformation for blind adaptation to the shape parameter of speech p.d.f. From the objective and subjective evaluation experiments, we show the improved noise reduction performance of the proposed method.

In this paper, we provide a new theoretical analysis of the amount of musical noise generated via generalized spectral subtraction based on higher order statistics. Power spectral subtraction is the most commonly used spectral subtraction method, and in our previous study a musical noise assessment theory limited to the power spectral domain was proposed. In this paper, we propose a generalization of our previous theory on spectral subtraction for arbitrary exponent parameters. We can thus compare the amount of musical noise between any exponent domains from the results of our analysis. We also clarify that less musical noise is generated when we choose a lower exponent spectral domain; this implies that there is no theoretical justification for using power/amplitude spectral subtraction.

In this paper, we propose a musical-noise-controllable algorithm for array signal processing with the aim for high-performance and high-quality noise reduction. Recently, many methods of integrating linear microphone array signal processing and nonlinear signal processing for noise reduction have been studied, but these methods often suffer from the problem of musical noise. In the proposed algorithm, channelwise spectral subtraction is applied before adaptive array signal processing. We also introduce a new automatic control algorithm to obtain the subtraction strength parameter used in the spectral subtraction, which depends on the amount of generated musical noise, measured by higher order statistics. We confirm the effectiveness of the proposed algorithm via objective and subjective evaluations.

This paper presents a novel method of enhancing esophageal speech using statistical voice conversion. Esophageal speech is one of the alternative speaking methods for laryngectomees. Although it doesn't require any external devices, generated voices sound unnatural. To improve the intelligibility and naturalness of esophageal speech, we propose a voice conversion method from esophageal speech into normal speech. A spectral parameter and excitation parameters of target normal speech are separately estimated from a spectral parameter of the esophageal speech based on Gaussian mixture models. The experimental results demonstrate that the proposed method yields significant improvements in intelligibility and naturalness. We also apply one-to-many eigenvoice conversion to esophageal speech enhancement for flexibly controlling enhanced voice quality.

Several recent methods for speech enhancement in presence of diffuse background noise use frequency domain blind signal separation to estimate the diffuse noise and a nonlinear post filter to suppress this estimated noise. This paper presents a frequency domain blind signal extraction method for estimating the diffuse noise in place of the frequency domain blind signal separation. The method is based on the minimization by means of a complex Newton algorithm of a cost function depending of the modulus of the extracted component. The proposed complex Newton method is compared to the gradient descent on the same cost function and to the blind signal separation approach.

This paper study the blind estimation of the diffuse background noise for the hands-free speech interface. Some recent papers showed that it is possible to use blind signal separation (BSS) to estimate the diffuse background noise by suppressing the speech component after all the components were separated. In particular, the scale indeterminacy of BSS is avoided by using the projection back method. In this paper, we study an alternative to the projection back for the noise estimation and justify the use of blind signal extraction BSE rather than BSS.

This paper presents a novel training method of an eigenvoice Gaussian mixture model (EV-GMM) effectively using non-parallel data sets for many-to-many eigenvoice conversion, which is a technique for converting an arbitrary source speaker's voice into an arbitrary target speaker's voice. In the proposed method, an initial EV-GMM is trained with the conventional method using parallel data sets consisting of a single reference speaker and multiple pre-stored speakers. Then, the initial EV-GMM is further refined using non-parallel data sets including a larger number of pre-stored speakers while considering the reference speaker's voices as hidden variables. The experimental results demonstrate that the proposed method yields significant quality improvements in converted speech by enabling us to use data of a larger number of pre-stored speakers.

In this paper, we propose a new blind speech extraction method consisting of a minimum mean-square error short-time spectral amplitude (MMSE STSA) estimator and noise estimation based on independent component analysis (ICA). First, we perform a computer simulation using the artificial noise whose stationarity could be controlled parametrically, and the obtained results indicate that the proposed method is superior to conventional methods, such as blind spatial subtraction array (BSSA) and the original MMSE STSA estimator under the non-point-source and nonstationary noise condition. Finally, we conduct an experiment in an actual railway-station environment, and objective and subjective evaluations to confirm the advantage of the proposed method in the real world.

In our previous work, we proposed a speaking-aid system converting electrolaryngeal speech (EL speech) to normal speech using a statistical voice conversion technique. The main weakness of our system is the difficulty of estimating natural contours of the fundamental frequency (F0) from EL speech including only built-in F0 contours. This paper proposes another speaking-aid system with an air-pressure sensor to enable laryngectomees to control F0 contours of the EL speech using their breathing air. The experimental result demonstrates that 1) the correlation coefficient of F0 contours between the converted and the target speech is improved from 0.58 to 0.78 by the use of the air-pressure sensor and 2) the synthetic speech converted by the proposed system sounds more natural and is more preferred to that by our conventional aid system.

This paper presents adaptive voice-quality control methods based on one-to-many eigenvoice conversion. To intuitively control the converted voice quality by manipulating a small number of control parameters, a multiple regression Gaussian mixture model (MR-GMM) has been proposed. The MR-GMM also allows us to estimate the optimum control parameters if target speech samples are available. However, its adaptation performance is limited because the number of control parameters is too small to widely model voice quality of various target speakers. To improve the adaptation performance while keeping capability of voice-quality control, this paper proposes an extended MR-GMM (EMR-GMM) with additional adaptive parameters to extend a subspace modeling target voice quality. Experimental results demonstrate that the EMR-GMM yields significant improvements of the adaptation performance while allowing us to intuitively control the converted voice quality.

INTERSPEECH2010: 11th Annual Conference of the International Speech Communication Association, September 26-30, 2010, Chiba, Japan.

In this paper, we propose a microphone array structure for a spoken-oriented robot dialog system that is designed to discriminate the direction of arrival (DOA) of the target speech and that of the robot internal noise. First, we investigate the performance of the noise estimation conducted by semi-blind source separation (SBSS) in presence of both the diffuse background noise and the robot internal noise. The result indicates that the noise estimation of the SBSS is not good. Next, we analyze the DOA of the robot internal noise in order to determine the reason of the above result; we find out that the internal noise is always in-phase at the microphone array and overlap spacial with the target speech. Based on this fact, we propose to change the microphone array structure from the broadside array to the end-fire array in order to discriminate the DOAs of the target speech and the internal noise. Finally, we evaluate the word accuracy in a dictation task in presence of both diffuse background noise and robot internal noise to confirm the advantage of the proposed structure. Simulation results shows that the proposed microphone array structure results in approximately 10% improvement of the speech recognition performance.

Non-audible murmur (NAM) is an unvoiced speech signal that can be received through the body tissue with the use of special acoustic sensors (i.e., NAM microphones) attached behind the talker's ear. The authors had previously reported experimental results for NAM recognition using a stethoscopic and a silicon NAM microphone. Using a small amount of training data from a single speaker and adaptation approaches, 93.9% of word accuracy was achieved for a 20 k Japanese vocabulary dictation task. In this paper, further analysis of NAM speech is made using distance measures between hidden Markov models (HMMs). It has been shown that owing to the reduced spectral space of NAM speech, the HMM distances are also reduced when compared with those of normal speech. In the case of Japanese vowels and fricatives, the distance measures in NAM speech follow the same relative inter-phoneme relationship as that in normal speech without significant differences. However, significant differences have been found in the case of Japanese plosives. More specifically, in NAM speech, the distances between voiced/unvoiced consonant pairs articulated in the same place drastically decreased. As a result, the inter-phoneme relationship as compared to normal-speech changed significantly, causing a substantial decrease in the recognition accuracy. A speaker-dependent phoneme recognition experiment has been conducted, obtained 81.5% NAM phoneme correct, showing a relationship between HMM distance measures and phoneme accuracy. In a NAM microphone, body transmission and loss of lip radiation act as a low-pass filter. As a result, higher frequency components are attenuated in a NAM signal. Because of spectral reduction, NAM's unvoiced nature, and the type of articulation, NAM sounds become similar, causing a larger number of confusions when compared with normal speech. Yet many of those sounds are visually different on face/mouth/lips, and the integration of visual information increases their discrimination. As a result, recognition accuracy increases as well. In this article, the visual information extracted from the talkers' facial movements is fused with NAM speech. The experimental results reveal a relative improvement of 10.5% on average when fused NAM speech and facial information were used compared with using only NAM speech.

A new blind source extraction method in widespread noise conditions is proposed, which is based on multiple frequency-domain independent component analysis (FDICA) combining projection back and spectral subtraction. In addition, We implement the proposed method to digital signal processor (DSP) for a more realistic real-time operation, and develop a new blind source extraction (BSE) microphone which can extract a target sound in real-time. In this paper, we illustrate and evaluate the proposed method and BSE microphone. And experimental results reveal that the extraction performance of the proposed method are superior to that of conventional methods, and we show the efficacy of microphone.

In this paper we propose a new unsupervised training method for nonlinear spatial filter using a new independent component analysis based on kernel infomax. The nonlinearity of the spatial filter used in this paper is equivalent to the integration of beamforming and spectral subtraction, and the whole structure is optimized by independent component analysis in the reproducing kernel Hilbert space. The optimized filter is shown to be capable of achieving better quality output than the conventional method based on time-frequency binary masking.

Statistical voice conversion is very effective for enhancing body transmitted speech recorded with Non-Audible Murmur (NAM) microphone. In this method, a probabilistic model to convert body transmitted speech into natural speech is trained previously. Because acoustic characteristics of body transmitted speech is sensitive to recording conditions such as a location of NAM microphone, significant degradation of the conversion performance is often caused in practical situations by acoustic mismatches between training and conversion processes. To alleviate this problem, we propose unsupervised acoustic compensation methods for body transmitted voice conversion. Experimental results demonstrate that the proposed methods significantly reduce the quality degradation of converted speech caused by the acoustic mismatches.

In this paper, we describe and review our recent development of hands-free speech dialogue system which is used for railway station guidance. In the application at the real railway station, robustness against reverberation and noise is the most essential issue for the dialogue system. To address the problem, we introduce two key techniques in our proposed hands-free system; (a) speech dialogue system construction with real speech database collection and language/acoustic model improvement, and (b) microphone array preprocessing using blind spatial subtraction array which can solve the reverberation-naiveness problem inherent in conventional microphone arrays. The experimental assessment of the proposed dialogue system reveals that our system can provide the recognition accuracy of more than 80% under realistic railway-station conditions.

In this paper, we propose a new ICA-based BSS algorithm including estimation of sources' probability density functions (PDFs) to adapt the nonlinear activation function to various noise conditions. In the proposed method, closed-form second-order ICA is introduced as a computational-cost-efficient preprocessing to extract sources' PDFs, which is beneficial for real-time application. Compared with various type of conventional ICAs, e.g., fixed activation-function type and ML-based type, our proposed algorithm can give a faster and higher convergence. Based on the proposed source-adaptive ICA, we show a real-time noise reduction results under diffuse noise environment. Also we can demonstrate our recently developed hands-free robot spoken dialogue system via real-time ICA.

In this paper, we conduct an analysis for reduction of musical noise in integration method of microphone array signal processing and nonlinear signal processing. In these days, for better noise reduction, integration methods of microphone array signal processing and nonlinear signal processing have been researched. However, non-linear signal processing causes musical noise. Since such musical noise make users uncomfortable, it is desired that musical noise is mitigated. Moreover, in these days, it is reported that higher-order statistics is strongly related with the amount of generated musical noise. Thus, we analyze the integrated method of microphone array signal processing and nonlinear signal processing, based on higher-order statistics. Also, we propose an architecture for reducing musical noise based on the analysis. The effectiveness of the proposed architecture and analysis correctness are shown via a computer simulation and a subjective evaluation.

In this paper, we review our proposed statistical voice conversion approaches to enhancing various types of body transmitted speech captured with non-audible murmur (NAM) microphone. Body transmitted speech conversion is a potential technique to bring a new paradigm to human-to-human speech communication. In addition to our previously proposed methods of enhancing body transmitted unvoiced speech for silent speech communication and of enhancing body transmitted artificial speech for speaking aid, we further propose conversion methods of enhancing body transmitted voiced speech for noise robust speech communication. An experimental result demonstrates that the proposed methods yield significant improvements in quality of body transmitted voiced speech.

In this paper, we reveal new findings about the generated musical noise in minimum mean-square error short-time spectral amplitude (MMSE STSA) processing. Recently we have proposed a objective metric of musical noise based on kurtosis change ratio on spectral subtraction (SS). Also we found an interesting relationship among the degree of generated musical noise, the shapes of signal-s probability density function, the strength parameter of SS processing. This paper is aimed to automatically evaluate the sound quality of various types of noise reduction methods using kurtosis change ratio. We give a mathematical analysis based on higher-order statistics viewpoint, and lead to a valuable relation in that MMSE STSA has a weakness in speech period distortion rather than noise period, and vice versa in SS.

This paper proposes a speaking-aid system for laryngectomees using GMM-based voice conversion that converts electrolaryngeal speech (EL speech) to normal speech. Because valid F0 information cannot be obtained from the EL speech, we have so far converted the EL speech to whispering. This paper conducts the EL speech conversion to normal speech using F0 counters estimated from the spectral information of the EL speech. In this paper, we experimentally evaluate these two types of output speech of our speaking-aid system from several points of view. The experimental results demonstrate that the converted normal speech is preferred to the converted whisper.

In this paper, we propose many-to-many voice conversion (VC) techniques to convert an arbitrary source speaker's voice into an arbitrary target speaker's voice. We have proposed one-to-many eigenvoice conversion (EVC) and many-to-one EVC. In the EVC, an eigenvoice Gaussian mixture model (EV-GMM) is trained in advance using multiple parallel data sets of a reference speaker and many pre-stored speakers. The EV-GMM is flexibly adapted to an arbitrary speaker using a small amount of adaptation data without any linguistic constraints. In this paper, we achieve many-to-many VC by sequentially performing many-to-one EVC and one-to-many EVC through the reference speaker using the same EV-GMM. Experimental results demonstrate the effectiveness of the proposed many-to-many VC.

INTERSPEECH2009: 10th Annual Conference of the International Speech Communication Association, September 6-10, 2009, Brighton, UK.

The speech enhancement architecture presented in this paper is specifically developed for hands-free robot spoken dialog systems. It is designed to take advantage of additional sensors installed inside the robot to record the internal noises. First a modified frequency domain blind signal separation (FD-BSS) gives estimates of the noises generated outside and inside of the robot. Then these noises are canceled from the acquired speech by a multichannel Wiener post-filter. Some experimental results show the recognition improvement for a dictation task in presence of both diffuse background noise and internal noises.

We propose a new blind spatial subtraction array (BSSA) consisting of a noise estimator based on independent component analysis (ICA) for efficient speech enhancement. In this paper, first, we theoretically and experimentally point out that ICA is proficient in noise estimation under a non-point-source noise condition rather than in speech estimation. Therefore, we propose BSSA that utilizes ICA as a noise estimator. In BSSA, speech extraction is achieved by subtracting the power spectrum of noise signals estimated using ICA from the power spectrum of the partly enhanced target speech signal with a delay-and-sum beamformer. This ldquopower-spectrum-domain subtractionrdquo procedure enables better noise reduction than the conventional ICA with estimation-error robustness. Another benefit of BSSA architecture is ldquopermutation robustness". Although the ICA part in BSSA suffers from a source permutation problem, the BSSA architecture can reduce the negative affection when permutation arises. The results of various speech enhancement test reveal that the noise reduction and speech recognition performance of the proposed BSSA are superior to those of conventional methods.

Recently, methods using blind signal separation were proposed to separate the signals received by a microphone array. In this paper, we propose a new frequency domain semi-blind source separation method for replacing the blind source separation method when it is possible to obtain additional information on some of the signals. This is of particular interest in situations like in hands-free speech recognition where the blind separation has to work on limited amount of data in a challenging environment. The proposed method incorporates references to some of the signals that are obtained by additional sensors. Some experimental results shows that the proposed method is able to incorporate the additional information efficiently and that the performances are improved in term of SNR and word accuracy in a speech recognition task.

We propose a robust and fast dereverberation technique for real-time speech recognition application. First, we effectively identify the late reflection components of the room impulse response. We use this information together with the concept of Spectral Subtraction (SS) to remove the late reflection components of the reverberant signal. In the absence of the clean speech in actual scenario, approximation is carried out in estimating the late reflection where the estimation error is corrected through multi-band SS. The multi-band coefficients are optimized during offiine training and used in the actual online dereverberation. The proposed method performs better and faster than the relevant approach using Multi-LPC and reverberant matched model. Moreover the proposed method is robust to speaker and microphone locations.

We propose methods to analyze and control source localization of stereo audio signals using blind source separation (BSS) based on independent component analysis (ICA). Although an inverse system of separation compensates distortion caused by ICA as reconstruction of stereo spatial characteristics, this technique is insufficient to analyze localization because it achieves compensation of distortion and reconstruction of spatial characteristics simultaneously. Thus we analyze spatial characteristics effectively by dividing the compensation into two steps: monaural-output compensation of distortion and its reconstruction of spatial characteristics. Additionally, we control the localization of each source by modifying the analyzed spatial characteristics. It is shown that the proposed method can be applied to stereo signals consisting of more than two sources.

Weakness against a user's position shifting is one of the most important problems of binaural reproduction using loudspeakers. In this paper we propose a new method of inverse filtering of room acoustics with high robustness against a user's position shifting by presenting a wavefront estimated from the binaural recording. To analyze and synthesize the wavefront, we introduce a new physical modelling of the superimposition of the wavefronts weighted by multiple sound sources. By following the fluctuation of the wavefront with a time-varying filter, the analysis and synthesis overcome the limitation of the number of sound sources. Utilizing arbitrary components of the generalized inverse matrix, wavefront approximation does not degrade the accuracy of reproduction at the controlled area of the inverse filter.

INTERSPEECH2008: 9th Annual Conference of the International Speech Communication Association, September 22-26, 2008, Brisbane, Australia.

As typical voice conversion methods, two spectral conversion processes have been proposed: 1) the frame-based conversion that converts spectral parameters frame by frame and 2) the trajectory-based conversion that converts all spectral parameters over an utterance simultaneously. The former process is capable of real-time conversion but it sometimes causes inappropriate spectral movements. On the other hand, the latter process provides the converted spectral parameters exhibiting proper dynamic characteristics but a batch process is inevitable. To achieve the real-time conversion process considering spectral dynamic characteristics, we propose a time-recursive conversion algorithm based on maximum likelihood estimation of spectral parameter trajectory. Experimental results show that the proposed method achieves the low-delay conversion process, e.g., only one frame delay, while keeping the conversion performance comparably high to that of the conventional trajectory-based conversion. Index Terms: speech synthesis, voice conversion, Gaussian mixture model, maximum likelihood estimation, time-recursive algorithm. 1.

In this paper, we evaluate our previously proposed speaking-aid system with voice conversion for laryngectomees. The proposed system employs a sound source unit generating extremely small signals to keep them from annoying other persons, and then it statistically converts articulated signals captured with a bodyattached microphone into natural speech. We have so far shown the effectiveness of the proposed system using speech data imitated by a non-laryngectomee, which have recorded in a sound proof room. In this paper, we further investigate 1) whether such small sound source signals cause the lack of auditory feedback under noisy environments and 2) whether the proposed system is effective for real laryngectomees. Experimental results demonstrate that 1) an explicit auditory feedback is useful to keep the speaker's articulation stable and 2) the voice conversion dramatically improves the naturalness of the laryngectomee's speech but it slightly degrades its intelligibility.

We have previously developed a one-to-many eigenvoice conversion (EVC) system enabling the conversion from a specific source speaker's voice into an arbitrary target speaker's voice. In this system, eigenvoice Gaussian mixture model (EV-GMM) is trained in advance with multiple parallel data sets composed of utterance pairs of the source and many pre-stored target speakers. The EV-GMM is effectively adapted to an arbitrary target speaker using a small amount of adaptation data. Although this system achieves the very flexible training of the conversion model, the quality of the converted speech is still not high enough. In order to alleviate this problem, we simultaneously apply the following promising techniques to the one-to-many EVC system: 1) STRAIGHT mixed excitation, 2) the conversion algorithm considering global variance, and 3) speaker adaptive training of the EV-GMM. Experimental results demonstrate that the proposed system causes remarkable improvements in the performance of EVC.

We investigate a novel method for speaker verification with nonaudible murmur (NAM) segments. NAM is recorded using a special microphone placed on the neck and is hard for other people to hear. We have already reported a method based on a support vector machine (SVM) using NAM segments to use a keyword phrase effectively. To further exploit keyword-specific features, we introduce a global alignment (GA) kernel and penalized logistic regression machine (PLRM). In the experiments using NAM from 55 speakers, our method achieved an error reduction rate of roughly 60% compared with the SVM-based method using a polynomial kernel.

In this paper, we describe development and evaluation of handsfree spoken dialogue system which is used for railway station guidance. In the application at the railway station, noise robustness is the most essential issue for the dialogue system. To address the problem, we introduce two key techniques in our proposed hands-free system; (a) blind spatial subtraction array (BSSA) as a preprocessing, which can efficiently reduce nonstationary and diffuse noises in real-time, and (b) robust voice activity detection (VAD) based on speech decoding for further improvement of speech recognition accuracy. The experimental assessment of the proposed dialogue system reveals that the combination of real-time BSSA and robust VAD can provide the recognition accuracy of more than 80% under adverse railway-station noise conditions.

The aim of this research is a human-oriented spoken dialog system which provides replies to a variety of users' utterances. The example-based response generation method searches a question and answer database (QADB) for the example question most similar to a user utterance. With this method, the system can answer a question difficult for a model to express. A QADB is constructed from question and answer pairs (QA pairs) by employing a large corpus. In order to enhance robustness to recognition errors of inarticulate utterances such as children utterances, we propose to use speech recognition results, instead of manual transcriptions, as example questions. We also introduce an optimization method that removes inappropriate QA pairs from a QADB to maximize response accuracy. We show that our method improves the response accuracy of utterances especially for children utterances in the open test.

Many-to-one eigenvoice conversion (EVC) allows the conversion from an arbitrary speaker's voice into the pre-determined target speaker's voice. In this method, a canonical eigenvoice Gaussian mixture model is effectively adapted to any source speaker using only a few utterances as the adaptation data. In this paper, we propose a many-to-one EVC based on maximum a posteriori (MAP) adaptation for further improving the robustness of the adaptation process to the amount of adaptation data. Results of objective and subjective evaluations demonstrate that the proposed method is the most effective among the other conventional many-to-one VC methods when using any amount of adaptation data (e.g., from 300 ms to 16 utterances).

The model of the human/machine hands-free speech interface is defined as a point source (the user voice) and a diffuse background noise. This situation is very different from the usual cocktail party model, separation of a mixture of speeches, that is usually treated in frequency domain blind signal separation (FD-BSS). In particular, the fast permutation solvers proposed for the cocktail party model results in poor separation performance in this case. In order to resolve the permutation more efficiently, this paper proposes a new approach that exploits the statistical discrepancy between the target speech and the diffuse background noise.

In this paper, we construct a hands-free robot spoken dialogue system based on the real-time blind spatial subtraction array (BSSA) and evaluate the system. BSSA is the blind source extraction method, and the source extraction in BSSA is carried out by subtracting the power spectrum of the estimated noise signal by the independent component analysis from the power spectrum of the target speech partly enhanced signal. Although BSSA can reduce noise signal efficiently, ICA consumes huge amount of computational costs. Thus it is difficult to run BSSA in real-time. In this paper, we newly propose a real-time architecture of BSSA and construct a hands-free robot spoken dialogue system based on the real-time BSSA. In the hands-free robot spoken dialogue system with the real-time BSSA, 6% improvement of the speech recognition result can be seen compared with the conventional speech enhancement methods.

In this paper, we investigate development and portability of ASR and Q&A modules of speech-oriented guidance systems for two different real environments. An initial prototype system has been constructed for a local community center using two years of human-labeled data collected by the system. Collection of real user data is required because ASR task and Q&A domain of a guidance system are defined by the target environment and potential users. However, since human preparation of data is always costly, most often only a relatively small amount real data will be available for system adaptation in practice. Therefore, the portability of the initial prototype system is investigated for a different environment, a local subway station. The purpose is to identify reusable system parts. The ASR module is found to be highly portable across the two environments. However, the portability of the Q&A module was only medium. From an objective analysis it became clear that this is mainly due to the environment-dependent domain differences between the two systems. This implicates that it will always be important to take the behavior of actual users under real conditions into account to build a system with high user satisfaction.

This paper presents insights gained from operating a public speech-oriented guidance system. A real-environment speech database (300 hours) collected with the system over four years is described and analyzed regarding usage frequency, content and diversity. Having the first two years of the data completely transcribed, simulation of system development and evaluation of system performance over time is possible. The database is employed for acoustic and language modeling as well as construction of a question and answer database. Since the system input is not text but speech, the database enables also research on open-domain speech-based information access. Apart from that research on unsupervised acoustic modeling, language modeling and system portability can be carried out. A performance evaluation of the system in an early stage as well as late stage when using two years of real-environment data for constructing all system components shows the relative importance of developing each system component. The system's response accuracy is 83% for adults and 68% for children.

Real-time two-stage blind source separation (BSS) method for convolutive mixtures of speech is now being studied by the authors, in which a single-input multiple-output (SIMO)-model-based independent component analysis (ICA) and a SIMO-model-based binary masking are combined. In addition, we have developed a pocket-size real-time DSP module implementing the two-stage BSS method. In this paper, we introduce a high-presence hearing-aid system which can reduce the interference sound and reproduce the target sound while keeping the directivity, and realize the system with the real-time BSS module. To evaluate it, we carried out the objective and subjective experiments using 9 users. From these results, it is revealed that the decomposition performance and the directivity maintenance of the proposed system are superior to those of conventional methods.

In this paper, first, we propose a computational-cost efficient blind source separation combining closed-form 2nd-order independent component analysis (ICA) and nonclosed-form higher-order ICA. The closed-form solution of the 2nd-order ICA has been recently presented by one of the authors. This finding motivates us to combine the closed-form 2nd-order ICA and higher-order ICA, where the preceding closed-form ICA produces a good initial value and the following higher-order ICA updates the separation filters from the advantageous status. Secondly, we utilize the proposed architecture to address an essential question that which type of statistics is more beneficial to ICA among non-stationarity and non-Gaussianity. This can be conducted owing to the attractive property that the closed-form ICA can provide a good estimate of the theoretical upper limitation of the separation performance among 2nd-order ICAs without suffering from poor-convergence problems. Experimental results reveal that the non-Gaussianity-based ICA can outperform the non-stationarity-based ICA.

In this paper, we investigate a new blind source separation (BSS) structure from a permutation-robustness viewpoint, to mitigate the permutation problem which commonly arises in frequency-domain independent component analysis (ICA). Permutation robustness means that how much the BSS method is not affected under a certain probability of arising permutation, unlike the conventional permutation-solving approaches. We address to analyze our previously proposed BSS architecture, so called blind spatial subtraction array (BSSA). In BSSA, source extraction is achieved by subtracting the power spectrum of the estimated noise via ICA from the power spectrum of partly speech-enhanced signal via delay-and-sum (DS) procedure. Indeed BSSA partially involves permutation problem in the ICA-based noise estimator part. However, BSSA can efficiently reduce the negative affection of the permutation owing to the over-subtraction in the spectral subtraction and defocusing properties in DS. Experiments using artificial and real-recording-based simulations reveal that the proposed method outperforms the conventional ICA.

This paper describes two flexible frameworks of voice conversion (VC), i.e., one-to-many VC and many-to-one VC. One-to-many VC realizes the conversion from a user's voice as a source to arbitrary target speakers' ones and many-to-one VC realizes the conversion vice versa. We apply eigenvoice conversion (EVC) to both VC frameworks. Using multiple parallel data sets consisting of utterance-pairs of the user and multiple pre-stored speakers, an eigenvoice Gaussian mixture model (EV-GMM) is trained in advance. Unsupervised adaptation of the EV-GMM is available to construct the conversion model for arbitrary target speakers in one-to-many VC or arbitrary source speakers in many-to-one VC using only a small amount of their speech data. Results of various experimental evaluations demonstrate the effectiveness of the proposed VC frameworks.

INTERSPEECH2007: 8th Annual Conference of the International Speech Communication Association, August 27-31, 2007, Antwerp, Belgium.

INTERSPEECH2007: 8th Annual Conference of the International Speech Communication Association, August 27-31, 2007, Antwerp, Belgium.

In order to develop a target speech recognition system with less cost of time and money, reusability of existing speech corpora is becoming one of the most important issues. This paper proposes a new technique to judge the reusability of existing speech corpora for a target task by utilizing a statistical multidimensional scaling method. In an experiment using twelve tasks in five speech corpora, our proposed method could show high correlation to the cross task recognition performance and judge the reusability of existing speech corpora correctly for the target task with lower cost.

We proposed a speaking aid system using statistical voice conversion for laryngectomees, whose vocal folds have been removed. This paper investigates the influence of various small sound sources on the voice conversion accuracy. Spectral envelopes and power of sound sources are controlled independently. In total 8 different kinds of sound source signals, e.g. pulse train, sierra wave and so on, are examined. Results of objective and subjective evaluations demonstrate that for voice conversion, sound sources with various spectral envelopes and power in a large degree are acceptable unless the power of them is comparable to that of silence parts.

One-to-many eigenvoice conversion (EVC) allows the conversion of a specific source speaker into arbitrary target speakers. Eigenvoice Gaussian mixture model (EV-GMM) is trained in advance with multiple parallel data sets consisting of the source speaker and many pre-stored target speakers. The EV-GMM is adapted for arbitrary target speakers using only a few utterances by estimating a small number of free parameters. Therefore, the initial EV-GMM directly affects the conversion performance of the adapted EV-GMM. In order to prepare a better initial model, this paper proposes Speaker Adaptive Training (SAT) of a canonical EV-GMM in one-to-many EVC. Results of objective and subjective evaluations demonstrate that SAT causes significant improvements in the performance of EVC.

We investigated a speaker verification method that uses non-audible murmur (NAM) segments using newly collected data and obtained several findings that will be useful when speaker verification systems are made in practice. NAM is recorded using a special microphone placed on the surface of the body, so it includes almost no external noise and is hard for other people to hear. By utilizing these properties, we have already reported a text-dependent method using NAM segments that can use a keyword phrase safely. This paper extends the examination with newly collected data consisting of NAM uttered by 18 male and 9 female imposter speakers and by 18 male and 10 female customer speakers. Experiments with various numbers of training utterances and sessions show that it is effective to use data recorded in multiple sessions. We also investigated the minimum number of training utterances needed in our method.

To expand CSR (continuous speech recognition) software to the mobile environmental use, we have developed embedded version of Julius (embedded Julius). Julius is open source CSR software, and has been used by many researchers and developers in Japan as a standard decoder on PCs. In this paper, we describe an implementation of the embedded Julius on a SH-4A microprocessor. SH-4A is a high-end 32-bit MPU (720 MIPS) with on-chip FPU. However, further computational reduction is necessary for the embedded Julius to operate realtime. Applying some optimizations, the embedded Julius achieves real-time processing on the SH-4A. The experimental results show 0.89 times RT(real-time), resulting 4.0 times faster than baseline CSR. We also evaluated the embedded Julius on large vocabulary (20,000 words). It shows almost real-time processing (1.25 times RT).

In real-time speech recognition applications, there is a need to implement a fast and reliable adaptation algorithm. We propose a method to reduce adaptation time of the unsupervised speaker adaptation based on HMM-sufficient statistics. We use only a single arbitrary utterance without transcriptions in selecting the N-best speakers' sufficient statistics created offline to provide data for adaptation to a target speaker. Further reduction of N-best implies a reduction in adaptation time. However, it degrades recognition performance due to insufficiency of data needed to robustly adapt the model. Linear interpolation of the global HMM-sufficient statistics offsets this negative effect and achieves a 50% reduction in adaptation time without compromising the recognition performance. We have reduced the adaptation time from 10 sec to 5 sec without degradation of the word accuracy. Furthermore, we compared our method with vocal tract length normalization (VTLN), maximum a posteriori (MAP) and maximum likelihood linear regression (MLLR). Moreover, we tested in office, car, crowd and booth noise environments in 10 dB, 15 dB, 20 dB and 25 dB SNRs

In this paper we introduce a new double-talk free spoken dialogue interface combining sound field control and a source separation technique based on independent component analysis (ICA). First, sound field control provides silent zones on the microphone elements and prevents the response sound from being observed. In the second step, we propose a novel semi-blind source separation algorithm to suppress the error caused by fluctuation of the room transfer function. By using a direct input of response sound signal to ICA, a source separation problem can be converted to a supervised learning problem. Since the problem becomes easier, the proposed method showed higher performances than the method using blind source separation

INTERSPEECH2006: the 9th International Conference on Spoken Language Processing (ICSLP), September 17-21, 2006, Pittsburgh, Pennsylvania, USA.

We propose a speaker verification method using non-audible murmur (NAM) segments, which are different from normal speech and hard for other people to catch them. To use NAM, we therefore take a text-dependent verification strategy in which each user utters her/his own keyword phrase and utilize not only speaker-specific but also keyword-specific acoustic information. We expect this strategy to yield a relatively high performance. NAM segments, which consist of multiple short-term feature vectors, are used as input vectors to capture keyword-specific acoustic information well. To handle segments with a large number of dimensions, we use the support vector machine (SVM). In experiments using NAM data of 19 male and 10 female speakers recorded in three different sessions, we achieved equal error rates of 0.04% (male) and 1.1% (female) when using 145-ms-long NAM segments. These rates are half or less those obtained with 25-ms-long input vectors.

The conversion method from Non-Audible Murmur (NAM) to ordinary speech based on the statistical voice conversion (NAM-to-Speech) has been proposed towards realization of "silent speech telephone." Although NAM-to-Speech converts NAM to intelligible voices with similar quality to speech, there is still a large problem, i.e., difficulties of the F0 estimation from unvoiced speech. In order to avoid this problem, we propose a conversion method from NAM to whisper that is a familiar and intelligible unvoiced speech (NAM-to-Whisper). Moreover, we enhance NAM-to-Whisper so that multiple types of body-transmitted unvoiced speech such as NAM and Body Transmitted Whisper (BTW) are accepted as input voices. We evaluate the performance of the proposed conversion method. Experimental results demonstrate that 1) intelligibility and naturalness of NAM are significantly improved by NAM-to-Whisper, 2) NAM-to-Whisper outperforms NAM-to-Speech, and 3) we can train a single conversion model successfully converting both NAM and BTW to the target voice.

The aim of this paper is to improve the naturalness of speech using a medical device such as an electrolarynx. There are several problems associated with using existing electrolarynxes, such as the fact the loud volume of the electrolarynx itself might disturb smooth interpersonal communication, and that the generated speech is unnatural. We propose a novel speaking-aid system for total laryngectomees using a new sound source as an alternative to the existing electrolarynx and a statistical voice-conversion technique. The new sound-source unit outputs extremely small signals that cannot be heard by people around the speaker. Artificial speech is recorded with a NAM microphone through soft tissues of the head. From the result of voice conversion, the body-transmitted artificial speech is consistently converted to a more natural voice. We also demonstrate that the speech recognition performance of the proposed system substantially increases in terms of objective evaluation.

The performance of voice conversion has been considerably improved through statistical modeling of spectral sequences. However, the converted speech still contains traces of artificial sounds. To alleviate this, it is necessary to statistically model a source sequence as well as a spectral sequence. In this paper, we introduce STRAIGHT mixed excitation to a framework of the voice conversion based on a Gaussian Mixture Model (GMM) on joint probability density of source and target features. We convert both spectral and source feature sequences based on Maximum Likelihood Estimation (MLE). Objective and subjective evaluation results demonstrate that the proposed source conversion produces strong improvements in both the converted speech quality and the conversion accuracy for speaker individuality.

This paper describes a novel framework of voice conversion (VC). We call it eigenvoice conversion (EVC). We apply EVC to the conversion from a source speaker's voice to arbitrary target speakers' voices. Using multiple parallel data sets consisting of utterance-pairs of the source and multiple pre-stored target speakers, a canonical eigenvoice GMM (EV-GMM) is trained in advance. That conversion model enables us to flexibly control the speaker individuality of the converted speech by manually setting weight parameters. In addition, the optimum weight set for a specific target speaker is estimated using only speech data of the target speaker without any linguistic restrictions. We evaluate the performance of EVC by a spectral distortion measure. Experimental results demonstrate that EVC works very well even if we use only a few utterances of the target speaker for the weight estimation.

To expand CSR (continuous speech recognition) software to the mobile environmental use, we have developed embedded version of "Julius". Julius is open source CSR software, and has been used by many researchers and developers in Japan as a standard decoder on PCs. Julius works as a real time decoder on a PC. However further computational reduction is necessary to use Julius on a microprocessor. Further cost reduction is needed. For reducing cost of calculating pdfs (probability density function), Julius adopts a GMS (Gaussian mixture selection) method. In this paper, we modify the GMS method to realize a continuous speech recognizer on microprocessors. This approach does not change the structure of acoustic models in consistency with that used by conventional Julius, and enables developers to use acoustic models developed by popular modeling tools. On simulation, the proposed method has archived 20% reduction of computational costs compared to conventional GMS, 40% reduction compared to no GMS. Finally, the embedded version of Julius was tested on a developmental hardware platform named "T-engine". The proposed method showed 2.23 of RTF (real time factor) resulting 79% of that of no GMS without any degradation of recognition performance

We propose a new algorithm for blind source separation (BSS), in which independent component analysis (ICA) and beamforming are combined to resolve the slow-convergence problem through optimization in ICA. The proposed method consists of the following three parts: (a) frequency-domain ICA with direction-of-arrival (DOA) estimation, (b) null beamforming based on the estimated DOA, and (c) integration of (a) and (b) based on the algorithm diversity in both iteration and frequency domain. The unmixing matrix obtained by ICA is temporally substituted by the matrix based on null beamforming through iterative optimization, and the temporal alternation between ICA and beamforming can realize fast- and high-convergence optimization. The results of the signal separation experiments reveal that the signal separation performance of the proposed algorithm is superior to that of the conventional ICA-based BSS method, even under reverberant conditions.

INTERSPEECH2005: the 9th European Conference on Speech Communication and technology, September 4-8, 2005, Lisbon, Portugal.

In this paper, we report non-audible murmur (NAM) recognition results in noisy environments and investigate the effect of the Lombard reflex on non-audible murmur recognition. Non-Audible murmur is speech uttered very quietly and captured through body tissue by a special acoustic sensor (e.g., NAM microphone). A system based on non-audible murmur recognition can be applied in cases when privacy is preferable in human-machine communication. Moreover, due to direct body-transmission, the environmental noises do not affect the performance markedly. Previously, we reported non-audible murmur automatic recognition in a clean environment with very promising results. We also carried out experiments using clean models and simulated noisy data, showing that the performance did not change significantly. Using, however, real noisy test data, the performance decreased markedly. To investigate this problem, we studied the Lombard reflex and conducted non-audible murmur recognition experiments using Lombard data. Results show, that Lombard reflex affects non-audible murmur recognition.

In this paper, we present the use of stethoscope and silicon NAM microphones in automatic speech recognition. NAM microphones are special acoustic sensors, which are attached behind the talker's ear and can capture not only normal (audible) speech, but also very quietly uttered speech (non-audible murmur). As a result, NAM microphones can be applied in automatic speech recognition systems when privacy is desired. Previously, we presented speech recognition experiments for non-audible murmur captured by a stethoscope microphone. In this paper, we also present recognition results using a more advanced NAM microphone, the so-called silicon NAM microphone. Using adaptation techniques and a small amount of training data, we achieved a 93.9% word accuracy for non-audible murmur recognition. We also report experimental results in noisy environments showing the effectiveness of using a NAM microphone in noisy environments. In addition to a dictation task, we also present a keyword spotting experiment based on non-audible murmur.

We developed the next generation of skin-attachment sensors for sampling NAM (Non-Audible Murmur) signals by using soft silicone, which has an acoustic impedance close to that of human flesh, as the prime medium of vibration. With new NAM microphones we could sample expanded target voice signal, suppressing air conduction noise signal to low by the experiment of synchronous stereo sampling of air and flesh conduction voices at the same gain. The bandwidth of the soft silicone type NAM microphone has improved and we obtain a much higher accuracy of both NAM and BTOS (Body Transmitted Ordinary Speech) recognition compared with the stethoscopic type. Aural comprehension test showed that accuracy of catching sentences of NAM and BTOS has improved with soft silicon type NAM microphone almost as high as that of air conduction voices. However, the extremely low accuracy of meaningless words is a problem to be solved for developing a "Non-Voice Phone".

Takemaru-kun system is a practical speech-oriented guidance system developed to examine spoken interface through long-term operation in a public place that collected natural human-machine interaction data. In (2)004 the following advances improving reliability of the system were introduced, which conduced acquiring positive increase of access from users: (1) Rejection of unintended speech based on Gaussian Mixture Models (GMMs); (2) Removal of short, unnecessary inputs of impulsive noise; (3) Child or adult user discrimination; (4) Web-based monitoring mechanisms. This paper summarizes the Takemaru-kun system and analysis of 177,789 data collected by two-years actual operation. Experiments with the collected data proved that a combination of GMM-based verification and short input removal can excise 85% of the invalid inputs, including laughter, incomprehensible utterances, and even some background utterances.

In order to realize a new human communication style using Non-Audible Murmur (NAM) that cannot be heard by people around a speaker, we perform conversion from NAM to ordinary speech (NAM-to-Speech). NAM-to-Speech has a possibility of realizing “non-speech telephone ” that is a technique for communicating each other by talking in NAM and hearing in speech. In this paper, we apply a statistical conversion method with Gaussian Mixture Model (GMM) to NAM-to-Speech. In advance, we train GMMs for representing correlations between acoustic features of NAM and those of speech using 50 utterance pairs of NAM and speech. In the conversion, we estimate acoustic spectral and F0 features of speech based on a maximum likelihood criterion, and then synthesize the converted speech with a vocoder. From results of subjective evaluations on intelligibility and naturalness, it is demonstrated that the NAMto-Speech with GMMs can convert NAM to more consistently natural voice. 1.

We propose a spatial subtraction array (SSA) and known noise superimposition to achieve a noise-robust hands-free speech recognition which can be used in human-robot interaction. In the proposed SSA, noise reduction is achieved by subtracting the estimated noise power spectrum from the target speech power spectrum to be enhanced in the mel-scale filter bank domain. This offers a realization of error-robust spatial spectral subtraction with few computational complexities. In addition, we introduce known noise superimposition technique in the mel-scale filter bank domain, and utilize the matched acoustic model for the known noise. This can compensate the acoustic model mismatch and mask the residual noise component in SSA. The experimental results obtained under a real environment reveal that word accuracy of the proposed method is greater than that of the conventional method even when the target user moves between -10 and +10 degrees around the microphone array.

We newly propose a real-time two-stage blind source separation (BSS) for binaural mixed signals observed at the ears of humanoid robot, in which a single-input multiple-output (SIMO)-model-based independent component analysis (ICA) and binary mask processing are combined. SIMO-model-based ICA can separate the mixed signals, not into monaural source signals but into SIMO-model-based signals from independent sources as they are at the microphones. Thus, the separated signals of SIMO-model-based ICA can maintain the spatial qualities of each sound source, and this yields that binary mask processing can be applied to efficiently remove the residual interference components after SIMO-model-based ICA. The experimental results obtained with a human-like head reveal that the separation performance can be considerably improved by using the proposed method in comparison to the conventional ICA-based and binary-mask-based BSS methods.

In this paper, we address a blind decomposition problem of binaural mixed signals observed at the ears of humanoid robot, and we introduce a novel blind signal decomposition algorithm using single-input multiple-output-model-based ICA (SIMO-ICA). The SIMO-ICA consists of multiple ICAs and a fidelity controller, and each ICA runs in parallel under the fidelity control of the entire separation system. The SIMO-ICA can separate the mixed signals, not into monaural source signals but into SIMO-model-based signals from independent sources as they are at the microphones in the robot ear. Thus, the separated signals of SIMO-ICA can maintain the spatial qualities of each sound source, i.e., they represent the decomposed sound scenes. Obviously the attractive feature of SIMO-ICA is highly applicable to not only speech recognition but also, e.g., humanoid-robot-based auditory tele-existence technology. The experimental results reveal that the spatial quality of the separated sound in SIMO-ICA is remarkably superior to that of the conventional method, particularly for the fidelity of the sound reproduction.

Confidence scoring based on word posterior probability is usually performed as a post process of speech recognition decoding, and also needs a large number of word hypotheses to get enough confidence quality. We propose a simple way of computing the word confidence using estimated posterior probability while decoding. At the word expansion of stack decoding search, the local sentence likelihoods that contain heuristic scores of unreached segment are directly used to compute the posterior probabilities. Experimental results showed that, although the likelihoods are not optimal, we can provide slightly better confidence measures compared with N-best lists, while the computation is faster than the 100-best method because no N-best decoding is required.

We propose a new algorithm for overdetermined blind source separation based on multistage independent component analysis (MSICA). To improve the separation performance, we have proposed MSICA in which frequency-domain ICA and time-domain ICA are cascaded. In the original MSICA, the specific mixing model, where the number of microphones is equal to that of sources, was assumed. However, additional microphones are required to achieve an improved separation performance under reverberant environments. This leads to alternative problems, e.g., a complication of the permutation problem. In order to solve them, we propose a new extended MSICA using subarray processing, where the number of microphones and that of sources are set to be the same in every subarray. The experimental results obtained under the real environment reveal that the separation performance of the proposed MSICA is improved as the number of microphones is increased.

The Takemaru-kun system is a real world speech-oriented guidance system located at the Ikoma-City North Community Center. The system has been operated daily from November, 2002, to provide visitors a speech interface for information retrieval. This system also aims at the field test of a speech interface and collecting actual utterance data. By analyzing and evaluating the collected utterances, the flexible processing requirements are discovered according to the user's age group. It becomes impossible to disregard the increase of child users when the system is installed in a public place. The paper proposes an automatic approach discriminating speakers between adult and child users, which is based on statistical learning. This proposal realizes a flexible spoken dialogue to both adult and child users. As for parameter vectors in machine learning, acoustic and linguistic properties extracted from speech recognition logarithm likelihood scores are adopted to discriminate a user's age group. Although GMM-based recognition uses only acoustic properties, this method can also consider linguistic properties. In experiments with SVM-based screening, we obtained a 92.4% discrimination rate to the actual users' utterances. The advantage of using linguistic properties is also shown. The paper also describes an overview of the Takemaru-kun system and the data collection status from the field test. Child speech recognition performance is evaluated using the collected utterances.

High-fidelity blind audio signal separation is addressed, adopting the extended ICA algorithm, single-input multiple-output (SIMO)-model-based ICA. The SIMO-ICA consists of multiple ICA parts and a fidelity controller, and each ICA runs in parallel under fidelity control of the entire separation system. SIMO-ICA can separate the mixed signals, not into monaural source signals, but into SIMO-model-based signals from independent sources as they are at the microphones. Thus, the separated signals of the SIMO-ICA can maintain the spatial qualities of each sound source. We apply the SIMO-ICA to the problem of blind separation of mixed binaural sounds, including the effect of the head-related transfer function (HRTF). Experimental results reveal that the performance of the proposed SIMO-ICA is superior to that of the conventional ICA-based method, and the separated signals of SIMO-ICA maintain the spatial qualities of each sound source.

This paper describes a new interface for a barge-in free spoken dialogue system combining an adaptive sound field control and a microphone array. In order to actualize robustness against the change of transfer functions due to the various interferences, the barge-in free spoken dialogue system which uses sound field control and a microphone array has been proposed by one of the authors. However, this method cannot follow the large change of transfer functions. To solve the problem, we introduce a new adaptive sound field control that follows the change of transfer functions. The experimental results reveal that the proposed method can improve the reduction accuracy of response sound in comparison with the conventional acoustic echo canceller as well as the previously proposed method which simply uses fixed sound field control system.

ICSLP2004: the 8th International Conference on Spoken Language Processing, October 4-8, 2004, Jeju Island, Korea.

In this paper, we introduce the Stethoscopic Non-Audible Murmur (NAM) microphone, and we focus on its application in automatic speech recognition systems. The NAM microphone is attached behind the talker's ear, and can capture very quietly uttered murmur (NAM speech). It is applicable in automatic speech recognition systems, when privacy is important in human-machine communication. Moreover, since the NAM microphone receives the speech signal directly from the body, it shows robustness against the environmental noises. In addition to these, it might be also used in special systems (speech recognition, speech transform, etc.) for sound-impaired people. By applying adaptation techniques, we performed automatic speech recognition experiments for NAM speech. Using Maximum A Posteriori (MAP) adaptation, and a combination with Maximum Likelihood Linear Regression (MLLR) adaptation we achieved for a 20k vocabulary dictation system a 93.5% word accuracy, which is a very promising result.

Continuous Speech Recognition Consortium (CSRC) was founded for further enhancement of Japanese Dictation Toolkit that had been developed by the support of a Japanese agency. Overview of its product software is reported in this paper. The open-source LVCSR (large vocabulary continuous speech recognition) engine Julius has been improved both in performance and functionality, and it is also ported to Microsoft Windows in compliance with SAPI (Speech API). The software is now used for not a few languages and plenty of applications. For plug-and-play speech recognition in various applications, we have also compiled a repository of acoustic and language models for Japanese. Especially, the acoustic model set realizes wider coverage of user generations and speech-input environments.

ICSLP2004: the 8th International Conference on Spoken Language Processing, October 4-8, 2004, Jeju Island, Korea.

Recently, speech recognition technique has started being used in various products. In order to make a good acoustic model, usually a lot of training speech data is needed. However, due to the right of voice and privacy issues, it is not easy to collect a lot of training data. Statistical models which have been trained and transformed from speech data do not have the above mentioned problem, and are comparatively easy to obtain. Therefore, a training technique which can make an acoustic model by using statistical models as training data is preferred. This paper describes a new method called Rapid EM Training based on Model-Integration and shows that the proposed method is evaluated positively.

In this paper, a barge-in free spoken dialogue system using sound field control and microphone array is proposed. In the conventional spoken dialogue system using an acoustic echo canceller, it is indispensable to estimate and update the room transfer function, especially when the transfer function is changed by various interferences. However the estimation process for the transfer function prevents the user from speaking freely and simultaneously with speech responses from the system. In order to resolve the problem, we have already proposed a barge-in free spoken dialogue system that controls a sound field using multiple loudspeakers. In this paper, a microphone array for acquisition of user's speech is newly introduced in the previously proposed system. By introducing the microphone array, we can reduce the number of loudspeakers to be required in the system, and make the interface for spoken dialogue system more robust against the change of room transfer functions.

We propose a new style of practical input interface for the recognition of non-audible murmur (NAM), i.e., for the recognition of inaudible speech produced without vibration of the vocal folds. We have developed a microphone attachment, which adheres to the skin, applying the principle of a medical stethoscope, found the ideal position for sampling flesh-conducted NAM sound vibration and retrained an acoustic model with NAM samples. Then, using the Julius Japanese Dictation Toolkit, we tested the possibilities for practical use of this method in place of an external microphone for analyzing air-conducted voice sound.

In this paper, we investigate the effect of using a novel cost, RMS (root mean square) cost, for segment selection for concatenative text-to-speech synthesis. The RMS cost is affected not only by the total degradation of naturalness but also by the local degradation of naturalness. From the results of experiments comparing this approach with segment selection based on a conventional average cost, it is found that: (1) in the segment selection based on the RMS cost a larger number of concatenations causing slight local degradation are performed in order to avoid concatenations causing greater local degradation; and (2) the effect of the RMS cost has little dependence on the size of the corpus. Moreover, we clarify that the naturalness of synthetic speech can be slightly improved by utilizing the RMS cost.

We propose a two-stage blind separation and deconvolution (BSD) algorithm for a convolutive mixture of temporally correlated signals, in which a new single-input multiple-output (SIMO)-model-based ICA (SIMO-ICA) and blind multichannel inverse filtering are combined. SIMO-ICA consists of multiple ICAs and a fidelity controller, and each ICA runs in parallel under fidelity control of the entire separation system. SIMO-ICA can separate the mixed signals, not into monaural source signals but into SIMO-model-based signals from independent sources as they are at the microphones. After the separation by SIMO-ICA, a simple blind deconvolution technique based on multichannel inverse filtering for the SIMO model can be applied even when the mixing system is the nonminimum phase system and each source signal is temporally correlated. The experimental results obtained under the reverberant condition reveal that the sound quality of the separated signals in the proposed method is superior to that in the conventional ICA-based BSD.

In this paper, we present a beamforming based semi-blind source separation technique, which can be applied efficiently for hands-free speech recognition of multiple talkers (including moving talkers, too). The main difference from the conventional blind source separation techniques lies in the fact that the proposed method does not attempt to separate explicitly the unknown signals in a pre-processing pass before speech recognition. In fact, localization of multiple talkers, separation of the signals, and speech recognition are integrated in a single pass. Each time frame, beams formed by a delay-and-sum beamformer are steered to every direction, and speech information is extracted. A modified Viterbi formula provides n-best hypotheses for each direction and word hypotheses. At the final frame, all hypotheses are clustered based on their direction information. The clusters, which correspond to the talkers include information about the recognized speech of the multiple talkers and about their direction. Experiments for recognition of two and three talkers showed very promising results. In the case of two talkers, and using simulated clean data we achieved for `top 5' hypotheses a recognition rate of series 95.02% on average, which is very promising result.

Voice conversion method is applied to synthesizing emotional speech from standard reading (neutral) speech. Pairs of neutral speech and emotional speech are used for conversion rule training. The conversion adopts GMM (Gaussian Mixture Model) with DFW (Dynamic Frequency Warping). We also adopt STRAIGHT, the high-quality speech analysis-synthesis algorithm. As conversion target emotions, (Hot) anger, (cold) sadness and (hot) happiness are used. The converted speech is evaluated objectively first using mel cepstrum distortion as a criterion. The result confirms the GMM-based voice conversion can reduce distortion between target speech and neutral speech. A subjective test is also carried out to investigate perceptual effect. From the viewpoint of influence of prosody, two kinds of prosody are used to synthesis. One is natural prosody extracted from neutral speech and the other is from emotional speech. The result shows that prosody mainly contribute to emotion and spectrum conversion can reinforce it. 1.

We propose a new style of practical input interface for the recognition of non-audible murmur (NAM), i.e., for the recognition of inaudible speech produced without vibration of the vocal folds. We developed a microphone attachment, which adheres to the skin, applying the principle of a medical stethoscope, found the ideal position for sampling flesh-conducted NAM sound vibration and retrained an acoustic model with NAM samples. Then using the Julius Japanese Dictation Toolkit, we tested the possibilities for practical use of this method in place of an external microphone for analyzing air-conducted voice sound. Additionally we propose laryngeal elevation index (LEI), a new index of prosody, which can show the prosody of NAM without F0, using simple processing of images from medical ultrasonography. We realized and defined NAM never used for input or communication and propose that we should make use of it for the interface of human-human and human-cybernetic machines.

In real acoustic environments, humans communicate with each other through speech by focusing on the target speech among environmental sounds. We can easily identify the target sound from other environmental sounds. For hands-free speech recognition, the identification of the target speech from environmental sounds is imperative. This mechanism may also be important for a self-moving robot to sense the acoustic environments and communicate with humans. Therefore, this paper first proposes Hidden Markov Model (HMM)-based environmental sound source identification. Environmental sounds are modeled by three states of HMMs and evaluated using 92 kinds of environmental sounds. The identification accuracy was 95.4%. This paper also proposes a new HMM composition method that composes speech HMMs and an HMM of categorized environmental sounds for robust environmental sound-added speech recognition. As a result of the evaluation experiments, we confirmed that the proposed HMM composition outperforms the conventional HMM composition with speech HMMs and a noise (environmental sound) HMM trained using noise periods prior to the target speech in a captured signal.

EUROSPEECH2003: 8th European Conference on Speech Communication and Technology, September 1-4, 2003, Geneva, Switzerland.

Noise and speaker adaptation techniques are essential to realize robust speech recognition in noisy environments. In this paper, first, a noise robust speech recognition algorithm is implemented by superimposing a small quantity of noise data on spectral subtracted input speech. According to the recognition experiments, 30dB SNR noise superimposition on input speech after spectral subtraction increases the robustness against different noises significantly. Next, we apply this noise robust speech recognition to the unsupervised speaker adaptation algorithm based on HMM sufficient statistics in different noise environments. The HMM sufficient statistics for each speaker are calculated from 25dB SNR office noise added speech database beforehand. We evaluate successfully our proposed unsupervised speaker adaptation algorithm in noisy environments with 20k dictation task using 11 kinds of different noises, including office, car, exhibition, and crowd noises.

We propose a new two-stage blind separation and deconvolution (BSD) algorithm for a convolutive mixture of speech, in which a new Single-Input Multiple-Output (SIMO)-model-based ICA (SIMO-ICA) and blind multichannel inverse filtering are combined. SIMO-ICA can separate the mixed signals, not into monaural source signals but into SIMO-model-based signals from independent sources as they are at the microphones. After SIMO-ICA, a simple blind deconvolution technique for the SIMO model can be applied even when each source signal is temporally correlated. The simulation results reveal that the proposed method can successfully achieve the separation and deconvolution for a convolutive mixture of speech.

Speech recognition technology has been widely used. Considering a training cost of an acoustic model, it is beneficial to reuse pre-existing acoustic models for making a suitable one for various apparatus and application. However, a complex acoustic model for high CPU power does not work for low CPU power. And a simple model for fast-processing-demanded application does not work well for high-precision-demanded ones. Therefore, it is important to adjust a model complexity according to apparatus or application, such as a number of mixture of Gaussians. This paper describes a new model-integration-type of training for obtaining a required number of mixture of Gaussians. This training can alter a number of mixture into a required one according to a specification of apparatus or application. We propose a model integration rapid training based on maximum likelihood, and evaluate the recognition performance successfully.

We propose a new algorithm for blind source separation (BSS), in which frequency-domain independent component analysis (FDICA) and time-domain ICA (TDICA) are combined to achieve a superior source-separation performance under reverberant conditions. Generally speaking, the conventional TDICA fails to separate source signals under heavily reverberant conditions because of the low convergence in the iterative learning of the inverse of the mixing system. On the other hand, the separation performance of the conventional FDICA under reverberant conditions also degrades significantly because the independence assumption of narrowband signals collapses when the number of subbands increases. In the proposed method, the separated signals of FDICA are regarded as the input signals for TDICA, and we can remove the residual cross-talk components of FDICA by using TDICA. The experimental results under the reverberant condition reveal that the signal-separation performance of the proposed method is superior to that of the conventional ICA-based BSS methods.

For a hands-free speech interface, it is very important to capture distant talking speech with high quality. A microphone array is an ideal candidate for this purpose. However, this approach requires localizing the target talker. Conventional talker localization algorithms in multiple sound source environments not only have difficulty localizing the multiple sound sources accurately, but also have difficulty localizing the target talker among known multiple sound source positions. To cope with these problems, we propose a new talker localization algorithm consisting of two algorithms. One is DOA (Direction Of Arrival) estimation algorithm for multiple sound source localization based on CSP (Cross-power Spectrum Phase) coefficient addition method. The other is statistical sound source identification algorithm based on GMM (Gaussian Mixture Model) for localizing the target talker position among localized multiple sound sources. In this paper, we particularly focus on the talker localization performance based on the combination of these two algorithms with a microphone array.

This paper proposes a novel unit selection algorithm for Japanese Text-To-Speech (TTS) systems. Since Japanese syllables consist of CV (C: Consonant, V: Vowel) or V, except when a vowel is devoiced, CV units are basic to concatenative TTS systems for Japanese. However, speech synthesized with CV units sometimes have discontinuities due to V-V concatenation; In order to alleviate such discontinuities, longer units (CV* or non-uniform units) have been proposed. However, the concatenation between V and V is still unavoidable. To address this problem, we propose a novel unit selection algorithm that incorporates not only phoneme units but also diphone units. The concatenation in the proposed algorithm is performed at the vowel center as well as at the phoneme boundary. Results of evaluation experiments clarify that the proposed algorithm outperforms the conventional algorithm.

The definition of "phoneme boundary timing" in a speech corpus affects the quality of concatenative speech synthesis systems. For example, if the selected speech unit is not appropriately match to the speech unit of the required phoneme environment, the quality may be degraded. In this paper, a dynamic segment boundary defi- nition is proposed. In the definition, the concatenation point is chosen from the start or end timings of spectral transition depending on the phoneme environment at the boundaries. For a listening test to compare the naturalness of conventional/proposed methods, 100 Japanese place names were selected randomly and synthesized. The ratio of naturalness was 1 to 3.3 (conventional v.s. proposed) by four subjects.

For the purpose of building Text-to-Speech (TTS) system that can generate high-quality and wide range speech in prosody, we conducted speech database construction. As a speech synthesizer, we use a hybrid system which consists of a unit selection module and prosody modification by STRAIGHT (vocoder type high quality analysis- synthesis method). Our viewpoint is to reduce an amount of prosody modification which causes quality deterioration. In other words, it is to generate any prosody at will within permissible prosody modification. Based on the aspect, we designed 9 sub-databases those consist of same phonetic balanced texts with different prosody. In this paper, we describe the designing policy and general features of the obtained database and the results of listening tests focused on the effectiveness about durational feature. They shows the advantage of the proposed database. and but it is also observed the necessity to change unit selection cost function according to output speech rate.

An efficient multi-path acoustic model based on database likelihoods for spontaneous speech recognition is presented. Although a multipath phone HMM that has several models for different target in parallel is considered effective to express multi-style or speed-variant nature of spontaneous speech, assuming various model to match at every time for all phones may cause mismatch of unintended mo- del, and spoil the model constraints. We propose defining a multipath model that has several different state resolutions only for the distortive phones selectively. The phone set is selected through an analysis of the likelihoods and duration times of phone segments in a spoken dialogue corpus using automatic viterbi alignment. Experiments on three test-sets showed that our multi-path model based on the phone selection can achieve better accuracy than a simple single-path model, whereas a full multi-path model without phone selection causes much degradation of accuracy.

Cross-language voice conversion is useful for many applications, and we are trying to apply the technique to a language training system for reducing voice individuality differences. In this paper, we describe experiments that test effectiveness of an extension of singlelanguage voice conversion, to include cross-language utterances. The performance was investigated by objective and perceptual evaluation using bilingual-speakers data for training. Then, the correlations between a computed distance measure and a human perceptual pronunciation evaluation score were compared before and after applying conversion. From these results, it was found that the crosslanguage voice conversion reduces speakers' voice differences between the pairs, and the phoneme based measures show somewhat clearer correspondences to the human perceptual score in vowels' test after applying voice conversion.

Recognition of distant-talking speech is indispensable for self-moving robots or teleconference systems. However, background noise and room reverberations seriously degrade the sound capture quality in real acoustic environments. A microphone array is an ideal candidate as an effective method for capturing distant-talking speech. AMNOR (Adaptive Microphone-array for NOise Reduction) was proposed an adaptive beamformer for capturing the desired distant signals in noisy environments by Kaneda et al. Although AMNOR has proven itself effective, it could be further improved if we knew the spectrum characteristics of desired distant signals in advance. Therefore, in this paper we regard speech as a desired distant signal and design AMNOR based on the average speech spectrum for distant-talking speech capture and recognition. As a result of evaluation experiments in real acoustic environments, we could confirm that the ASR (Automatic Speech Recognition) performance was improved 5 ~ 10% by AMNOR based on average speech spectrum in noisy environments.

We propose a new algorithm for blind source separation (BSS), in which independent component analysis (ICA) and beamforming are combined to resolve the low-convergence problem through optimization in ICA. The proposed method consists of the following four parts: (1) frequency-domain ICA with direction-of-arrival (DOA) estimation, (2) null beamforming based on the estimated DOA, (3) diversity of (1) and (2) in both iteration and frequency domain, and (4) subband elimination (SBE) based on the independence among the separated signals. The temporal alternation between ICA and beamforming can realize fast- and high-convergence optimization. Also SBE enforcedly eliminates the subband components in which the separation could not be performed well. The experiment in a real car environment reveals that the proposed method can improve the qualities of the separated speech and word recognition rates for both directional and diffusive noises.

Noise and speaker adaptation techniques are essential to realize robust speech recognition in real noisy environments . In this paper, we applied spectral subtraction to an unsupervised speaker adaptation algorithm in noisy environments. The adaptation algorithm consists of the following five steps. (1) Spectral subtraction is carried out for noise added database. (2) Noise matched acoustic models are trained by using noise added speech database. (3) HMM sufficient statistics for each speaker are calculated from noise added speech database, and stored. (4) According to one arbitrary utterance, speakers close to a test speaker are selected by using speaker GMMs. (5) Speaker adapted acoustic models are constructed from HMM sufficient statistics of the selected speakers. We evaluated our unsupervised speaker adaptation algorithm in noisy environments in the 20k dictation task. The recognition experiments show that our speaker adapted acoustic model can achieve 82% word accuracy in 20dB SNR, which is about 6% higher than that of the noise matched models trained by Forward-Backward algorithm.

We implemented a humanoid robot, ASKA, in our university reception desk for the computerized university guidance. ASKA can recognize a user's question utterance, and answer the user's question by its text-to-speech voice, hand gesture and head movement. This paper describes the speech related parts of ASKA. ASKA can deal with a wide task domain of 20k large vocabulary using a word trigram model and an elaborated speaker-independent acoustic model. ASKA can also make a response with keyword and key-phrase detection in the N-best speech recognition results. The word recognition rate for the reception task is 90.9%, and the rate for the out-of-domain task is 78.9%. The correct response rate for the reception task is 61.7%. Users can enjoy their question-answering with ASKA.

This paper focuses on microphone arrays to realize distant-talking speech recognition in real environments. In distant-talking situations, users can speak at arbitrary positions while moving. Therefore, it,is very important for high quality speech acquisition using microphone arrays to localize a talker accurately. However, it is very difficult to localize a moving talker in noisy and reverberant environments. The talker localization errors result in performance degradation of speech recognition. One way to solve this problem is to integrate the speech recognition process and the talker localization into a unified framework. This paper proposes a new speech recognition algorithm based on a three-dimensional (3-D) Viterbi search. The 3-D Viterbi method extracts a direction-time sequence of parameter vectors by steering a beam to every direction in every frame, then finds the most likely path in a 3-D trellis space composed of talker directions, input frames and HMM states. This means that speech recognition and talker localization are performed simultaneously within a statistical framework. To evaluate the performance of the 3-D Viterbi method, recognition experiments for real environment data were carried out. The results confirmed that the 3-D Viterbi method drastically improves the recognition performance for the moving talker case as well as for the fixed-position talker case.

Deals with the recognition of distant talking speech and, particularly, with the simultaneous recognition of multiple sound sources. A problem that must be solved in the recognition of distant talking speech is talker localization. In some approaches, the talker is localized by using short- and long-term power. The 3-D Viterbi search based method proposed by Yamada et al.(1998), integrates talker localization and speech recognition. This method provides high recognition rates but its application is restricted to the presence of one talker. In order to deal with multiple talkers, we extended the 3-D Viterbi search method to a 3-D N-best search method enabling the recognition of multiple sound sources. The paper describes our baseline 3-D N-best search-based system and two additional techniques, namely, a likelihood normalization technique and a path distance-based clustering technique. The paper also describes experiments carried out in order to evaluate the performance of the system.

We address a method to efficiently select Gaussian mixtures for fast acoustic likelihood computation. It makes use of context-independent models for selection and back-off of corresponding triphone models. Specifically, for the k-best phone models by the preliminary evaluation, triphone models of higher resolution are applied, and others are assigned likelihoods with the monophone models. This selection scheme assigns more reliable back-off likelihoods to the un-selected states than the conventional Gaussian selection based on a VQ codebook. It can also incorporate efficient Gaussian pruning at the preliminary evaluation, which offsets the increased size of the pre-selection model. Experimental results show that the proposed method achieves comparable performance as the standard Gaussian selection, and performs much better under aggressive pruning condition. Together with the phonetic tied-mixture modeling, acoustic matching cost is reduced to almost 14% with little loss of accuracy.

In real environments, the presence of room reverberations seriously degrades the quality in sound capture. To solve this problem, multiple beamforming, which forms directivity not only in the direction of the desired sound source but also in the direction of reflection images, was proposed by Flanagan et al. (1993). However, it is difficult to apply this method practically in real environments, since this application requires that the distortion of reflection sound signals by wall impedances be equalized. The paper proposes a multiple beamforming algorithm that equalizes the amplitude-spectrum and phase-spectrum of reflection signals by a cross-spectrum method. Evaluation experiments are conducted in real environment. In a SDR (signal to distortion ratio) evaluation, the proposed multiple beamformer achieves signal distortion reduction more effectively than the conventional single beamformer and the conventional multiple beamformer without equalization. In addition, in an automatic speech recognition evaluation, the equalized multiple beamformer achieves a higher recognition performance than those of the above conventional beamformers.

In the voice conversion algorithm based on the Gaussian Mixture Model (GMM) applied to STRAIGHT, quality of converted speech is degraded because the converted spectrum is exceedingly smooth. We propose the GMM-based algorithm with dynamic frequency warping to avoid the over-smoothing. We also propose an addition of the weighted residual spectrum, which is the difference between the GMM-based converted spectrum and the frequency-warped spectrum, to avoid the deterioration of conversion-accuracy on speaker individuality. Results of the evaluation experiments clarify that the converted speech quality is better than that of the GMM-based algorithm, and the conversion-accuracy on speaker individuality is the same as that of the GMM-based algorithm in the proposed method with the properly-weighted residual spectrum.

Describes an efficient method for unsupervised speaker adaptation. This method is based on (1) selecting a subset of speakers who are acoustically close to a test speaker, and (2) calculating adapted model parameters according to the previously stored sufficient HMM statistics of the selected speakers' data. In this method, only a few unsupervised test speaker's data are required for the adaptation. Also, by using the sufficient HMM statistics of the selected speakers' data, a quick adaptation can be done. Compared with a pre-clustering method, the proposed method can obtain a more optimal speaker cluster because the clustering result is determined according to test speaker's data on-line. Experimental results show that the proposed method attains better improvement than MLLR from the speaker independent model. Moreover the proposed method utilizes only one unsupervised sentence utterance, while MLLR usually utilizes more than ten supervised sentence utterances.

There have been higher demands recently for Automatic Speech Recognition (ASR) systems able to operate robustly in acoustically noisy environments. This paper proposes a method to effectively integrate audio and visual information in audiovisual (bi-modal) ASR systems. For such integration, the following issues are important: (1) The synchronization of the audio and visual information, and (2) The optimization of a system in its environment. In (1), the individual feature of the speech and lip movements has the time lag, and has the correlation. To address this problem, we introduce an integration method using HMM composition. In (2), we examine whether the GPD algorithm can adaptively estimate the stream weights. Evaluation experiments show that the proposed method improves the recognition accuracy for noisy speech.

EUROSPEECH2001: the 7th European Conference on Speech Communication and Technology, September 3-7, 2001, Aalborg, Denmark.

Julius is a high-performance, two-pass LVCSR decoder for researchers and developers. Based on word 3-gram and context-dependent HMM, it can perform almost real-time decoding on most current PCs in 20k word dictation task. Major search techniques are fully incorporated such as tree lexicon, N-gram factoring, cross-word context dependency handling, enveloped beam search, Gaussian pruning, Gaussian selection, etc. Besides search efficiency, it is also modularized carefully to be independent from model structures, and various HMM types are supported such as shared-state triphones and tied-mixture models, with any number of mixtures, states, or phones. Standard formats are adopted to cope with other free modeling toolkit. The main platform is Linux and other Unix workstations, and partially works on Windows. Julius is distributed with open license together with source codes, and has been used by many researchers and developers in Japan.

Voice conversion is a technique for producing utterances using any target speakers' voice from a single source speaker's utterance. In this paper, we apply cross-language voice conversion between Japanese and English to a system based on a Gaussian Mixture Model (GMM) method and STRAIGHT, a high quality vocoder. To investigate the effects of this conversion system across different languages, we recorded two sets of bilingual utterances and performed voice conversion experiments using a mapping function which converts parameters of acoustic features for a source speaker to those of a target speaker. The mapping functions were trained using bilingual databases of both Japanese and English speech. In an objective evaluation using Mel cepstrum distortion (Mel CD), it was confirmed that the system can perform cross-language voice conversion with the same performance as that within a single-language.

It is very important for a hands-free speech interface to capture distant talking speech with high quality. A microphone array is an ideal candidate for this purpose. However, this approach requires localizing the target talker. To cope with this problem, we propose a new talker localization method consisting of two algorithms. One algorithm is for multiple sound source localization based on CSP (Cross-power Spectrum Phase) analysis. The other algorithm is for sound source identification among localized multiple sound sources towards talker localization. In this paper, we particularly focus on the latter statistical sound source identification among localized multiple sound sources with statistical speech and environmental sound models based on GMMs (Gaussian Mixture Models) and a microphone array towards talker localization.

EUROSPEECH2001: the 7th European Conference on Speech Communication and Technology, September 3-7, 2001, Aalborg, Denmark.

We propose a new algorithm for blind source separation (BSS), in which independent component analysis (ICA) and beamforming are combined to resolve the low-convergence problem through optimization in ICA. The proposed method consists of the following three parts: (1) frequency-domain ICA with direction-of-arrival (DOA) estimation, (2) null beamforming based on the estimated DOA, and (3) integration of (1) and (2) based on the algorithm diversity in both iteration and frequency domain. The inverse of the mixing matrix obtained by ICA is temporally substituted by the matrix based on null beamforming through iterative optimization, and the temporal alternation between ICA and beamforming can realize fast- and high-convergence optimization. The results of the signal separation experiments reveal that the signal separation performance of the proposed algorithm is superior to that of the conventional ICA-based BSS method, even under reverberant conditions.

In the voice conversion algorithm based on the Gaussian Mixture Model (GMM), quality of the converted speech is degraded because the converted spectrum is exceedingly smoothed. In this paper, we newly propose the GMM-based algorithm with the Dynamic Frequency Warping (DFW) to avoid the over-smoothing. We also propose that the converted spectrum is calculated by mixing the GMM-based converted spectrum and the DFW-based converted spectrum, to avoid the deterioration of conversion-accuracy on speaker individuality. Results of the evaluation experiments clarify that the converted speech quality is better than that of the GMM-based algorithm, and the conversionaccuracy on speaker individuality is the same as that of the GMM-based algorithm in the proposed algorithm with the proper weight for mixing spectra.

An unsupervised acoustic model adaptation algorithm using MLLR and speaker selection for noisy environments is proposed. The proposed algorithm requires only one arbitrary utterance and environmental noise data. The adaptation procedure is composed of the following four steps. (1) Speaker selection from a large number of database speakers is carried out using GMM speaker models based on one arbitrary utterance. (2) Initial speaker adapted HMM acoustic models are calculated from the HMM sufficient statistics of the selected speakers, where the sufficient HMM statistics are pre-calculated and stored. (3) A small subset of the clean speech database from the selected speakers and the environment noise data are superimposed. (4) MLLR adaptation is carried out using the noise-superimposed speech database from the selected speakers. The proposed algorithm is evaluated in a 20k vocabulary dictation task for newspaper in noisy environments. We attain 85.7% word correct rate in 25dB SNR, which is slightly better than the matched model by the E-M training using noise superimposed whole speech database. The proposed algorithm is also 7% better than the HMM composition algorithm.

This paper describes an efficient method of unsupervised speaker adaptation. This method is based on (1) selecting a subset of speakers who are acoustically close to a test speaker, and (2) calculating adapted model parameters according to the previously stored sufficient statistics of the selected speakers' data. In this method, only a few unsupervised test speaker's data are necessary for the adaptation. Also, by using the sufficient HMM statistics of the selected speakers' data, a quick adaptation can be done. Compared with a pre-clustering method, the proposed method can obtain a more optimal cluster because the clustering result is determined according to test speaker's data on-line. Experimental results show that the proposed method attains better improvement than MLLR from the speaker-independent model. The proposed method is evaluated in details and discussed.

This paper presents a hands-free speech recognition method based on HMM composition and separation for speech contaminated not only by additive noise but also by an acoustic transfer function. The method realizes an improved user interface such that a user is not encumbered by microphone equipment in noisy and reverberant environments. The use of HMM composition has already been proposed for countering additive noise. In this paper, the same approach is extended to handle convolutional acoustic distortion in a reverberant room, by using an HMM to model the acoustic transfer function. The states of this HMM correspond to different positions of the sound source. It can represent the positions of the sound sources, even if the speaker moves. This paper also proposes a new method, HMM separation, for estimating the HMM parameters of the acoustic transfer function on the basis of a maximum likelihood manner. The proposed method is obtained through the reverse of the process of HMM composition, where the model parameters are estimated by maximizing the likelihood of adaptation data uttered from an unknown position. Therefore, measurement of impulse responses is not required. The paper also describes the performance of the proposed methods for recognizing real distant-talking speech. The results of experiments clarify the effectiveness of the proposed method.

A phonetic tied-mixture (PTM) model for efficient large vocabulary continuous speech recognition is presented. It is synthesized from context-independent phone models with 64 mixture components per state by assigning different mixture weights according to the shared states of triphones. Mixtures are then re-estimated for optimization. The model achieves a word error rate of 7.0% with a 20000-word dictation of newspaper corpus, which is comparable to the best figure by the triphone of much higher resolutions. Compared with conventional PTMs that share Gaussians by all states, the proposed model is easily trained and reliably estimated. Furthermore, the model enables the decoder to perform efficient Gaussian pruning. It is found out that computing only two out of 64 components does not cause any loss of accuracy. Several methods for the pruning are proposed and compared, and the best one reduced the computation to about 20%.

Accurate localization of multiple sound sources is indispensable for the microphone array-based high quality sound capture. For single sound source localization, the CSP (cross-power spectrum phase analysis) method has been proposed. The CSP method localizes a sound source as a crossing point of sound directions estimated using different microphone pairs. However, when localizing multiple sound sources, the CSP method has a problem that the localization accuracy is degraded due to cross-correlation among different sound sources. To solve this problem, this paper proposes a new method which suppresses the undesired cross-correlation by synchronous addition of CSP coefficients derived from multiple microphone pairs. Experiment results in a real room showed that the proposed method improves the localization accuracy when increasing the number of the synchronous addition.

This paper describes a hands-free speech recognition method based on HMM composition and separation for speech contaminated not only by additive noise but also by an acoustic transfer function. The method realizes an improved user interface such that a user is not encumbered by microphone equipment in noisy and reverberant environments. In this approach, an attempt is made to model acoustic transfer functions by means of an ergodic HMM. The states of this HMM correspond to different positions of the sound source. It can represent the positions of the sound sources, even if the speaker moves. The HMM parameters of the acoustic transfer function are estimated by HMM separation. The method is obtained through the reverse of the process of HMM composition, where the model parameters are estimated by maximizing the likelihood of adaptation data uttered from an unknown position. Therefore, measurement of impulse responses is not required. In this paper, we record the speech of a distant moving speaker in real environments. The results of experiments for the speech of a distant moving speaker clarified the effectiveness of HMM composition and separation.

This paper proposes a noise-tolerant method for fundamental frequency (F0) extraction. This method includes several new ideas, including the estimation of the instantaneous frequencies of the higher harmonic components, and the design of an adaptive weighting function based on a bandwidth equation that combines the F0 information in the harmonic components. To evaluate the proposed method, we constructed a relatively large database of simultaneous recordings of speech waveforms and EGG (Electro Glotto Graphy). The database consists of 30 sentences pronounced by 14 male and 14 female normal subjects, i.e., 840 sentences in total. The duration of the sound is about 35 minutes including about 20 minutes of voicing. The experiments were performed with additive noise for four pitch extraction methods, i.e., the proposed method, the original TEMPO, an improved cepstrum method, and a common F0 extraction program in ESPS. The results were as follows: 1) the proposed method is always better than any of the other methods when the SNR is greater than about 2 dB; 2) for high SNR values (> 15 dB), the correct rates of the proposed method and the original TEMPO are about 95% and much better than the improved cepstrum method (92%) and the ESPS function (89%); and 3) all of the methods degrade to less than 62% when the SNR is 0 dB. As a result, the proposed method improves the performance for low SNR values and also maintains high accuracy inherent from the original TEMPO for high SNR values.

This paper describes the investigation of manipulating positions in a speech pitch when lengthening or shortening the pitch period, that is, lowering or raising fundamental frequency of speech. The experimental results revealed that the preferable positions were at the first half of the pitch period for pitch shortening, and at the second half of it for pitch lengthening. The findings are expected to improve the quality of speech synthesis on pitch modulation.

A sharable software repository for Japanese LVCSR (Large Vocabulary Continuous Speech Recognition) is introduced. It is designed as a baseline platform for research and developed by researchers of different academic institutes under a governmental support. The repository consists of a recognition engine (Julius), Japanese acoustic models and statistical language models as well as Japanese morphological analysis tools. These modules can be easily integrated and replaced under a plug-and-play framework, which makes it possible to fairly evaluate components and to develop specific application systems. Assessment of these modules and systems in a 20000-word dictation task is reported. The software repository is freely available to the public.

Bimodal speech recognition systems, with the use of visual information to supplement acoustic information, have been shown to yield better recognition performance than purely acoustic systems, especially when background noise is present. The early integration strategy for HMM-based audio-visual speech recognition is one promising approach, where the output probability is obtaned by product of output probabilites of audio and visual streams. This paper addresses a novel method which optimizes stream weights so as to maximize recognition performance. The proposed method estimates the stream weights based on a normalized log likelihood which is derived by ratio of likelihood of a correct word and highest likelihood of incorrect words. The isolated word recognition experiment results show that the audio-visual speech recognition by proposed method attains 56.2% (10 dB), 55.2% (0dB) and 15.2% (20dB) better performance compared to that only using audio information. The results also show the proposed method can reduce a number of adaptation words.

This paper describes a new blind source separation (BSS) method on microphone array using the subband independent component analysis (ICA) and beamforming. The proposed array system consists of the following three sections: (1) subband-ICA-based BSS section, (2) null beamforming section, and (3) integration of (1) and (2) based on the algorithm diversity. Using this technique, we can resolve the low-convergence problem on optimization in ICA. Signal separation and speech recognition experiments clarify that the noise reduction rate (NRR) of about 18 dB is obtained under the nonreverberant condition, and NRRs of 8 dB and 6 dB are obtained in the case that the reverberation times are 150 msec and 300 msec. These performances are superior to those of both simple ICA-based BSS and simple beamforming method. Also, the improvements of the proposed method in word recognition rates are superior to those of the conventional ICA-based BSS method under all reverberant conditions.

The voice conversion algorithm based on the Gaussian mixture model (GMM) has also been proposed by Stylianou et al. In this algorithm, the acoustic space of a speaker is represented continuously. In this paper, we apply this GMM-based voice conversion algorithm to STRAIGHT proposed by Kawahara et al., which is recognized as a high quality vocoder. In order to evaluate this voice conversion algorithm, we perform subjective and objective experiments on speech quality and speaker individuality, comparing with the method based on the codebook mapping. As results, the performance of the GMM-based voice conversion algorithm is better than that of the codebook mapping method. Effects by the amount of training data for the voice conversion algorithms are also investigated, as well as the number of the Gaussian mixtures. These evaluation results clarify that the GMM-based voice conversion algorithm is successfully applied to STRAIGHT.

The goal of this paper is to locate and understand the fine fundamental problems that exist in the representation of speech sounds by a very high quality speech analysis/synthesis engine namely STRAIGHT. The approach followed here is the evaluation of this system using subjective measures. We use the diagnostic rhyme test (DRT) to evaluate the intelligibility of speech analysed and synthesised by this system for various analysis frame-rates. Consequently we catagorise the fine problems and suggest possible improvements. The results from the DRT have indicated that STRAIGHT can produce speech with an average DRT score of 95 between 1-5 ms analysis frame-rate. In addition, a set of subjective quality measures using MOS and MNRU tests have been conducted. These tests have been carried out for three different versions of the STRAIGHT system: versions 17, 23 and 30. The DRT has been carried out using version 23 only. Based on the subjective evaluation results, a discussion of possible improvements to the STRAIGHT system is given.

The recognition of distant talking speech in a noisy and reverberant environments is key issue in any speech recognition system. A so-called hands-free speech recognition system plays an important role in the natural and friendly human-machine interface. Considering the practical use of a speech recognition system, we realize that such a system has to deal, also, with the case of the presence of multiple sound sources, including multiple talkers, as well as other noise sources. This paper proposes a novel method which recognizes multiple talkers simultaneously in real environments by extending the 3-D Viterbi search to a 3-D N-best search algorithm. While the 3-D Viterbi method finds the most likely path in the 3-D trellis space, the proposed method considers multiple hypotheses for each direction in every frame. Combinations of the direction sequence and the phoneme sequence of multiple sources are included in the N-best list. The paper investigates the performance of the proposed method through experiments using real utterances of multiple talkers.

An efficient representation of short-time phase characteristics of speech sounds is proposed, based on findings which suggest the perceptual importance of phase characteristics. Subjective tests indicated that the synthesized speech sounds by the proposed method are indistinguishable from the original speech sounds with a moderate data compression. The proposed representation uses lower-order coefficients of the inverse Fourier transform of the group delay of speech. It also alleviates the voiced/unvoiced decision, which is an indispensable part in conventional speech coding algorithms. These features make our method potentially very useful in many applications like speech morphing.

A user-friendly speech interface in a car cabin is highly needed for safety reasons. This paper describes a robust speech recognition method that can cope with additive noise and multiplicative distortions. A known additive noise, a source signal of which is available, might be canceled by NLMS-VAD (normalized least mean squares with frame-wise voice activity detection). On the other hand, an unknown additive noise, a source signal of which is not available, is suppressed with CSS (continuous spectral subtraction). Furthermore, various multiplicative distortions are simultaneously compensated with E-CMN (exact cepstrum mean normalization) which is speaker dependent/environment-dependent CMN for speech/non-speech. Evaluation results of the proposed method for car cabin environments are finally described.

A microphone array is a promising solution for realizing hands-free speech recognition in real environments. Accurate talker localization is very important for speech recognition using a microphone array. However localization of a moving talker is difficult in noisy reverberant environments. Talker localization errors degrade the performance of speech recognition. To solve the problem, this paper proposes a new speech recognition algorithm which considers multiple talker direction hypotheses simultaneously. The proposed algorithm performs a Viterbi search in 3-dimensional trellis space composed of talker directions, input frames, and HMM states. As a result, a locus of the talker and a phoneme sequence of the speech are obtained by finding an optimal path with the highest likelihood. To evaluate the performance of the proposed algorithm, speech recognition experiments are carried out on simulated data and real environment data. These results show that the proposed algorithm works well even if the talker moves.

ICSLP1998: the 5th International Conference on Spoken Language Processing, November 30 - December 4, 1998, Sydney, Australia.

In this paper we present the first public Japanese speech corpus for large vocabulary continuous speech recognition (LVCSR) technology, which we have titled JNAS (Japanese Newspaper Article Sentences). We designed it to be comparable to the corpora used in the American and European LVCSR projects. The corpus contains speech recordings (60 hrs.) and their orthographic transcriptions for 306 speakers (153 males and 153 females) reading excerpts from the newspaper's articles and phonetically balanced (PB) sentences. This corpus contains utterances of about 45,000 sentences as a whole with each speaker reading about 150 sentences. JNAS is being distributed on 16 CD-ROMs.

The project of Japanese LVCSR (Large Vocabulary Continuous Speech Recognition) platform is introduced. It is a collaboration of researchers of different academic institutes and intended to develop a sharable software repository of not only databases but also models and programs. The platform consists of a standard recognition engine, Japanese phone models and Japanese statistical language models. A set of Japanese phone HMMs are trained with ASJ (Acoustic Society of Japan) databases of 20K sentence utterances per each gender. Japanese word N-gram (2-gram and 3-gram) models are constructed with a corpus of Mainichi newspaper of four years. The recognition engine JULIUS is developed for assessment of both acoustic and language models. The modules are integrated as a Japanese LVCSR system and evaluated on 5000-word dictation task. The software repository is available to the public.

In this paper, we evaluate performance of model adaptation by the previously proposed HMM decomposition method on telephone speech recognition. The HMM decomposition method separates a composed HMM into a known phoneme HMM and an unknown noise and channel HMM by maximum likelihood (ML) estimation of the HMM parameters. A transfer function (telephone channel) HMM is estimated using adaptation speech data by applying the HMM decomposition twice in the linear spectral domain for noise and in the cepstral domain for channel. The telephone speech data for evaluation are recorded through 10 kinds of ordinary analog telephone handsets and cordless telephone handsets. The test results show that the average phrase accuracy with the clean speech HMMs is 60.9% for the ordinary analog telephone handsets, and 19.6% for the cordless telephone handsets. By the HMM decomposition method, the average phrase accuracy is improved to 78.1% for the ordinary analog telephone handsets, and 50.5% for the cordless telephone handsets.

ICSLP1998: the 5th International Conference on Spoken Language Processing, November 30 - December 4, 1998, Sydney, Australia.

ICSLP1998: the 5th International Conference on Spoken Language Processing, November 30 - December 4, 1998, Sydney, Australia.

ICSLP1998: the 5th International Conference on Spoken Language Processing, November 30 - December 4, 1998, Sydney, Australia.

We investigate methods using the hidden Markov model (HMM) to drive a lip movement sequence with input speech. We have already investigated a mapping method based on the Viterbi decoding algorithm which converts an input speech to a lip movement sequence through the most likely HMM state sequence conducted by audio HMMs. However, the method contains a substantial problem of producing errors along incorrectly decoded HMM states. This paper newly proposes a method to re-estimate the visual parameters using the HMMs of the audio-visual joint probability under the expectation-maximization (EM) algorithm. In experiments, the proposed mapping method using the EM algorithm shows an error reduction of 26% compared to a method using the Viterbi algorithm at incorrectly decoded bi-labial consonants.

The performance of a speech recognizer is degraded drastically in reverberant environments. The authors propose a novel algorithm which can model an observation signal by composition of HMMs of clean speech, noise and an acoustic transfer function. However, estimating HMM parameters of the acoustic transfer function is still a serious problem. In their previous paper, they measured real impulse responses of training positions in an experiment room. It is inconvenient and unrealistic to measure impulse responses for every possible new experiment room. The paper presents a new method for estimating HMM parameters of the acoustic transfer function from some adaptation data by using an HMM decomposition algorithm which is an inverse process of the HMM composition. Its effectiveness is confirmed by a series of speaker dependent and independent word recognition experiments on simulated distant-talking speech data.

This paper describes a new approach to ML-SSS (Maximum Likelihood Successive State Splitting) algorithm that uses tied- mixture representation of the output probability density function instead of a single Gaussian during the splitting phase of the ML-SSS algorithm. The tied-mixture representation results in a better state split gain, because it is able to measure diferences in the phoneme environment space that ML-SSS can not. With this more informative gain the new algorithm can choose a better split state and corresponding data. Phoneme clustering experiments were conducted which lead up to 38% of error reduction if compared to the ML-SSS algorithm.

EUROSPEECH1997: the 5th European Conference on Speech Communication and Technology , September 22-25, 1997, Rhodes, Greece.

EUROSPEECH1997: the 5th European Conference on Speech Communication and Technology , September 22-25, 1997, Rhodes, Greece.

Hands-free speech recognition is a very important issue for a natural human machine interface. The distant talking speech in real environments is distorted by noise and reverberation of the room. This paper introduces characteristics of the room acoustical distortion and their influences on speech recognition accuracy. Then the paper tries to give a prospect of the solution based on previous studies and our research efforts. Especially a microphone array based-method and a model adaptation method are discussed. The microphone array can reduce the influences of the acoustical distortion by beam-forming. On the other hand, the model adaptation method can estimate the acoustical transfer function and adapt the speech models against the distorted observation signals. Furthermore, this paper also addresses hands-free speech recognition by incorporating automatic lip reading.

This paper investigates the Cepstrum Mean Normalization(CMN) which has been widely acknowledged useful for compensation of multiplicative distortions. However, the performance of usual CMN is limited because the normalization by a single cepstrum mean vector is not enough to compensate many factors of multiplicative distortion in real environments. To solve this problem, a new method E-CMN is proposed. The method estimates two cepstrum mean vectors, one for speech and the other for non-speech for each speaker and subtracts them from an input cepstrum This method is capable of compensating various kinds of multiplicative distortion collectively to normalize input spectra. Furthermore, a new model-adaptive approach E-CMN/PMC, based on E- CMN and HMM composition, is proposed for environments with additive noise and multiplicative distortions. This method is simplified in a sense that it is possible to add speech models and an additive noise model without any iterative operations. Matching gains for all frequency bands of speech models to the noise model are uniquely estimated as a cepstrum mean vector for speech. The performance of E-CMN/PMC in adverse car environsnents is finally evaluated.

This paper presents a robust speech recognition method based on the HMM composition for the noisy room acoustics distorted speech. The method realizes an improved user interface such as the user is not encumbered by microphone equipment. The proposed HMM composition is obtained by naturally extending the HMM composition method of an additive noise to that of the convolutional room acoustics distortion. The HMM composition is conducted by 2 steps: (1) composition of HMMs of a speech and acoustical transfer function in the cepstrum domain, and (2) composition of distorted speech and noise HMMs in the linear spectral domain. The speaker dependent/independent word recognition experiments are carried out using the speech database contaminated by the additive noise and convolutional room acoustics distortion. The evaluation experiments are also conducted for unknown testing sound source positions. These results clarified the effectiveness of the proposed method.

This paper proposes robust speech recognition with Speaker Localization by a Arrayed Microphone (SLAM) to realize hands-free speech interface in noisy environments.In order to localize a speaker direction accurately in low SNR conditions, a speaker localization algorithm based on extracting a pitch harmonics is introduced.To evaluate the performance of the proposed system, speech recognition experiments are carried out both in computer simulation and real environments.These results show that the proposed system attains the much higher speech recognition performance than that of a single microphone not only in computer simulation but also in real environments.

The paper describes an efficient search algorithm for spontaneous speech recognition. This algorithm merges candidates that have the same meaning in beam search. To do this effectively, the authors change their phoneme-synchronous search algorithm into a quasi-frame-synchronous one. The algorithm is evaluated using a telephone directory assistance task containing more than 70000 subscriber names. The average sentence understanding rate for eight speakers is improved from 58% to 65% by merging candidates at the meaning level.>

This paper describes a phonetic typewriter and a dictation machine that utilize the underlying statistical structure of phoneme or character sequences. The approach of using syllable or character trigrams is applied to language source modeling. The language source models are obtained by calculating trigram probabilities from a large text database. These models are combined with the HMM-LR continuous speech recognition system.3,6 The phonetic typewriter is tested using 274 phrases uttered by one male speaker. The syllable source model achieves a 94.9% phoneme recognition rate with the test-set phoneme perplexity of 3.9. Without the syllable source model, the phoneme recognition rate is only 73.2%. A trigram model based on characters is also evaluated. This character source model can reduce the syllable perplexity significantly to 7.7, compared with 10.5 of the syllable source model. The character source model achieves a 78.5% character transcription rate for the 274 phrase utterances. The experimental results show that a syllable source model and a character source model are very effective for realizing a Japanese dictation machine.

The authors describe two approaches for adapting a specific syllable trigram model to a new task. One uses a small amount of text data similar to the target task, and the other uses supervised learning using the most recent input phrases. The effect of each adaptation is verified with syllable perplexity and phrase recognition. Where the syntactic knowledge was only the syllable trigram model, the perplexity was reduced from 54.5 to 18.1 for the adaptation using 100 phrases of similar text, and was reduced to 14.6 by the supervised learning. The recognition rates were also improved from 42.3% to 46.6% and 50.9%, respectively. Text similarity for speech recognition is also studied.>

A hidden Markov modeling technique that uses a statistical modeling of vector quantization (VQ)-code transitions is proposed. A bigram-constrained HMM is obtained by combining a VQ-code bigram and the conventional speaker-independent HMM. The proposed model reduces overlapping of the feature distributions between different phonemes by restricting the local VQ-code transitions. The output probabilities in the model are conditioned by the VQ-code of the previous frame. Therefore, the output probability distribution of the model changes depending on the previous frame even in the same state. A speaker-dependent bigram-constrained HMM is obtained using a VQ-code bigram calculated from utterances of an input speaker. A speaker-independent bigram-constrained HMM is obtained using a VQ-code bigram calculated from utterances of many speakers. The model was evaluated by an 18-Japanese-consonant recognition experiment using 5240 words. The speaker-independent bigram-constrained HMM achieved an average recognition accuracy of 76.3% which is 5.5% higher than the conventional speaker-independent HMM.>

The authors describe a Japanese dictation system that uses a stochastic language model based on sequences of Japanese characters. The trigram probabilities, which are obtained from a text database consisting of Kanji and Kana are used to construct a source model. A Japanese dictation system generally requires Kana-to-Kanji conversion if the system uses a phoneme based unit for the acoustic processing. However, a system that uses a Kanji-and-Kana character source model can generate an output Kanji-and-Kana sequence directly from input speech without using Kana-to-Kanji conversion. The system is tested using 274 phrases uttered by one male speaker, and achieves 58.4% phrase transcription rate. When the system uses a pronunciation dictionary and eliminates the candidates whose Kanji readings are contextually inappropriate, the phrase transcription rate increases to 63.9%. It is confirmed that a Japanese character source model is efficient for a Japanese dictation system.>

The authors describe the robustness of six types of phoneme-based HMMs (hidden Markov models) against speaking-style variations. The six types of models are VQ (vector quantization)-based and fuzzy VQ-based discrete HMMs, and single-Gaussian and mixture-Gaussian HMMs with either diagonal or full covariance matrices. The mixture-Gaussian HMM with diagonal covariance matrices, the fuzzy VQ-based discrete HMM, and the single-Gaussian HMM with full covariance matrices show better results than the other three in 18-Japanese-consonant recognition experiments. The authors also propose a model-adaptation technique that combines multiple models using the deleted interpolation. This technique makes models easy to apply to different-speaking-style speech.>

A phonetic typewriter that utilizes the underlying statistical structure of phoneme/character sequences is described. The syllable/character trigram approach to language modeling is adopted to make language source models. These are obtained by calculating trigram probabilities, using a large text database. The phonetic typewriter is tested using 279 phrases uttered by one male speaker, and the syllable source model achieves a 94.9% phoneme recognition rate with the test-set phoneme perplexity of 3.9. Without the syllable trigram, the phoneme recognition rate is only 73.2%. A trigram model based on characters is also evaluated. This model can reduce the phoneme perplexity significantly compared with that of the syllable trigram.>

First, the part of spectral difference that is due to the difference in language is assessed. This is investigated using a bilingual speaker's speech data. It is found that the interlanguage (between English and Japanese) difference is smaller than the interspeaker difference. Listening tests indicate that the difference between English and Japanese is very small. Second, a model for cross-language voice conversion is described. In this approach, voice conversion is considered a mapping problem between two speakers' spectrum spaces. The spectrum spaces are represented by codebooks. From this point of view, a cross-language voice conversion model and measures for the model are proposed. The converted speech from male to female is as understandable as the unconverted speech and, moreover, it is recognized as female speech.>

An improvement of the hidden Markov model (HMM) LR continuous-speech recognizer using multiple codebooks, HMM state duration control and fuzzy vector quantization is described. The system recognizes Japanese phrases (Bunsetsu) according to a context-free grammar including 1035 words. In speaker-dependent conditions, a phrase recognition rate of 88.4% (99.0% for the top five candidates) was attained. The system was tested with speaker-adaptation based on a codebook mapping algorithm. An average speaker-adapted phrase recognition rate of 81.6% (98.8% for the top five candidates) was attained.>

A method of dealing with articulatory speaker variations in hidden Markov models (HMMs) for speaker adaptation is proposed. Speech data from many speakers are spectrally mapped onto a standard speaker. These data are used to teach the HMM the interspeaker articulatory variations that subsist across the spectral mapping. The proposed method is compared to other adaptation methods through the /b,d,g/ recognition task. The results show 82.5% recognition accuracy, which is better than the rates of other methods. Evaluation experiments on a Japanese all phoneme recognition task and a continuous-speech recognition task are reported. Average recognition rates for Japanese all phonemes are 71.3% and 93.2%, for the best candidate and the top-three candidates, respectively. These are 0.7% and 1.5% higher than the rates of the basic spectrum mapping method. In the continuous-speech recognition experiment, average phrase recognition rates are 74.9% and 96.2%, for the best candidate and the top-five candidates, respectively.>

A description of integrated training methods of time-delay neural networks (TDNNs) for spotting all Japanese phonemes is presented. The time-shift invariance of the TDNN is confirmed by the use of 2620 testing words, with 95.8% of the phonemes correctly spotted. These experiments show that the spotting performance of the TDNN is high, while types of tendencies toward insertion and deletion errors are clarified. To reduce these spotting errors, integrated training methods using various training token positions are proposed. These methods allow the TDNN to correctly spot phonemes at a rate of 98.0% and also make it possible to realize large-vocabulary, vocabulary-independent speech recognition. To verify this, large-vocabulary speech recognition with 5240 common Japanese words was performed using a predictive LR parser. Recognition rates of 92.6% and 97.6% were obtained for the first and second choices respectively.>

A comparative study of speaker adaptation using neural network spectral mapping and fuzzy vector quantization (VQ)-based spectral mapping is described. The speaker adaptation experiments were carried out using a database of 216 phonetically balanced words uttered by three speakers. The accuracy of spectral mapping is measured and evaluated by interspeaker spectral distortion. The results show the fuzzy VQ-based spectral mapping algorithm to be about 6% better in spectral distortion than nonlinear spectral mapping by a feedforward neural network. An investigation of the actual spectrogram shows that the fuzzy VQ-based spectral mapping works better than the neural network mapping.>

A method is presented for phoneme segmentation by an expert system utilizing spectrogram reading strategy and knowledge. The expert system detects phonemes in a spectrogram and determines their boundaries as well as their coarse categories. To simulate a human expert spectrogram reading process, the system performs assumption-based inference with certainty factors, and top-down acoustic feature extraction under phonetic context hypotheses. The system, into which Japanese consonant segmentation knowledge is incorporated, is able to detect about 90% of the phonemes correctly. In particular, the phoneme boundaries detected by the system are as accurate as those detected by human experts. The result is that the phonemes obtained by the expert system can be identified using a stochastic phoneme recognition method.>

An island-driven speech recognition system using word-spotting and word-verification processes based on HMM (hidden Markov model) phone units is proposed. First, the system detects a keyword as an island in continuous speech. After that, the system expands the island by verifying neighbor words. Word models for detection or verification are composed of HMM phone units. The proposed speech recognition system was tested by word-spotting and phrase-recognition experiments using 75 sentences which contained 549 phrases uttered by three male speakers. The noun-detection rate is 96.5% with 140 false alarms (in 549 phrases). The phrase-recognition rate is 96.2% up to the three best candidates.>

The authors propose a speaker adaptation algorithm which does not depend on speech recognition algorithms. The proposed spectral mapping algorithm is based on three ideas: (1) accurate representation of the input vector by separate vector quantization and fuzzy vector quantization, (2) continuous spectral mapping from one speaker to another by fuzzy mapping, and (3) accurate establishment of spectral correspondence based on the fuzzy relationship of the membership function obtained from supervised training. The spectrum dynamic features are also utilized. The algorithm is applied to hidden Markov models (HMMs) and neural networks and evaluated using a database of 216 phonetically balanced words and 5240 important Japanese words uttered by three speakers. The HMM speaker adapted recognition rate for /b,d,g/ is 79.5%. The average recognition rate for the top-three choices is about 91%. The algorithm was applied to neural networks and resulted in almost the same performance. The algorithm was also applied to voice conversion, and a preference score of 65.6% was obtained.>

Using traditional statistical approaches, it is difficult to develop an N-gram word prediction model for constructing an accurate word recognition system because of the increased demand for sample data and parameters to memorize probabilities. To solve this problem, NETgrams, which are neural networks for N-gram word category prediction in text are proposed. NETgrams can easily be expanded from bigram to N-gram networks without exponentially increasing the number of free parameters. Training results show that the NETgrams are comparable to the statistical model and compress information. Results of analyzing the hidden layer (microfeatures) show that the word categories are classified into some linguistically significant groups. It is confirmed that NETgrams perform effectively for unknown data, i.e NETgrams interpolate sparse training data naturally just like the deleted interpolation. A method for speeding up the back-propagation algorithm, which can automatically determine better parameters and achieve a shorter training time is proposed.>

The authors present techniques for spotting Japanese CV syllables/phonemes in input speech based on TDNNs. They constructed a TDNN which can discriminate a single CV syllable or phoneme group. In Japanese, there are only about one hundred syllables, or fewer than 30 phonemes, which makes it feasible to prepare and train the TDNN to spot all possible syllables or phonemes extracted as training tokens from training words. Syllable and phoneme spotting experiments show excellent results, including a syllable spotting rate of better than 96.7% correct. These spotting techniques are proved to be a significant step toward continuous speech recognition.>

It is shown that neural networks for speech recognition can be constructed in a modular fashion by exploiting the hidden structure of previously trained phonetic subcategory networks. The performance of resulting larger phonetic nets was found to be as good as the performance of the subcomponent nets by themselves. This approach avoids the excessive learning times that would be necessary to train larger networks and allows for incremental learning. Large time-delay neural networks constructed incrementally by applying these modular training techniques achieved a recognition performance of 96.0% for all consonants and 94.7% for all phonemes.>

The authors propose a new voice conversion technique through vector quantization and spectrum mapping. The basic idea of this technique is to make mapping codebooks which represent the correspondence between different speakers' codebooks. The mapping codebooks for spectrum parameters, power values and pitch frequencies are separately generated using training utterances. This technique makes it possible to precisely control voice individuality. To evaluate the performance of this technique, hearing tests are carried out on two kinds of voice conversions. One is a conversion between male and female speakers, the other is a conversion between male speakers. In the male-to-female conversion experiment, all converted utterances are judged as female, and in the male-to-male conversion, 65% of them are identified as the target speaker.>

A time-delay neural network (TDNN) for phoneme recognition is discussed. By the use of two hidden layers in addition to an input and output layer it is capable of representing complex nonlinear decision surfaces. Three important properties of the TDNNs have been observed. First, it was able to invent without human interference meaningful linguistic abstractions in time and frequency such as formant tracking and segmentation. Second, it has learned to form alternate representations linking different acoustic events with the same higher level concept. In this fashion it can implement trading relations between lower level acoustic events leading to robust recognition performance despite considerable variability in the input speech. Third, the network is translation-invariant and does not rely on precise alignment or segmentation of the input. The TDNNs performance is compared with the best of hidden Markov models (HMMs) on a speaker-dependent phoneme-recognition task. The TDNN achieved a recognition of 98.5% compared to 93.7% for the HMM, i.e., a fourfold reduction in error.>

A trigram language model based on word categories is introduced in order to improve word recognition results by use of linguistic information. A trigram model based on word sequences requires a lot of memory and training samples to store and estimate its probabilities. To avoid these almost unsolvable problems, a trigram model of words whose probabilities are estimated from the trigram of categories and word occurrence probabilities in the dictionary is introduced. The probabilities of the trigram of categories and the word probabilities in the dictionary are estimated using the Brown Corpus Text Database[1]. This trigram model is efficiently applied to improve word recognition results using a dynamic programming technique. Moreover, probabilities of special word sequences (frozen word sequences) are extracted from the Brown Corpus Text Database and these probabilities are also integrated in the dynamic programming algorithm. Word recognition through speaker adaptation is carried out using three input speakers from the IBM office correspondence task database[3]. The word recognition rate was 80.9%. The trigram model improves the word recognition rate to 89.1%.

Vector quantization (VQ) is a technique that reduces the computation amount and memory size drastically. In this paper, speaker adaptation algorithms through VQ are proposed in order to improve speaker-independent recognition. The speaker adaptation algorithms use VQ codebooks of a reference speaker and an input speaker. Speaker adaptation is performed by substituting vectors in the codebook of a reference speaker for vectors of the input speaker's codebook, or vice versa. To confirm the effectiveness of these algorithms, word recognition experiments are carried out using the IBM office correspondence task uttered by 11 speakers. The total number of words is 1174 for each speaker, and the number of different words is 422. The average word recognition rate using different speaker's reference through speaker adaptation is 80.9%, and the rate within the second choice is 92.0%.

This paper describes a new spoken word recognition approach based on the top-down phoneme segmentation. Fourteen phoneme recognition functions are introduced to deal with various coarticulations. This new approach has two advantages. First, a precise phoneme recognition can be achieved because of the phoneme level top-down verification. Second, only phoneme symbol sequences are required for the vocabulary knowledge source, because the coarticulation knowledge is included in phoneme level knowledge sources. Experimental recognition results for 100 city names uttered by 50 speakers indicate that the phoneme concatenations showing strong coarticulation must be segmented as a unit to achieve a high recognition rate.

This paper describes new technique for use in a word recognition system. This recognition system is especially efffective in speaker-dependent large vocabulary word recognition based on multiple reference templates. In this system, word templates are represented as sequences of descrete phoneme-like (pseudo-phoneme) templates which are automatically determined from a training set of word utterances by a clustering technique. In speaker-dependent 641 city names word recognition experiments, 96.3% recognition accuracy was obtained using 256 phoneme-like templates.

Structure and performance in an acoustic processor in the conversational speech recognition system, called Voice Q-A System II, are described, together with a comparison with the old system. The acoustic processor adopts the LPC peak weighted spectral matching measure. The acoustic processor is much improved on phoneme segmentation and vowel recognition. The task is train seat reservation service, which contains 112 words. Input is conversational speech with a short pause between adjacent phrases. The recognition test was made on the acoustic processor, which is connected with the linguistic processor. Results show 96.9% phrase recognition rate and 66.8% phoneme recognition rate on the average for nine male speakers.

The speech recognition system composing a part of the question-answering system operated by conversational speech is described. The recognition system consists of two stages of process : acoustic processing stage and linguistic processing stage. In the acoustic processing stage, input speech is analyzed and transformed into the phoneme sequence which usually contains ambiguities and errors caused in the segmentation and phoneme recognition. In the linguistic processing stage, the phoneme sequence containing ambiguities and errors is converted into the correct word sequence by the use of the linguistic knowledge such as phoneme rewriting rules, lexicon, syntax, semantics and pragmatics. The voice-operated question-answering system for seat reservation is constructed by computer simulation technique and the promising results are obtained.