This paper presents an instantaneous pitch estimation method based on data adaptive time domain filtering and multivariate synchrosqueezing transform (SST). The filtering approach is implemented with bivariate empirical mode decomposition (bEMD) using white Gaussian noise (wGn) as the reference signal. The bEMD decomposes speech and wGn together into a finite set of intrinsic mode functions (IMFs). The log-energy distribution of wGn's IMFs is employed to determine the threshold used in filtering. The IMFs of speech signal selected by such pre-filtering method is used to construct time-frequency representation (TFR) with multivariate SST. The frequency components are properly localized in the obtained TFR. Spatial filtering and post-processing are applied to TFR prior to estimate the instantaneous pitch. The experimental results illustrate the noise robustness and superiority of the proposed algorithm.

Abstract This paper provides an analysis of several practical issues related to the theory and implementation of Grapheme-to-Phoneme (G2P) conversion systems utilizing the Weighted Finite-State Transducer paradigm. The paper addresses issues related to system accuracy, training time and practical implementation. The focus is on joint n-gram models which have proven to provide an excellent trade-off between system accuracy and training complexity. The paper argues in favor of simple, productive approaches to G2P, which favor a balance between training time, accuracy and model complexity. The paper also introduces the first instance of using joint sequence RnnLMs directly for G2P conversion, and achieves new state-of-the-art performance via ensemble methods combining RnnLMs and n-gram based models. In addition to detailed descriptions of the approach, minor yet novel implementation solutions, and experimental results, the paper introducesPhonetisaurus, a fully-functional, flexible, open-source, BSD-licensed G2P conversion toolkit, which leverages the OpenFst library. The work is intended to be accessible to a broad range of readers.

This work investigated crosslinguistic perception of Mandarin utterances conveying six classes of attitudes, i.e., dominant/ submissive, friendly/hostile, polite/rude, serious/joking, praising/blaming, and sincere/insincere. Five groups of subjects were tested: native Mandarin speakers, Japanese L2 learners of Mandarin, French L2 learners of Mandarin, naive Japanese without Mandarin ability, and naive French without Mandarin ability. A set of Mandarin attitudinal utterances elicited in role-play dialogues were used as stimuli. Perceptual experiments showed that native subjects performed the best in identifying attitudes, and L2 learners judged better than naive foreigners. A complex interaction was found between attitude and the listener’s L1/L2 experience which was also closely related to culture. Also, the correlations between perceptual patterns and prosodic features were examined, and the results suggested that the prosodic cues for certain attitudes might be dependent on the listener’s language/culture background.

This paper proposes a model of speaker-normalized acoustic-toarticulatory mapping using statistical voice conversion. A mapping function from acoustic parameters to articulatory parameters is usually developed with a single speaker’s parallel data. Hence the constructed mapping model can work appropriately only for this specific speaker, and applying this model to other speakers degrades the performance of acoustic-to-articulatory mapping. In this paper, two models of speaker conversion and acoustic-to-articulatory mapping are implemented using Gaussian Mixture Models (GMM), and by integrating these two models, we propose two methods of speaker-normalized acoustic-to-articulatory mapping. One is concatenating these models sequentially, and the other integrates the two models into a unified model, where acoustic parameters of a speaker can be converted directly to articulatory parameters of another speaker. Experiments show that both methods can improve the mapping accuracy and that the latter method works better than the former method. Especially in the case of velar stop consonants, the mapping accuracy is higher by 0.6 mm. Index Terms: acoustic-to-articulatory mapping, Gaussian mixture model, voice conversion, speaker normalization

The exemplar-based approaches, which model signals as a sparse linear combination of exemplars of signals, are proved to have state-of-the-art performance in noise robust ASR, especially on low SNRs. However, since both the speech exemplars and noise exemplars are built from training data and are fixed throughout the process of enhancing speech features, the conventional approach is especially weak for unknown types of noise. Therefore, in this paper, we propose a semi-supervised approach which automatically adapt noise exemplars to the target noise, while keeping the speech exemplars fixed. Continuous digits recognition experiments show that this approach is much more robust for unknown noise. The recognition errors are reduced by 36.2%.

This paper describes a novel approach to construct a mapping function between a given speaker pair using probability density functions (PDF) of matrix variate. In voice conversion studies, two important functions should be realized: 1) precise modeling of both the source and target feature spaces, and 2) construction of a proper transform function between these spaces. Voice conversion based on Gaussian mixture model (GMM) is the de facto standard because of their flexibility and easiness in handling. In GMM-based approaches, a joint vector space of the source and target is first constructed, and the joint PDF of the two vectors is modeled as GMM in the joint vector space. The joint vector approach mainly focuses on precise modeling of the ‘joint’ feature space, and does not always construct a proper transform between two feature spaces. In contrast, the proposed method constructs the joint PDF as GMM in a matrix variate space whose row and column respectively correspond to the two functions, and it has potential to precisely model both the characteristics of the feature spaces and the relation between the source and target spaces.

In this paper, we propose the use of deep neural networks to expand conventional methods of statistical feature enhancement based on piecewise linear transformation. Stereo-based piecewise linear compensation for environments (SPLICE), which is a powerful statistical approach for feature enhancement, models the probabilistic distribution of input noisy features as a mixture of Gaussians. However, soft assignment of an input vector to divided regions is sometimes done inadequately and the vector comes to go through inadequate conversion. Especially when conversion has to be linear, the conversion performance will be easily degraded. Feature enhancement using neural networks is another powerful approach which can directly model a non-linear relationship between noisy and clean feature spaces. In this case, however, it tends to suffer from over-fitting problems. In this paper, we attempt to mitigate this problem by reducing the number of model parameters to estimate. Our neural network is trained whose output layer is associated with the states in the clean feature space, not in the noisy feature space. This strategy makes the size of the output layer independent of the kind of a given noisy environment. Firstly, we characterize the distribution of clean features as a Gaussian mixture model and then, by using deep neural networks, estimate discriminatively the state in the clean space that an input noisy feature corresponds to. Experimental evaluations using the Aurora 2 dataset demonstrate that our proposed method has the best performance compared to conventional methods.

This paper proposes a new method of estimating perceptual femininity (PF) of an input utterance using Gaussian Mixture Model (GMM) supervectors and support vector regression (SVR). The method is used to develop a femininity estimation tool, which is introduced to voice therapy of Gender Identity Disorder (GID) clients, especially MtF (Male to Female) transsexuals. In our previous study [1], we developed a PF estimator, where a male GMM and a female GMM of spectral features and those of pitch features were built and their likelihood scores of an input utterance were combined by linear regression to estimate PF. In this work, inspired by recent speaker recognition models [2], we replace the four likelihood scores from the four GMMs with supervectors composed by a spectral GMM and a pitch GMM estimated from an input utterance. Further, instead of simple linear regression, we introduce SVR, which is discriminative linear regression. Experiments using an MtF speech corpus show that the proposed method improves correlation between human and machine scores of PF and also reduces squared prediction error.

Artificial Bandwidth Extension (ABE) has been introduced to improve perceived speech quality and intelligibility of narrowband telephone speech. Most of the existing algorithms divided ABE into 2 sub-problems, namely extension of the excitation signal and that of the spectral envelope. In this paper, we propose a new method for spectral envelope extension based on REgularized piecewise linear mapping with DIscriminative region weighting And Long-span features (REDIAL). REDIAL is a revised version of SPLICE, a well-known method for speech enhancement. In REDIAL, however, discriminative model is introduced for space division step of the original SPLICE. The proposed REDIAL-based method approximates non-linear transformation from narrowband features to their wideband counterpart by a summation of piecewise linear transformations. The proposed method was compared with the widely used GMM-based method, through objective and subjective evaluations in both speaker-dependent and speaker-independent conditions. Both evaluations showed that the proposed method significantly outperforms the conventional GMM-based method.

UTokyo Repositoryは本学で生産されたさまざまな学術成果を電子的形態で集中的に蓄積・保存し、世界に発信することを目的としたインターネット上の発信拠点です。 The UTokyo Repository is the system to store and provide digital resources created by members of the University of Tokyo. Its main purpose is to develop digital collections, make them available online, and preserve them for long-term access.

This paper introduces the first online and free framework for teaching and learning Japanese prosody including word accent and phrase intonation. This framework is called OJAD (Online Japanese Accent Dictionary) [1] and it provides three functions. 1) Visual, auditory, systematic, and comprehensive illustration of patterns of accent change (accent sandhi) of verbs and adjectives. Here only the changes caused by twelve kinds of fundamental conjugation are focused upon. 2) Visual illustration of the accent pattern of a given verbal expression, which is a combination of a verb and its postpositional auxiliary words. 3) Visual illustration of the pitch pattern of an any given sentence and the expected positions of accent nuclei in the sentence. The third function is implemented by using an accent change prediction module that we developed for Japanese text-to-speech (TTS) synthesizers [2, 3]. Experiments show that accent nucleus assignment to given texts by the proposed framework is much more accurate than that by native speakers. Subjective assessment and objective assessment by teachers and learners show very high pedagogical effectiveness of the framework. Index Terms: language education, Japanese prosody, accent sandhi, OJAD, speech synthesis, assessment experiments

This work investigates two related issues in the area of WFST-based G2P conversion. The first is the impact that the approach utilized to convert a target word to an equivalent finite-state ma-chine has on downstream decoding efficiency. The second issue considered is the impact that the approach utilized to represent the joint n-gram model via the WFST framework has on the speed and accuracy of the system. In the latter case two novel algorithms are proposed, which extend the work from [1] to enable the use of failure-transitions with joint n-gram models. All solutions presented in this work are available as part of the open-source, BSD-licensed Phonetisaurus G2P toolkit [2]. Index Terms: G2P, WFST, model conversion

This paper proposes a feature enhancement method that can achieve high speech recognition performance in a variety of noise environments with feasible computational cost. As the well-known Stereo-based Piecewise Linear Compensation for Environments (SPLICE) algorithm, the proposed method learns piecewise linear transformation to map corrupted feature vectors to the corresponding clean features, which enables efficient operation. To make the feature enhancement process adaptive to changes in noise, the piecewise linear transformation is performed by using a subspace of the joint space of corrupted and noise feature vectors, where the subspace is chosen such that classes (i.e., Gaussian mixture components) of underlying clean feature vectors can be best predicted. In addition, we propose utilizing temporally adjacent frames of corrupted and noise features in order to leverage dynamic characteristics of feature vectors. To prevent overfitting caused by the high dimensionality of the extended feature vectors covering the neighboring frames, we introduce regularized weighted minimum mean square error criterion. The proposed method achieved relative improvements of 34.2% and 22.2% over SPLICE under the clean and multi-style conditions, respectively, on the Aurora 2 task.

SPLICE is one of the speech enhancement methods based on feature conversion, which shows a high performance with a relatively small amount of calculation. After modeling noisy speech features as GMM, conversion functions are obtained for individual GMM components. The original SPLICE estimates clean feature vectors as a weighted summation of the converted versions of input vectors. Since the conversion functions are determined and fixed only by using training data, the effectiveness of the original SPLICE will be lower in the case of unseen noisy environments. In this paper, we propose a novel method to adapt the conversion functions to work well in unseen environments. First, to realize adaptive conversion functions, we characterize those functions using their super vectors. Then, we conduct PCA on the super vectors to reduce the number of parameters to be adapted. By representing the super vectors through their PCA-based base functions and weights, we implement an efficient adaptation method of conversion functions, which we call Eigen-SPLICE here after. Evaluation experiments show that Eigen-SPLICE has reduced word error rate by 21.0% relative to the conventional SPLICE, and by 24.1% relative to EMS SPLICE in the test set B of the AURORA-2 task.

In this paper, we demonstrate the potential of incorporating syllable-level information in acoustic modeling. The unit of syllable is not rigorously defined, which leads to a problem for its use. In this study, we derive syllable structures from the sonorant-band intensity profile of speech signal. We analyze the error statistics of a phone-based context-dependent speech recognizer and find interesting error patterns. Phone errors mainly occur inside a syllable but not at syllable boundaries. Pronunciation variation can thus be regarded as the replacement of phonetic elements within the time span of a solitary syllable. We apply simple rules to model the pronunciation variation phenomenon. A lexical modeling approach modifies the bi-phone transcription in the dictionary. It leads to a significant increase of phone correctness. The results shed light on a more intuitive and direct approach to model pronunciation variation within the scope of syllables.

This paper presents a multiple kernel learning (MKL) approach to speech/music discrimination (SMD). The time-frequency representation (spectrogram) implemented by short-time Fourier transform (STFT) of audio segment is decomposed by wavelet packet transform into different subband levels. The subbands, which contain rich texture information, are used as features for this discrimination problem. MKL technique is used to select the optimal subbands to discriminate the audio signals. The proposed MKL based algorithm is applied for SMD of a standard dataset. The experimental results show that the proposed technique yields noticeable improvements in classification accuracy and tolerance toward different noise types compared to the existing methods.

One of the most effective approaches to noise robust speech recognition is to remove the noise effect directly from corrupted MFCC vectors. However, VTS enhancement, which is a typical method for performing MFCC enhancement, provides limited improvement when the noise is highly non-stationary. This is because the VTS enhancement method cannot use a time-varying noise model to keep the computational cost at an acceptable level. This paper proposes a method that can enhance MFCC vectors and their dynamic parameters by using noise estimates that change on a frame-by-frame basis at a practical computational cost. The proposed method employs stereo data-based feature mapping like the well known SPLICE algorithm. The novelty of the proposed method lies in that it uses the joint space spanned by a concatenated vector of corrupted and noise features. It is also proposed to use linear discriminant analysis to effectively reduce the dimensionality of the joint space. The proposed method achieves 19.1% and 8.3% relative error reduction from the SPLICE and noise-mean normalized SPLICE algorithms, respectively.

In Japanese, every content word has its own mora-based H/L pitch pattern when it is uttered in isolation, called accent type. When reading out a written sentence, however, this lexical H/L pattern is often changed according to the context, known as word accent sandhi. In our previous work, an accent sandhi predictor was developed using CRF [1], and in this paper, the predictor is improved through feature engineering especially fo-cusing on phrases including numerals and those including loan-words. This is because our previous work showed that the pre-diction performance was relatively low for those phrases. To optimize the features used for CRF, it is critical to take into ac-count the mechanism of word accent sandhi. We review linguis-tic and technical literature that attempted to characterize accent sandhi in the phrases including numerals and loanwords and, by reflecting these characteristics, the features are re-designed. Experiments show that the proposed predictor improved the per-formance relatively by 37 % and 41%, respectively. Index Terms: word accent sandhi, accent nucleus, text-to-speech, Japanese education, rule-based, corpus-based, CRF

Automatic Speech Recognition (ASR) applications often em-ploy a mixture of static and dynamic grammar components, and can thus benefit from the ability to efficiently modify the sys-tem vocabulary and other parameters in an on-line mode. This paper presents a novel, generic approach to dynamic grammar handling in the context of the Weighted Finite-State Transducer (WFST) paradigm. The method relies on a straightforward ex-tension of the lexicon and underlying grammar components, and leverages the ideas of on-the-fly composition and delayed construction to efficiently generate the recognition search space on-the-fly. The alternative partitioning of component models that this approach implies can also result in significant stor-age savings. In contrast to previous works in this area, the proposed method relies only on generic WFST operations and the context-dependency, lexicon and grammar components that form the basis of standard ASR cascades.

This work introduces a modified WFST-based mul-tiple to multiple EM-driven alignment algorithm for Grapheme-to-Phoneme (G2P) conversion, and pre-liminary experimental results applying a Recurrent Neural Network Language Model (RNNLM) as an N-best rescoring mechanism for G2P conversion. The alignment algorithm leverages the WFST framework and introduces several simple structural constraints which yield a small but consistent improvement in Word Accuracy (WA) on a selection of standard base-lines. The RNNLM rescoring further extends these gains and achieves state-of-the-art performance on four standard G2P datasets. The system is also shown to be significantly faster than existing solu-tions. Finally, the complete WFST-based G2P frame-work is provided as an open-source toolkit.

This paper introduces speaker adaptive training techniques to tensor-based arbitrary speaker conversion. In voice conversion studies, realization of conversion from/to an arbitrary speaker’s voice is one of the important objectives. For this purpose, eigen-voice conversion (EVC), which is based on an eigenvoice Gaus-sian mixture model (EV-GMM), was proposed. Although the EVC can effectively construct the conversion model for arbi-trary target speakers using only a few utterances, increase of the utterances used to construct the conversion model does not always improve the conversion performance. This is because the EV-GMMmethod has an inherent problem in representation of GMM supervectors. We previously proposed tensor-based speaker space as a solution for this problem, and realized more flexible control of speaker characteristics. In this paper, to aim larger improvement of the performance of VC, speaker adaptive training and tensor-based speaker representation are integrated. The proposed method can construct the flexible and precise con-version model, and experimental results of one-to-many voice conversion demonstrate the effectiveness of the proposed ap-proach. Index Terms: voice conversion, Gaussian mixture model, eigenvoice, Tucker decomposition, speaker adaptive training

Multimodal speech recognition is a promising approach to realize noise robust automatic speech recognition (ASR), and is currently gathering the attention of many researchers. Multimodal ASR utilizes not only audio features, which are sensitive to background noises, but also non-audio features such as lip shapes to achieve noise robustness. Although various methods have been proposed to integrate audio-visual features, there are still continuing discussions on how the vest integration of audio and visual features is realized. Weights of audio and visual features should be decided according to the noise features and levels: in general, larger weights to visual features when the noise level is low and vice versa, but how it can be controlled? In this paper, we propose a method based on piecewise linear transformation in feature integration. In contrast to other feature integration methods, our proposed method can appropriately change the weight depending on a state of an observed noisy feature, which has information both on uttered phonemes and environmental noise. Experiments on noisy speech recognition are conducted following to CENSREC-1-AV, and word error reduction rate around 24% is realized in average as compared to a decision fusion method.

In recent years Computer-Assisted Language Learning (CALL) systems have been widely used in foreign language education. Some systems use automatic speech recognition (ASR) technologies to detect pronunciation errors and estimate the proficiency level of individual students. When speech recording is done in a CALL classroom, however, utterances of a student are always recorded with those of the others in the same class. The latter utterances are just background noise, and the performance of automatic pronunciation assessment is degraded especially when a student is surrounded with very active students. To solve this problem, we apply a noise reduction technique, Stereo-based Piecewise Linear Compensation for Environments (SPLICE), and the compensated feature sequences are input to a Goodness Of Pronunciation (GOP) assessment system. Results show that SPLICE-based noise reduction works very well as a means to improve the assessment performance in a noisy classroom.

Pronunciation errors are often made by learners of a foreign language. To build a Computer-Assisted Language Learning (CALL) system to support them, automatic error detection is essential. In this study, Japanese learners of Chinese are focused on. We investigated in automatic detection of their typical and frequent phoneme production errors. For this aim, four databases are newly created and we propose a detection method using Support Vector Machine (SVM) with structural features. The proposed method is compared to two baseline methods of Goodness Of Pronunciation (GOP) and Likelihood Ratio (LR) under the task of phoneme error detection. Experiments show that the proposed method performs much better than both of the two baseline methods. For example, the false rejection rate is reduced by as much as 82%. However, the results also indicate some drawbacks of using SVM with structural features. In this paper, we discuss merits and demerits of the proposed method and in what kind of real applications it works effectively.

When dysarthrics, individuals with speaking disabilities, try to communicate using speech, they often have no choice but to use speech synthesizers which require them to type word sym-bols or sound symbols. Input by this method often makes real-time communication troublesome and dysarthric users struggle to have smooth flowing conversations. In this study, we are developing a novel speech synthesizer where speech is gener-ated through hand motions rather than symbol input. In re-cent years, statistical voice conversion techniques have been proposed based on space mapping between given parallel ut-terances. By applying these methods, a hand space was mapped to a vowel space and a converter from hand motions to vowel transitions was developed. It reported that the proposed method is effective enough to generate the five Japanese vowels. In this paper, we discuss the expansion of this system to conso-nant generation. In order to create the gestures for consonants, a Speech-to-Hand conversion system is firstly developed using parallel data for vowels, in which consonants are not included. Then, we are able to automatically search for candidates for consonant gestures for a Hand-to-Speech system. Index Terms: Dysarthria, speech production, hand motions, media conversion, arrangement of gestures and vowels

In many schools, English is taught as international communication tool and the goal of English pronunciation training is generally to acquire intelligible enough pronunciation, which is not always native-sounding pronunciation. However, the definition of the intelligible pronunciation is not easy because it depends on the speaking skill of a speaker, the predictability of a content, and the language background of a listener. One kind of accented pronunciation, which is intelligible enough for some listeners, is often less intelligible for others. This paper focuses on objective intelligibility of Japanese English through the ears of American English speakers with little exposure to Japanese English. A large listening test was conducted using ERJ (English Read by Japanese) database. A balanced subset of this database were presented over a telephone line to the American listeners who were asked to repeat what they heard. Totally, 17,416 repetitive responses were collected and they were transcribed manually. This paper describes the design of this experiment and some results of analyzing the results of transcription. Index Terms: English pronunciation training, foreign accent, intelligibility, listening test, ERJ database

This paper introduces the Transducersaurus toolkit which provides a set of classes for generating each of the fundamental components of a typical WFST ASR cascade, including a Context-dependency transducer, a Lexicon, a stochastic language model and an optional silence class model. The toolkit further implements a simple scripting language in order to facilitate the construction of cascades with a variety of popular combination and optimization methods and provides integrated support for the T 3 and Juicer WFST decoders, and both Sphinx and HTK format acoustic models. New results for two standard WSJ tasks are also provided, comparing a variety of cascade construction and optimization algorithms. These results illustrate the flexibility of the toolkit as well as the tradeoffs inherent in various build algorithms. Index Terms: Speech Recognition, WFST, LVCSR

This paper describes a novel approach to flexible control of speaker characteristics using tensor representation of speaker space. In voice conversion studies, realization of conversion from/to an arbitrary speaker’s voice is one of the important objectives. For this purpose, eigenvoice conversion (EVC) based on an eigenvoice Gaussian mixture model (EV-GMM) was proposed. In the EVC, similarly to speaker recognition approaches, a speaker space is constructed based on GMM supervectors which are high-dimensional vectors derived by concatenating the mean vectors of each of the speaker GMMs. In the speaker space, each speaker is represented by a small number of weight parameters of eigen-supervectors. In this paper, we revisit construction of the speaker space by introducing the tensor analysis of training data set. In our approach, each speaker is represented as a matrix of which the row and the column respectively correspond to the Gaussian component and the dimension of the mean vector, and the speaker space is derived by the tensor analysis of the set of the matrices. Our approach can solve an inherent problem of supervector representation, and it improves the performance of voice conversion. Experimental results of oneto-many voice conversion demonstrate the effectiveness of the proposed approach. Index Terms: voice conversion, Gaussian mixture model, eigenvoice, tensor analysis, Tucker decomposition

This paper presents an improved and efficient method of speech enhancement using non-linear weighted noise subtraction (NWNS) and empirical mode decomposition (EMD). In the first stage, a noise subtraction method is used for enhancement of noisy speech by adopting a non-linear subtractive factor. The factor is estimated by a two stage refinement of estimated degree of noise (EDON). This stage performs well for wideband stationary noises. Most of the real world noise is narrowband non-stationary and EMD is a powerful tool for analyzing non-linear and non-stationary signals like speech. EMD decomposes any signals into a finite number of band limited signals called intrinsic mode functions (IMFs). In the next stage, we apply EMD through a time domain adaptive thresholding strategy adapted to the IMFs. This stage performs well for narrow band non-stationary noises. We use white, pink and high frequency channel noises in order to show the performance of the proposed algorithm.

This paper presents a robust voiced/unvoiced classification method by using linear model of empirical mode decomposition (EMD) controlled by Hurst exponent. EMD decomposes any signals into a finite number of band limited signals called intrinsic mode functions (IMFs). It is assumed that voiced speech signal is composed of trend due to vocal cord vibration and some noise. No trend is present in unvoiced speech signal. A linear model is developed using IMFs of the noise part of the speech signal. Then a specified confidence interval of the linear model is set as the data adaptive energy threshold. If there exists at least one IMF exceeding the threshold and its fundamental period is within the pitch range, the speech is classified as voiced and unvoiced otherwise. The experimental results show that the proposed method performs superior compared to the recently developed voiced/unvoiced classification algorithms with noticeable performance.

This paper presents an efficient pitch estimation algorithm of noisy speech signal using the combination of dominant harmonic modification (DHM) and data adaptive time domain filtering approach. The noisy speech signal is pre- filtered within the range of fundamental frequency. The dominant harmonic (DH) is determined in pre-filtered signal and enhanced the amplitude of DH. Normalized autocorrelation function (NACF) is computed from the modified signal. Empirical mode decomposition (EMD) is applied to the obtained autocorrelation function to implement the data adaptive filtering. Such time domain filtering reduces the effects of the harmonics other than DH in the NACF domain. The signal obtained by the partial reconstruction in EMD domain is effectively used to determine the pitch period. The experimental results show that the proposed one performs better for noisy and clean speech signals.

In this paper, we propose a novel speaker adaptation technique, regularized-MLLR, for Computer Assisted Language Learn-ing (CALL) systems. This method uses a linear combination of a group of teachers ’ transformation matrices to represent each target learner’s transformation matrix, thus avoids the over-adaptation problem that erroneous pronunciations come to be judged as good pronunciations after conventional MLLR speaker adaptation, which uses learners ’ “imperfect ” speech as target utterances of adaptation. Experiments of automatic scor-ing and error detection on public databases show that the pro-posed method outperforms conventional MLLR adaption in pronunciation evaluation and can avoid the problem of over adaptation. Index Terms: Computer Assisted Language Learning (CALL), speaker adaption, pronunciation evaluation, goodness of pro-nunciation (GOP), maximum likelihood linear regression (MLLR) 1.

Automatic pronunciation assessment has several difficulties. Adequacy in controlling the vocal organs is often estimated from the spectral envelopes of input utterances but the envelope patterns are also affected by other factors such as speaker iden-tity. Recently, a new method of speech representation was pro-posed where these non-linguistic variations are effectively re-moved through modeling only the contrastive aspects of speech features. This speech representation is called speech struc-ture. However, the often excessively high dimensionality of the speech structure can degrade the performance of structure-based pronunciation assessment. To deal with this problem, we integrate multilayer regression analysis with the structure-based assessment. The results show higher correlation between hu-man and machine scores and also show much higher robustness to speaker differences compared to widely used GOP-based analysis. Index Terms: CALL, speech structure, regression, GOP 1.

This paper presents a data adaptive technique of cardiovascular disease diagnosis by analyzing electrocardiogram (ECG) signals. The separation of high-frequency QRS and low frequency signal are performed by employing empirical mode decomposition (EMD). Biomedical signals like heart wave commonly change their statistical properties over time, tending to be nonstationary for which EMD is a powerful tool of decomposition. EMD is used to decompose ECG signal into a finite set of band-limited signals termed as intrinsic mode functions (IMFs). Then the low and high frequency components of ECG signals are obtained partial reconstruction intrinsic mode functions and the residual. The related signal processing tools are applied to extract high and low frequency parts to diagnosis the cardiovascular diseases.

This paper focuses on a pitch estimation method of noisy speech signal using the combination of empirical mode decomposition (EMD) and discrete Fourier transform (DFT). The noisy speech signal is filtered within the range of fundamental frequency. Normalized autocorrelation function (NACF) is computed from the pre-filtered noisy speech signal. The NACF is decomposed by EMD to generate a finite number of band limited signal called Intrinsic Mode Function (IMF). DFT is applied to NACF to determine the dominant frequency of the analyzing speech frame. The IMF with fundamental period closest to that of the dominant frequency is selected as the target IMF containing the fundamental period. The performance of the proposed pitch estimation method is compared in terms of gross pitch error (GPE) with the recent algorithms. The experimental results show that the proposed one performs better for noisy and clean speech signals.

Automatic estimation of pronunciation proficiency has its specific difficulty. Adequacy in controlling the vocal organs can be estimated from spectral envelopes of input utterances but the envelope patterns are also affected easily by different speakers. To develop a pedagogically sound method for automatic estimation, the envelope changes caused by linguistic factors and those by extra-linguistic factors should be properly separated. For this aim, in our previous study [1], we proposed a mathematically-guaranteed and linguistically-valid speaker-invariant representation of pronunciation, called speech structure. After the proposal, we have examined that representation also for ASR [2], [3], [4] and, through these works, we have learned better how to apply speech structures to various tasks. In this paper, we focus on a proficiency estimation experiment done in [1] and, based on our recently proposed techniques for the structures, we carry out that experiment again but under new and different conditions. Here, we use smaller units of structural analysis, speaker-invariant substructures, and relative structural distances between a learner and a teacher. Results show that correlations between human and machine rating are improved and also show extremely higher robustness to speaker differences compared to widely used GOP scores. Further, we also demonstrate that the proposed representation can classify learners purely based on their pronunciation proficiency, not affected by their age and gender.

A total corpus-based process of generating prosodic features from text is developed. The process first predicts pauses and phone durations, and then generates F0contours. Since F0contour generation is based on the generation process model, it is rather easy to manipulate the generated F0contours in command level. A method was developed for generating sentence F0contours, when a focus is placed in one of the ldquobunsetsurdquo of an utterance. The method is to predict differences in the F0model commands between with and without focus utterances, and apply them to the F0model commands predicted beforehand by the baseline method. The validity of the method was proved by the experiment on F0contour generation and speech synthesis.

Individuals with speaking disabilities, particularly people suf-fering from dysarthria, often use a TTS synthesizer for speech communication. Since users always have to type sound symbols and the synthesizer reads them out in a monotonous style, the use of the current synthesizers usually renders real-time opera-tion and lively communication difficult. This is why dysarthric users often fail to control the flow of conversation. In this pa-per, we propose a novel speech generation framework which makes use of hand gestures as input. People usually use tongue gesture transitions for speech generation but we develop a spe-cial glove, by wearing which, speech sounds are generated from hand gesture transitions. For development, GMM-based voice conversion techniques (mapping techniques) are applied to esti-mate a mapping function between a space of hand gestures and another space of speech sounds. In this paper, as an initial trial, a mapping between hand gestures and Japanese vowel sounds is estimated so that topological features of the selected gestures in a feature space and those of the five Japanese vowels in a cepstrum space are equalized. Experiments show that the spe-cial glove can generate good Japanese vowel transitions with voluntary control of duration and articulation. Index Terms: Dysarthria, speech production, hand motions, media conversion, arrangement of gestures and vowels

In this paper, we investigate the effects and problems of MLLR speaker adaptation when applied to pronunciation evaluation. Automatic scoring and error detection experiments are conducted on two publicly available databases of Japanese learners’ English pronunciation. As we expected, overadaptation causes misjudge of pronunciation accuracy. Following these experiments, two novel methods, Forced-aligned GOP scoring and Regularized-MLLR adaptation, are proposed to solve the adverse effects of MLLR adaption. Experimental results show that the proposed methods can better utilize MLLR adaptation and avoid over-adaptation. Index Terms: Computer Assisted Language Learning (CALL), speaker adaption, pronunciation evaluation, goodness of pronunciation (GOP), maximum likelihood linear regression (MLLR)

In China, there are hundred kinds of dialects. By traditional dialectology, they are classified into seven big dialect regions and most of them also have many sub-dialects and sub-subdialects. As they are different in various linguistic aspects, people from different dialect regions often cannot communicate orally. But for the sub-dialects of one dialect region, although they are sometimes still mutually unintelligible, more common features are shared. In this paper, a dialect pronunciation structure, which has been used successfully in dialectbased speaker classification in our previous work [1], is examined for the task of speaker classification and distance measurement among cities based on sub-dialects of Mandarin. Using the finals of the dialectal utterances of a specific list of written characters, a dialect pronunciation structure is built for every speaker in a data set and these speakers are classified based on the distances among their structures. Then, the results of classifying 16 Mandarin speakers based on their sub-dialects show that they are linguistically classified with little influence of their age and gender. Finally, distances among sub-sub-dialects are similarly calculated and evaluated. All the results show high validity and accordance to linguistic studies. Index Terms: Sub-dialects of Mandarin, pronunciation structure, speaker classification

One of the most challenging problems in speech recognition is to deal with inevitable acoustic variations caused by nonlinguistic factors. Recently, an invariant structural representation of speech was proposed [1], where the non-linguistic variations are effectively removed though modeling the dynamic and contrastive aspects of speech signals. This paper describes our recent progresses on this problem. Theoretically, we prove that the maximum likelihood based decomposition can lead to the same structural representations for a sequence and its transformed version. Practically, we introduce a method of discriminant analysis of eigen-structure to deal with two limitations of structural representations, namely, high dimensionality and too strong invariance. In the 1st experiment, we evaluate the proposed method through recognizing connected Japanese vowels. The proposed method achieves a recognition rate 99.0%, which is higher than those of the previous structure based recognition methods [2, 3, 4] and word HMM. In the 2nd experiment, we examine the recognition performance of structural representations to vocal tract length (VTL) differences. The experimental results indicate that structural representations have much more robustness to VTL changes than HMM. w Moreover, the proposed method is about 60 times faster than the previous ones. Index Terms: Speech recognition, invariant structure, PCA, discriminative analysis

Several experiments have been carried out that revealed weaknesses of the current Text-To-Speech (TTS) systems in their emotional expressivity. Although some TTS systems allow XML-based representations of prosodic and/or phonetic variables, few publications considered, as a pre-processing stage, the use of intelligent text processing to detect affective information that can be used to tailor the parameters needed for emotional expressivity. This paper describes a technique for an automatic prosodic parameterization based on affective clues. This technique recognizes the affective information conveyed in a text and, accordingly to its emotional connotation, assigns appropriate pitch accents and other prosodic parameters by XML-tagging. This pre-processing assists the TTS system to generate synthesized speech that contains emotional clues. The experimental results are encouraging and suggest the possibility of suitable emotional expressivity in speech synthesis.

This paper describes a new and improved method for the frame-work of structure to speech conversion we previously proposed. Most of the speech synthesizers take a phoneme sequence as input and generate speech by converting each of the phonemes into its corresponding sound. In other words, they simulate a human process of reading text out. However, infants usually acquire speech communication ability without text or phoneme sequences. Since their phonemic awareness is very immature, they can hardly decompose an utterance into a sequence of phones or phonemes. As developmental psychology claims, in-fants acquire the holistic sound patterns of words from the utter-ances of their parents, called word Gestalt, and they reproduce them with their vocal tubes. This behavior is called vocal im-itation. In our previous studies, the word Gestalt was defined physically and a method of extracting it from a word utterance was proposed. We already applied the word Gestalt to ASR, CALL, and also speech generation, which we call structure to speech conversion. Unlike reading machines, our framework simulates infants ’ vocal imitation. In this paper, a method for improving our speech generation framework based on a struc-tural cost function is proposed and evaluated. Index Terms: speech synthesis, the structural representation, vocal imitation, a structural cost function

Recently, a novel and structural representation of speech was proposed [1, 2], where the inevitable acoustic variations caused by nonlinguistic factors are effectively removed from speech. This structural representation captures only microphone- and speaker-invariant speech contrasts or dynamics and uses no absolute or static acoustic properties directly such as spectrums. In our previous study, the new representation was applied to recognizing a sequence of isolated vowels [3]. The structural models trained with a single speaker outperformed the conventional HMMs trained with more than four thousand speakers even in the case of noisy speech. We also applied the new models to recognizing utterances of connected vowels [4]. In the current paper, a multiple stream structuralization method is proposed to improve the performance of the structural recognition framework. The proposed method only with 8 training speakers shows the very comparable performance to that of the conventional 4,130-speaker triphone-based HMMs.

IN this paper, we prove that the direction of cepstrum vectors strongly depends on vocal tract length and that this dependency is represented as rotation in the n dimensional cepstrum space. In speech recognition studies, vocal tract length normalization (VTLN) techniques are widely used to cancel age- and gender-differences. In VTLN, a frequency warping is often carried out and it can be implemented as a linear transformation in a cepstrum space; c = Ac. However, the geometric properties of this transformation matrix A have not been well discussed. In this study, its properties are made clear using n dimensional geometry and it is shown that the matrix rotates any cepstrum vector similarly and apparently. Experimental results using resynthesized speech demonstrate that cepstrum vectors extracted from a speaker of 180 [cm] in height and those from another speaker of 120 [cm] in height are reasonably orthogonal. This result makes clear one of the reasons why children's speech is very difficult for conventional speech recognizers to deal with adequately.

In foreign language learning, shadowing has been used as a method for improving speaking and listening ability. In this method, learners are required to repeat a presented native utterance as closely and quickly as possible. Since learners have to follow the speaking rate of the presented utterance, their pronunciation often becomes very inarticulate and unintelligible. These features of shadowing make it very difficult to build a reliable scoring system for shadowing productions. In this paper, two techniques are proposed and investigated for automatic scoring of shadowing productions. Experiments show that good correlations are found between automatic scores and TOEIC overall proficiency scores. Index Terms: shadowing, automatic scoring, articulatory effort, goodness of pronunciation, bottom-up clustering

Abstract This paper presents a method of voiced/unvoiced (V/Uv) classification of noisy speech signals. Empirical mode decomposition (EMD), a newly developed tool to analyze nonlinear and non-stationary signals is used to filter the additive noise with the speech signal. The normalized autocorrelation of the filtered speech signal is computed to enhance the periodicity if any. It is considered that the voiced speech signal is periodically correlated and the unvoiced signal is not. A statistical model of determining periodic correlation is used to differentiate voiced and unvoiced speech with low SNR. The experimental results show that the use of EMD improves the classification performance and the overall efficiency is noticeable as compared to other existing algorithms. Index Terms : empirical mode decomposition, normalized autocorrelation, periodic correlation, voiced/unvoiced speech 1. Introduction Reliable classification of short time speech signal into voiced and unvoiced is a crucial preprocessing step in many speech processing applications and is essential in most analysis and synthesis system. For example: different strategy could be adopted for voiced and unvoiced parts in speech enhancement using spectral subtraction. The essence of classification is to determine whether the speech production system involves the vibration of the vocal cords [1]. The discrimination problem is an important one and has been worked on extensively during the last three decades [2]. The discrimination can effectively be performed using a single feature or parameter which is closely associated with the voicing and non-voicing activities of speech signal. Many algorithms have been reported for solving the detection problem [3] – [7]. In [3], Gaussian mixture model with cepstrum coefficients features is proposed for robust V/Uv classification. A higher order statistics (HOS) based method is proposed in [4] for V/Uv detection and pitch estimation simultaneously. The matching pursuit algorithm is used in [5] with Gabor decomposition. The wavelet transform is proposed in pitch and V/Uv detection in [6]. A statistical model applied in autocorrelation domain is also reported in [7]. In most of the existing algorithms are not so much noise robust and also the intensive threshold and training data are required for classification. Such requirements are troublesome for the use in application domain. The proposed method is noise robust and based on the statistical model for periodicity detection in speech signal without any training requirement. To reduce the effect of noise on speech signal, a data adaptive time domain filtering is proposed using newly developed empirical mode decomposition method [8]. Although speech signal is non-stationary in nature, Fourier based frequency domain filtering assumes that it is piecewise stationary. The speech decomposition is performed by fitting some predefined bases without satisfying its non-stationary nature. Whereas, EMD based approach decompose the speech signal as non-stationary time series and hence better performance in noise filtering. A method for determining whether an observed time series contains a periodically correlated sequence is employed here. It is based on the statistical tests for the coherence between spectral components for the presence of a periodically correlated covariance structure in a time series [9]. The autocorrelation function (ACF) makes the periodicity more prominent if any. The proposed periodic correlation model is applied in the autocorrelation domain rather than original time domain of the speech signal. The periodicity detection method is implemented in spectral domain to classify the speech segment into voiced or unvoiced one based on that it contains periodic correlated sequence or not respectively.

This paper proposes a new framework of speech generation by imitating “infants ’ vocal imitation”. Most of the speech synthe-sizers take a phoneme sequence as input and generate speech by converting each of the phonemes into a sound sequentially. In other words, they simulate a human process of reading text out. However, infants usually acquire speech generation abil-ity without text or phoneme sequences. Since their phonemic awareness is very immature, they can hardly decompose a word utterance into a sequence of phones. In this situation, as devel-opmental psychology states, infants acquire the holistic sound pattern of words from the utterances of their parents, called word Gestalt, and they reproduce it with their vocal tubes. This behavior is called vocal imitation. In our previous studies, the word Gestalt was defined physically and a method of extract-ing it from an utterance was proposed and used successfully for ASR and CALL. In this paper, a method of converting the word Gestalt back to speech is proposed and evaluated. Unlike a read-ing machine, our proposal simulates infants ’ vocal imitation. Index Terms: speech synthesis, vocal imitation, word Gestalt, invariant structure, Bhattacharyya distance, searching problem

In speech recognition studies, vocal tract length normalization (VTLN) techniques are widely used to cancel age- and gender-difference. In VTLN, the distortion is often modeled as a lin-ear transform in a cepstrum space; ĉ=Ac. In our previous study, the geometrical properties ofA were discussed and it was shown that the matrix can be approximated as rotation matrix. In this study, a new method of better approximating A is pro-posed. Using eigenvalues ofA, its quasi-rotational distortion is factorized into multiple rotation operations and multiple magni-fication operations. Using this method, the intrinsic ambiguity of the rotation angle used in our previous study is resolved. In-stead, multiple rotation angles are introduced to understand bet-ter what kind of geometrical distortionsA induces to cepstrum vectors. Experiments show the validity of the new method and a new speech feature is also derived by the new method. Index Terms: frequency warping, rotation matrix, vocal tract length, eigenvalue, rotational plane

Mandarin speech synthesis was conducted by generating prosodic features by the proposed method and segmental features by HMM-based method. The proposed method generates sentence fundamental frequency (F0) contours by representing them as a superposition of tone components on phrase components. The tone components are realized by concatenating their fragments at tone nuclei predicted by a corpus-based method, while the phrase components are generated by rules under the generation process model (F0model) framework. The method includes prediction of phoneme/pause durations in a statistical method as the first step. Through a listening test on the quality of synthetic speech, it was shown that a better quality was obtainable by the method as compared to that by the full HMM-based method. It was also shown that a better quality is obtainable as compared to the case of generating F0contours without super-positional scheme.

The major difficulty of prosody modeling and automatic tone recognition of continuous Mandarin speech is the complex interaction of tones and prosody/intonation on FO contours. In this study, we propose a latent prosody model (LPM) aiming to jointly model the affections of tone and prosody state on FO. The main purposes are twofold including (1) automatic prosody state labeling and (2) improving tone recognition accuracy. The basic idea is to introduce latent prosody state variables into an additive statistic model of FO which already considers the affecting factors of tone and speaker. Experiments on the Tree-Bank corpus showed that LPM not only gave meaningful prosody state labeling results but also improved the average tone recognition rate from 80.86% of a multi-layer perceptron (MLP) baseline to 82.55%.

This paper describes the development of an estimator of perceptual femininity (PF) of an input utterance using speaker recognition techniques. The estimator was designed for its clinical use and the target speakers are gender identity disorder (GID) clients, especially MtF (male to female) transsexuals. The voice therapy for MtFs is composed of three kinds of training; 1) raising the baseline F0range, 2) changing the baseline voice quality, and 3) enhancing fo dynamics to produce an exaggerated intonation pattern. The first two focus on static acoustic properties of speech and the voice quality is mainly controlled by size and shape of the articulators, which can be acoustically characterized by the spectral envelope. Gaussian mixture models (GMM) of fo values and spectrums were built separately for biologically male speakers and female ones. Using the four models, PF was estimated automatically for each of 142 utterances of 111 MtFs. The estimated values were compared with the PF values obtained through listening tests. Results showed very high correlation (R=0.86), which is comparable to the intra-rater correlation.

Speech acoustics vary due to differences in gender, age, microphone, room, lines, and a variety of factors. In speech recognition research, to deal with these inevitable non-linguistic variations, thousands of speakers in different acoustic conditions were prepared to train acoustic models of individual phonemes. Recently, a novel representation of speech dynamics was proposed [1, 2], where the above non-linguistic factors are effectively removed from speech as if pitch information is removed from spectrum by its smoothing. This representation captures only speaker- and microphone-invariant speech dynamics and no absolute or static acoustic properties such as spectrums are used. With them, speaker identity has to remain in speech representation. In our previous study, the new representation was applied to recognizing a sequence of isolated vowels [3]. The

This paper introduces a novel speech enhancement method based on Empirical Mode Decomposition (EMD) and softthresholding algorithms. A modified soft thresholding strategy is adapted to the intrinsic mode functions (IMF) of the noisy speech. Due to the characteristics of EMD, each obtained IMF of the noisy signal will have different noise and speech energy distribution, thus will have a different noise variance. Based on this specific noise variance, by applying the proposed thresholding algorithm to each IMF separately, it is possible to effectively extract the existing noise components. The experimental results suggest that the proposed method is significantly more effective in removing the noise components from the noisy speech signal compared to recently reported techniques. The significantly better SNR improvement and the speech quality prove the superiority of the proposed algorithm. Index Terms: speech enhancement, empirical mode decomposition, soft-thresholding

We used a command-response additive F0 model to analyze F0 patterns of Japanese spoken by native speakers of Mandarin Chinese. Compared to native speakers of Japanese, we found that Chinese speakers exhibit the following characteristics: (a) higher pitch, (b) more phrases, (c) bunsetsu decomposition, and (d) utterance-final plunging. These characteristics physically manifest themselves as: (a) higher baseline F0, (b) more phrase commands, (c) more accent commands, and (d) negative commands. These characteristics may be subjectively perceived as: (a) tinnier speech (possible L1 marker but does not degrade communication), (b) disjoint phrases (requires mental consolidation), (c) choppy prosodic words (requires reconstruction), and (d) abrupt utterance termination (possibly misconstrued as emphatic or rude). We believe these difficulties arose from tonal and syllable-timed interference, which can be overcome by prosodic control and planning. Index Terms: F0 contour, command-response model, L2 learning, Japanese, accent, phrase.

A total scheme of generating prosodic features from a text input was constructed. The method consists of corpus-based prediction of pauses, phone durations and fundamental frequencies (F0's), in this order, and information predicted in an earlier process is utilized in the following processes. Since prediction of F0's is done on the command values of F0 contour generation process model instead of direct F0 values, a stable and flexible control of F0 contours is possible. By adding constraints on the accent command timings as a post processing, a better quality was realized when speech was synthesized using prosodic features generated by the method. Validity of the developed method was confirmed through the listening test of the synthetic speech. Index Terms: speech synthesis, prosodic features, F0 contour 1.

Speaker-invariant structural representation of speech was proposed [1], where only the phonic contrasts between speech sounds were extracted to form their external structure. The acoustic substances were completely discarded. Considering a mapping function between speaker A’s acoustic space and B’s space, the speech dynamics was mathematically proven to be invariant between the two irrespective of the form of the function [2]. This structural and dynamic representation was applied to describe the pronunciation of learners [3]. Since the nonlinguistic factors were removed effectively, the representation could highlighted the non-nativeness in the individual pronunciations. For vowel learning, it was automatically estimated for each of the learners which vowels to correct by priority [4]. Unlike the conventional approach, the estimation was done without the direct use of sound substances such as spectrums. In this paper, using the vowel charts of the learners plotted by an expert phonetician, the validity of this contrastive or relative approach is examined by comparing it with the conventional absolute approach. Results show the high validity of the proposed method. Index Terms: phonic contrasts, pronunciation structure, CALL

This paper presents a pitch estimation method of noisy speech signal using empirical mode decomposition (EMD). The normalized autocorrelation function (NACF) of the noisy speech signal is decomposed into a finite set of band-limited signals termed as intrinsic mode functions (IMFs) using EMD. The periodicity of one IMF is supposed to be equal to the accurate pitch period. A conventional autocorrelation based pitch period detection method is used to select the IMF with pitch period. The accurate pitch period is obtained from the selected IMF. The pitch estimation performance in term of gross pitch error (GPE) of the proposed algorithm is compared with recently proposed methods. The experimental results show that the EMD based algorithm performs better in pitch estimation of noisy speech. Index Terms: empirical mode decomposition, pitch estimation, normalized autocorrelation.

Due to recent advancements in speech technologies, a large number of spoken dialogue systems have been constructed. However, since most of them adopt existing text-to-speech syn-thesizers, it is rather difficult to reflect the linguistic informa-tion obtained during the reply sentence generation well in out-put speech. A framework is necessary for correctly reflecting higher-level linguistic information, such as syntactic structure and discourse information. We have constructed a spoken dia-logue system on road guidance and realized concept-to-speech conversion, where output speech is generated in a unified pro-cess. Tag LISP forms keep the syntactic structures throughout the process in order to reflect the linguistic information in the prosody of output speech. Furthermore, by making it possible to insert not only words but also phrase templates in tags, various sentences were generated with a minor increase of templates. Validity of the methods is shown through experiments. Index Terms: spoken dialogue system, concept-to-speech con-version 1.

Syntactic and discourse boundaries are signalled by prosodic cues as well as linguistic cues in speech. We investigated whether there is a correspondence between prosodic or linguistic cues and the boundary strengths. We measured the rates of filled pauses (FPs) and conjunctions, and the durations of silent and filled pauses and conjunctions at four types of boundaries in casual presentations in Japanese. The results showed that the rates of FPs and conjunctions and the durations of silent pauses correspond to the boundary strengths. However, no significant correspondence was found between the duration of FPs or conjunctions and the boundary strengths. The results suggest that how long the speaker pauses and whether he or she utters a FP or a conjunction is relevant to the boundary strengths. However, the durations of FPs and conjunctions are likely to be affected by the other factors such as planning difficulties of the following parts of speech. Index Terms: syntactic boundary, discourse boundary, silent pause, filled pause, conjunction

A novel technique is developed to separate the audio sources from a single mixture. The method is based on decomposing the Hilbert spectrum (HS) of the mixed signal into independent source subspaces. Hilbert transform combined with empirical mode decomposition (EMD) constitutes HS, which is a fine-resolution time-frequency representation of a nonstationary signal. The EMD represents any time-domain signal as the sum of a finite set of oscillatory components called intrinsic mode functions (IMFs). After computing the spectral projections between the mixed signal and the individual IMF components, the projection vectors are used to derive a set of spectral independent bases by applying principal component analysis (PCA) and independent component analysis (ICA). A k-means clustering algorithm based on Kulback-Leibler divergence (KLd) is introduced to group the independent basis vectors into the number of component sources inside the mixture. The HS of the mixed signal is projected onto the space spanned by each group of basis vectors yielding the independent source subspaces. The time-domain source signals are reconstructed by applying the inverse transformation. Experimental results show that the proposed algorithm performs separation of speech and interfering sound from a single mixture

Speech acoustics varies from speaker to speaker, microphone to microphone, etc. Recently, a novel method was proposed to separate these static non-linguistic features from speech as spectral smoothing can separate pitch information from speech (N. Minematsu, 2005). Absolute properties of speech events, such as formants and spectrums, are completely discarded and only the phonic differences or contrasts between the events are extracted to form their external structure. This structure is called the acoustic universal structure and regarded as physical implementation of structural phonology because the structure is considered to represent only the linguistic and para-linguistic information. In this paper, the structural size is focused on and its correlation with the para-linguistic information is examined. Results showed that the size can be interpreted as magnitude of articulatory efforts made in speech production

This paper presents a method of audio signal separation from stereo mixtures using binary masking in time-frequency (TF) domain based on the spatial location of the audio sources. The TF representation of audio signal is obtained by Hubert spectrum (HS). The Hubert transformation together with empirical mode decomposition (EMD) produces HS which is a fine-resolution TF representation of any nonlinear and non-stationary signal. The sources are localized in the space of time and intensity differences between two microphones' signals. The separation is performed by masking the target signal in TF domain considering that the sources are disjoint orthogonal. The experimental results of the proposed method show a noticeable improvement of separation efficiency

Perception experiments suggest that natives judge non-native unfilled pauses as indiscriminate and indecisive. Multiple regression analyses of unfilled pauses indicate a connection between syntactic structure and pause location and duration. Native speakers uniformly pause at large syntactic breaks with marked duration, whereas non-natives' unfilled pauses are spread over various locations, possibly reflecting limited syntactic planning. Our method might be used to synthesize appropriate unfilled pauses in text-to-speech systems, and to train pausing behavior in automated pronunciation learning systems for non-native learners. Index Terms: unfilled pauses, second language learning, syntactic structure, perception experiments, multiple regression analyses

We formerly conducted emotional speech synthesis using our corpus-based method of generating fundamental frequency (F0) contours from text. The method predicts command values of F0 contour generation process model instead of directly predicting F0 value of each time frame. A better control of F0 contours was realized by taking the emotional level of each bunsetsu into account: adding information on which bunsetsu(s) the emotion is especially placed to the command predictor inputs. In the case of anger, F0 contours closer to the target contours are obtained by adding emotional levels. Speech synthesis was conducted by generating F0 contours in two ways: using commands predicted by taking emotional levels into account and those not. The result of perceptual experiment indicated that emotion was conveyed well by adding emotional levels. Index Terms: speech synthesis, emotion, F0 contour 1.

A method is developed for recognizing lexical tone types of Standard Chinese syllables in continuous speech. Neural network (four-layered perceptron) is adopted as classifier. The method includes two steps; first recognizing tone types using prosodic features of voiced part, and then re-recognizing by viewing only on tone nucleus, which is a portion of the syllable showing rather stable fundamental frequency (F0) contour regardless of tone types of the preceding and following syllables. The voiced part (or tone nucleus) is divided into 20 segments, and F 0, delta-F 0, F 0 slope and short-term energy of each segment are served as inputs to the neural network. In order to cope with tone coarticulation, prosodic feature parameters for the last 5 segments of the preceding syllable and the initial 5 segments of the following syllable are included in the neural network inputs. Information on syllable length is also added to the inputs. Tone recognition experiment was conducted for a female speaker's utterances included in HKU96 corpus. The average recognition rate was 86.5 % including neutral tone syllables, when the tone nucleus model was not used. It increased to 86.9 %, when the model was used. The obtained rate is higher by more than 3 points as compared to that obtained by the hidden-Markov-model-based tone recognizer developed by the authors formerly. Index Terms: tone recognition, tone nucleus model, neural network, Standard Chinese

A program for self assessment of Japanese pronunciation by Englishspeaking learners was developed using a language model built with input from a language teacher in collaboration with speech engineers. This collaboration enhanced the program's capacity for accurate assessment and provides practical support to users by linking evaluation with feedback, and an editorial function of error patterns. The program drew positive responses from participants in a trial run. This paper discusses the development of our language model, the function and evaluation of this self assessment program.

Disfluencies are intrinsic in spontaneous speech. Although it is known that there is a wide range of frequencies and types of disfluencies among speech, little is known about factors affecting speakers∂ choice of disfluency types. We first conducted a correspondence analysis using ratios of seven types of fillers, and other disfluencies, in 174 presentations and quantified the data. We conducted a cluster analysis using 4 dimension scores from the correspondence analysis, and extracted five filler-type groups. We then examined frequent types of presentations (formal or casual) and speaker attributes (gender and age) in each group. The results indicate that speakers∂choice of filler types is affected by speech levels, speakers∂gender and age, and that relevant factors differ depending on the type of fillers. Index Terms: disfluency type, fillers, sociolinguistic factors, speaker variation, Japanese

In this paper, a localization based approach of audio signal separation from binary mixtures is carried out. The audio sources are localized in the spatial domain (azimuth plane) using the delay and amplitude variation cues between two microphones' signals. A coherence based technique is introduced here to localize the audio sources in adverse acoustical environment. The mixture signals are decomposed into a desired number of sub-bands with empirical mode decomposition (EMD) which is a data adaptive filtering scheme suitable for nonlinear and non-stationary signals. Data independent minimum variance beamforming is employed to separate the component sources in underdetermined condition (more sources than sensors). The experimental results of the proposed algorithm show noticeable separation efficiency. It is also found that the sub-band implementation improves the performance compared with and full-band approach

This paper applies the structural representation of the pronunciation for computer-aided language learning (CALL). This representation was proposed to remove non-linguistic features such as age, gender, speaker, etc from speech acoustics (N. Minematsu et al., 2005). The removal was performed by extracting only the interrelations of speech events and discarding their absolute properties such as formants and spectrum envelopes. All the extracted interrelations mathematically form the external phonological structure of the events. Using this representation, in S. Asakawa et al., (2005), the vowel structure of a language learner was extracted and it was shown that the structural development via training can be traced and visualized adequately. This structural visualization can be regarded as pronunciation portfolio (N. Minematsu et al., 2004). This paper shows that the new representation can classify the language learners adequately and indicate which vowels should be corrected by priority.

Emphasis and question are two factors that have significant effects on F/sub 0/ contours for various languages, among which tone languages require more careful study because their F/sub 0/ contours show complex interaction between lexical tones and sentence intonation. This paper employs the command-response model for the process of F/sub 0/ contour generation to investigate the effects of these two factors for Cantonese, a typical tone language with nine tones. Analysis shows that the major effect of emphasis is on phrase commands, whereas the polarity and the amplitude of the tone commands in the emphasized part are hardly affected so that the inherent tone command patterns are maintained. In the intonation question, the inherent tone command in the later part of the sentence-final syllable is always substituted by a positive tone command. The particle question, on the other hand, maintains the inherent tone command for the question particle. In both types of questions, a sentence-final phrase command is added or enhanced, and a particular ending tone command is attached, the amplitude of which can indicate the degree of inquisitive intention. By comparison, the effect of emphasis starts from the target part for emphasis but is not confined to it, whereas the effect of question is localized in the sentence-final part and especially concentrated within the ending syllable. Nevertheless, both of them can be represented in the framework of the command-response model, by which F/sub 0/ contours for expressive speech can be generated efficiently.

Although in most spoken dialogue systems, text-to-speech conversion devices are used for reply speech generation. However, use of such devices makes it difficult to well reflect higher-level linguistic (and para-/non- linguistic) information obtainable during sentence generation process on reply speech. This situation degrades the reply speech quality mainly from the aspect of prosodic features. A method is necessary to directly converting content of reply into speech. This method, known as concept-to-speech conversion, was realized for the reply speech generation in our spoken dialogue system on road guidance. It is an improved version of our formerly developed one for an agent dialogue system. Reply sentence generation was conducted by pasting words and/or phrases at tag positions of a sentence frame, which was prepared in a tag-LISP form. In order to realize the concept-to-speech conversion, syntactic structure of phrases in user's inputs is kept and is utilized for the sentence generation. Several improvements, such as prosodic phrase boundary positioning using probability of word sequences, are also added to prosodic control in speech synthesis. In the spoken dialogue system, a user was guided to reach a place marked on a map through conversation. Several schemes on dialogue management were implemented to solve the problems caused due to the imperfect information on the roads given to the user and the system. A trial use of the system showed that a smooth conversation between the user and the system was possible. The result clearly indicated a better prosodic control for the newly developed method as compared to the original method.

Cantonese is a well-known Chinese dialect with a quite complex tone system. We have successfully applied the command-response model to represent F/sub 0/ contours of Cantonese speech by defining a set of appropriate tone command patterns. It provides an efficient means to describe Cantonese F/sub 0/ contours with high accuracy. Both qualitative and quantitative descriptions of the command patterns of nine tones are presented. The various cues for identifying each tone are investigated, based on which a set of rules is derived to synthesize F/sub 0/ contours of Cantonese. Perceptual experiments have been conducted to test the validity of the rules for each tone and to evaluate the naturalness of synthetic speech based on those rules.

Acoustic representation of speech provided by phonetics, spectrogram, is noisy representation in that it shows every acoustic aspect of speech. Age, gender, shape, microphone, room, line, etc. are completely irrelevant to the pronunciation assessment. However, the spectrogram is affected inevitably by these factors. Recently, a novel acoustic representation of speech was proposed, where dimensions of these non-linguistic factors can hardly be seen[1, 2]. Every acoustic substance of speech is discarded and only their interrelations are extracted to represent the pronunciation structurally. Using this method, individual learners were described as distorted phonemic structures[3] and automatic scoring of the pronunciation was investigated[3, 4]. This paper describes two new analyses using the proposed method. The first analysis is done to examine whether the method can trace the development of a student’s pronunciation appropriately using only a limited amount of speech. The second one focuses on the prosodic aspect of the pronunciation, especially stressed and unstressed vowels. The former indicates that the proposed method can show history of the student’s development adequately and the latter clarifies that size of the pronunciation structure is highly correlated with the pronunciation proficiency. 1.

Word emphasis and question intonation have significant effects on F0 contours for various languages. This paper employs the command-response F0 model to investigate the effects of the two factors for Cantonese, a typical tone language with nine tones. Analysis shows that the main effect of word emphasis associated with narrow focus is not on tone command but on phrase command. An additional or a larger phrase command tends to occur immediately preceding the emphasized word. For the tone command of the emphasized word, duration may be lengthened but polarity and amplitude are hardly affected so that the inherent tone patterns are kept. Also, echo question will replace the tone command in the later part of the sentence-final syllable by a positive tone command of much larger amplitude than usual. Both the effects of the two factors can be embedded in the framework of our model.

A corpus-based method was developed for automatic extraction of the F0 contour generation process model parameters (phrase and accent commands). The method first smoothes the observed F0 contour by a piecewise 3rd order polynomial function and finds points of inflection. Then several parameters related to the points are used as input parameters for the predictor of the model commands. Finally the predicted commands are tuned to the observed contour by the analysis-by-synthesis. An experiment was conducted using ATR 504 sentence speech corpus, and the performance close to a rule-based method, also developed by the authors, was obtained. An experiment was further conducted by adding linguistic information of the content of the utterance (such as accent type, depth of bunsetsu boundary) to input parameters. The performance was largely improved; the extraction rates reached around 90 % for phrase commands and 84 % for accent commands.

Abstract This paper presents a content-based approach of audio source indexing without any prior knowledge about the sources. The empirical mode decomposition ( EMD ) scheme, capable of decomposing nonlinear and non-stationary signals into some bases, is employed to implement the sub-band approach of the audio discrimination technique. The feature vectors are derived from each of selected sub-bands of the target frame. Linear predictive cepstrum coefficient ( LPCC ) is used as the main feature vector and Kullback-Leibler divergence ( KLd ) is performed as the scoring function to measure the similarity of the feature vectors. The higher order statistics ( HOS ) is employed to compute the LPCC . The use of HOS makes LPCC less affected by Gaussian noise. The experimental results show that the sub-band approach produces better discrimination efficiency than that of the full-band technique. This discrimination method is also suitable to solve the source permutation ambiguity in separation of multiple and concurrent moving sources from the mixture(s).

Speech acoustics varies from speaker to speaker, microphone to microphone, room to room, line to line, etc. Physically speaking, every speech sample is distorted. Socially speaking, however, speech is the easiest communication media for humans. In order to cope with the inevitable distortions, speech engineers have built HMMs with speech data of hundreds or thousands of speakers and the models are called speaker-independent models. But they often need to be adapted to the input speaker or environment and this fact claims that the speaker-independent models are not really speaker-independent. Recently, a novel acoustic representation of speech was proposed, where dimensions of the above distortions can hardly be seen. It discards every acoustic substance of speech and captures only their interrelations to represent speech acoustics structurally. The new representation can be interpreted linguistically as physical implementation of structural phonology and also psychologically as speech Gestalt. In this paper, the first recognition experiment was carried out to investigate the performance of the new representation. The results showed that the new models trained from a single speaker with no normalization can outperform the conventional models trained from 4,130 speakers with CMN.

This paper presents a method for quantitatively estimating intonation variation in Mandarin speech. Intonation variation is relative to identical lexical tone structures, and its estimation is performed on two sets of fundamental frequency ( ) contours: one for norms and the other as variants. This is done by transforming target values in pairs from the norms to the variants in which the prosodic contribution to these contours is analyzed as sequences of targets, all of which are confined to the basic elements of the underlying lexical tone structures. The tone transformations are constrained under an assumption of the structural formulation of contours proposed previously. When the norms take the base values of the four lexical tones measured from isolated words in a neutral mood and voice, this method solves acoustic correlations of tone and intonation from the observed contours. The method was implemented on a computer, and its capability of estimating intonation variation was shown through the analysis and synthesis of contours.

A new method for generating sentence F 0 contours of Mandarin speech is proposed.The method assumes the F 0 contour generation process model, but generates the tone and phrase components in different ways and sums them to produce a sentence F 0 contour.The tone component is generated concatenating F 0 patterns of tone nuclei, which are predicted by a corpus-based scheme (binary decision trees).Experiments of F 0 contour generation were conducted by using 100 news utterances by a female speaker.The results showed that the method could generate F0 contours close to those of target speech.A perceptual evaluation was also conducted on the synthetic speech using the F0 contours generated by the method.An average score of 4.5 in a 5-point scale indicates the high naturalness, verifying the validity of the method.

Corpus based studies of spontaneous speech showed that filled pauses tended to precede relatively long and complex constituents. We examined whether listeners made use of such a tendency in speech processing. We tested the hypothesis that when listeners heard filled pauses they tended to expect a relatively long and complex phrase to follow. In the experiment participants listened to sentences referring to both simple and compound shapes presented on a computer screen. Their task was to press a button as soon as they had identified the shape that they heard. The sentences involved two factors: complexity and fluency. As the complexity factor, a half of the sentences described compound � shapes with long and complex phrases and�the other half described simple shapes with short and simple phrases.�As the fluency factor phrases describing a shape had a preceding filled pause, a preceding silent pause of the same length�as the filled pause, or no preceding pause. The results showed that response times for the complex phrases were significantly shorter after�filled or silent pauses than when there was no pause. In contrast, there was no significant difference between the three conditions for the simple phrases. The results support the hypothesis and indicate that it is the duration of filled pauses that give listeners cues to the complexity of upcoming phrases. 1.

Tone nuclei in continuous speech are regarded as efficient targets for either tone recognition or intonation function decomposition. The paper presents our statistically robust method to segment and locate tone nuclei in continuous speech. The method includes: an iterative segmental K-means segmentation of the tonal F0 contours, which is further aided with t-test based segment amalgamation; a linear discriminant function based tone nucleus discriminator, whose features are selected by the sequential feature selection method. The developed system achieved 97.5% correct tone nuclei on a speaker dependent task. The tone recognizer based on the detected tone nuclei improved tone recognition rate by over 6% more than the baseline ones using the full tonal syllable features.

A new scheme of N-gram language modeling was pro-posed for Japanese, where word N-grams were calculated separately for the two cases: crossing and not crossing bunsetsu boundaries. Here, bunsetsu is a basic gram-matical (and pronunciation) unit of Japanese. A similar scheme using accent phrase boundaries instead of bun-setsu boundaries has already been proposed by the au-thors with a certain success, but it suffered from the train-ing data shortage, because assignment of accent phrase boundaries requires a speech corpus. In contrast, bun-setsu boundaries can be detected automatically from a written text with a rather high accuracy using a parser. It was shown from the experiment that a perplexity re-duction was possible by estimating bunsetsu boundaries from the history longer than N-1 words in the case of N-gram modeling and by selecting one from two types of models (crossing and not crossing bunsetsu bound-aries) according to the estimation. When 1 or 3 years of Mainichi Newspaper corpus was used for the training of tri-grams, the proposed scheme could reduce the perplex-ity by around 8 % from the baseline modeling (without separation). The proposed language modeling was ap-plied to a continuous speech recognition, and it showed that an improvement in word recognition rate was possi-ble especially when the training corpus was small (1 year of newspaper). 1.

As a major Chinese dialect, Cantonese is well known for its complex tone system. This paper applies the command-response model to represent the F0 contours of Cantonese speech. Analysis-by-Synthesis is conducted on both utterances of carrier sentences and utterances with less constrained structures, from which a set of appropriate tone command patterns is derived. By intrinsically incorporating the effects of tone coarticulation, word accentuation and phrase intonation, the model provides a high accuracy of approximation to the F0 contours of Cantonese, and hence serves as a much better method to quantitatively describe the continuous F0 contours than the traditional tone letter scale notation system. The constraints in timing and amplitude of tone commands are also investigated, which can be used for synthesis of F0 contours. 1.

In our fully automatic corpus-based method of generating fundamental frequency (F0) contours for emotional speech synthesis, an improvement was realized related to the process of corpus preparation. The method assumes the generation process model and predicts its command parameters using binary regression trees with inputs of linguistic information of the sentence to be synthesized. Because of the model constraint, a certain quality is still kept in synthesized speech even if the prediction is done incorrectly. The speech corpus includes three types of emotional speech (anger, joy, sadness) and calm speech uttered by a female narrator. The command parameters necessary for the training (and testing) of the method were automatically extracted from speech using a program developed by the authors. Since the accuracy of the extraction largely affects the prediction performance, a constraint is newly applied on the position of phrase commands during the extraction. Also, since performance of phrase command prediction dominates the overall accuracy of generated F0 contours, the method was modified to predict phrase commands first. The mismatches between the predicted and target contours for angry speech were similar to those for calm speech. Synthesis of emotional speech was conducted with text inputs. The segmental features were handled by the HMM synthesis method and the phoneme durations are predicted in a similar corpus-based method. Perceptual experiment was conducted using the synthesized speech, and the result indicated that the anger could be well conveyed by the developed method. The result came worse for joy and sadness. 1.

Prosody is known to play an important role in human speech perception process. Therefore, there is an increasing need to use prosodic features for the advancement of speech recognition technology. However, prosody is related to various levels of information, from linguistic, para-linguistic, to non-linguistic, and, therefore, its acoustic manifestation is rather complicated with large variations. This fact prevents prosody to be incorporated in speech recognition process. In the current paper, discussions are given on how we can utilize prosodic features, showing our research works as examples. First, an idea of including word likelihood viewed from the accent type into the recognition process is shown. Second, a scheme of using prosody to control the pruning size in the decoding process is given. Prosodic features should be modeled rather differently form segmental features. Lastly, a new language model constructed by including prosodic events is explained. 1.

In this paper we decompose the Hilbert Spectrum of an audio mixture into a number of subspaces to segregate the sources. Empirical mode decomposition (EMD) together with Hilbert transform produces Hilbert spectrum (HS), which is a fine-resolution time-frequency representation of a non-stationary signal. EMD decomposes the mixture signal into some intrinsic oscillatory modes called intrinsic mode function (IMF). HS is constructed from the instantaneous frequency responses of IMFs. Some frequency independent basis vectors are derived using independent component analysis (ICA). Kulback-Laibler divergence based k-means clustering algorithm is proposed to group the basis vectors into number of desired sources. Then projecting HS on to the grouped basis vectors derives the independent source subspaces. The time domain source signals are assembled by applying some post processing on the subspaces. We have also produced some experimental results using our proposed separation algorithm. 1.

Anchoring based discriminating features were proposed efficient for tone discrimination of Chinese continuous speech, and have been successfully applied before to tone classification of speaker dependent experiment. This paper presents its application to speaker independent tone classification experiments. Furthermore, we made detailed comparison experiments on the efficiencies of three groups of features: the left context dependent, the right context dependent anchoring F0 features, and the conventional F0 features. Experimental results showed that a combination of all three groups achieved a significant improvement of absolute 6.4% from 82.6% by the baseline system to 89.0%. When the three groups of features are used individually, both groups of the anchoring features led to better results than the conventional features, and the left context dependent anchoring features led to the highest performance.

This paper presents a multilevel framework to cope with the complex variations in Chinese sentential F0 contours in order to recognize lexical tones. Tone nucleus model is to get rid of the influence of intrinsic F0 transition loci at sub-syllable level. The pitch anchoring concept is used to normalize tonal F0 contours at syllable level. The hypo- and hyper-intonation model is used to account for the interplay of tone coarticulation and higher level prosodic effects. The whole approach achieved significant higher performance than the conventional method.

A method was developed to utilize linguistic information (lexical accent types and syntactic boundaries) to improve the performance of the automatic extraction of the F0 contour generation process model commands. The extraction scheme is first to smooth the observed F0 contour by a piecewise 3rd order polynomial function and to locate accent command positions by taking the derivative of the function. If the results of automatic extraction differ from those estimated from the linguistic information, they are modified according to the several rules. The results showed that some errors could be corrected by the use of linguistic information, especially when the initial word of an accent phrase is type 0 (flat) accent. As a whole, the correct extraction rate (recall rate) was increased from 79.8 % to 82.3 % for phrase commands and from 81.6 % to 85.9 % for accent commands.

A system was developed for teaching non-Japanese learners pronunciation of Japanese lexical accents. The system first identifies word accent types in a learner's utterance using F0 change between two adjacent morae as the feature parameter. As for the representative F0 value for a mora, we defined one with a good match to the perceived pitch. The system notices the user if his/her pronunciation is good or not, and, then, generates audio and visual corrective feedbacks. Using TDPSOLA technique, the learner's utterance is modified in its prosodic features by referring to teacher's features, and offered to the learner as the audio corrective feedback. The visual feedback is also offered to enhance the modifications that occurred. Accent type pronunciation training experiments were conducted for 8 non-Japanese speakers, and the results showed that the training process could be facilitated by the feedbacks especially when they were asked to pronounce sentences.

A corpus-based method of generating fundamental frequency (F0) contours of various speaking styles from text was developed. Instead of directly predicting F0 values, the method predicts command values of the F0 contour generation process model. Because of the model constraint, the resulting F0 contour keeps certain naturalness even when the prediction is done incorrectly. The method includes a scheme of automatic extraction of the model commands, which is necessary to prepare the training corpuses for various speaking styles. By introducing constraints on phrase command locations, a better extraction was realized, led to a better performance of the method. Speech synthesis was conducted using HMM speech synthesizer for calm speech and three types of emotional speech. The perceptual experiment showed the designated emotions could be well conveyed with the F0 contours generated by the developed method.

A concept to speech generation was realized in an agent dialogue system, where an agent (a stuffed animal) walked around in a small room constructed on a computer display to complete some jobs with instructions from a user. The communication between the user and the agent was done through speech. If the agent could not complete the job because of some difficulties, it tried to solve the problems through conversations with the user. Different from other spoken dialogue systems, the speech output from the agent was generated directly from the concept, and was synthesized using higher linguistic information. This scheme could largely improve the prosodic quality of speech output. In order to realize the concept to speech conversion, the linguistic information was handled as a tree structure in the whole dialogue process.

NAGAUTA (長唄) is a classical style of the Japanese singing. It has very original and unique prosodic patterns in its singing, where an abrupt and sharp change of F0 is always observed at a transition from a note to another. This F0 change is often found even where the transition is not accompanied by a change of tone. In this paper, we propose a model to synthesize this unique F0 pattern from the standard notation. Further, this paper shows an interesting phenomenon about power movements at the F0 changes. Acoustic analysis of NAGAUTA singing samples reveals that sharp increases of F0 and sharp decreases of power are observed synchronously. Although no discussion on physical mechanisms of this phenomenon is done in this paper, another model to generate this unique power pattern is also proposed. Evaluation experiments are done through listening and their results indicate high validity of the two proposed models.

In this paper, two techniques are proposed to enhance the nonnative (Japanese English) speech recognition performance. The first technique effectively integrates orthographic representation of a phoneme as an additional context in state clustering in training tied-state triphones. Non-native speakers often learned the target language not through their ears but through their eyes and it is easily assumed that their pronunciation of a phoneme may depend upon its grapheme. Here, correspondence between a vowel and its grapheme is automatically extracted and used as an additional context in the state clustering. The second technique elaborately couples a Japanese English acoustic model using triphones, mapping between the two models should be carefully trained because phoneme sets of both the models are different. Here, several phoneme recognition experiments are done to induce the mapping, and based upon the mapping, a tentative method of the coupling is examined. Results of LVCSR experiments show high validity of both the proposed methods.

This paper proposes a technique to estimate speakers’ perceptual age automatically only with acoustic information of their utterances. Firstly, we experimentally collected data of how old individual speakers in databasessound to listeners. Speech samples of approximately 500 male speakers with a very wide range of the real age were presented to listeners, who were asked to estimate the age only by hearing. Using the results, the perceptual age of the individual speakers was defined in two ways as label (averaged age over the listeners) and distribution. Then, each of the speakers was acoustically modeled by GMMs. Finally, the perceptual age of an input speaker was estimated as weighted sum of the perceptual age of all the other speakers in the databases, where the weight for speaker i was calculated as a function of likelihood score of the input speaker as speaker i. Experiments showed that correlation was about 0.9 between the perceptual age estimated by the listening test and that estimated by the proposed method. This paper also introduces some techniques to realize robust estimation of the perceptual age.

Due to various difficulties in pronunciation, utterances by nonnative speakers may be lacking in fluency. The Japanese pronunciation is said to have mora-synchronism, and, therefore, we assume that the disfluency may cause larger variations in vowel durations. Analyses of vowel (and CV) durations were conducted for Japanese sentence utterances by 2 non-Japanese speakers and one Japanese speaker (all female speakers). Larger variations were clearly observed in non-Japanese utterances. Then, 10 Japanese speakers were asked to rate the non-Japanese utterances. Strong negative correlations were observed between durational variations and pronunciation ratings. Based on the result, a method was developed for automatic evaluation of nonJapanese utterances. The ratings by the method were shown to be close to those by native speakers. Also, in order to offer a corrective feedback in learner’s voice, non-Japanese utterances were modified in their vowel durations by referring to native Japanese utterances. The modification was done using TD–PSOLA scheme. The result of listening test indicated some improvements in nativeness.

A new method was developed for accurately separating source and articulation filter characteristics of speech. This method is based on the AR-HMM modeling, where the residual waveform is expressed as the output sequence from an HMM. To realize an accurate analysis, a scheme of dividing HMM state was newly introduced. Using the AR-filter parameter values obtained through the analysis, we can construct a vocoder-type formant synthesizer, where the residual waveform is used as the excitation source. Through the listening test on the vowel sounds synthesized using AR-filter from a vowel and excitation waveform from another vowel, it was shown that a “flexible” synthesis with a high controllability on the acoustic parameters were possible by our formant synthesis configuration.

This paper proposes a technique which automatically estimates speakers' age only with acoustic, not linguistic, information of their utterances. This method is based upon speaker recognition techniques. In the current work, we firstly divided speakers of two databases, JNAS and S(senior)-JNAS, into two groups by listening tests. One group has only the speakers whose speech sounds so aged that one should take special care when he/she talks to them. The other group has the remaining speakers of the two databases. After that, each speaker group was modeled with GMM. Experiments of automatic identification of elderly speakers showed the correct identification rate of 91 %. To improve the performance, two prosodic features were considered, i.e, speech rate and local perturbation of power. Using these features, the identification rate was improved to 95%. Finally, using scores calculated by integrating GMMs with prosodic features, experiments were carried out to automatically estimate speakers' age. The results showed high correlation between speakers' age estimated subjectively by humans and automatically calculated score of ‘agedness’.

The process of generating the F0contour of speech has been modeled quite accurately in mathematical tenns by Fujisaki and his coworkers, but the extraction of parameters of the underlying commands from an observed F0contour is an inverse problem that can be solved only by successive approximation. In order to guarantee an efficient and accurate search for the solution, one needs to start with a set of initial values that are close enough to the optimum. This paper presents a method for pre-processing a measured F0contour to obtain its approximation consisting of third-order polynomial segments that are continuous and differentiable everywhere. It is shown that the proposed method allows one to obtain first-order approximations to the parameters of accent commands for about 90% of all the accent commands, and of phrase commands for about 84% of all the phrase commands.

A corpus-based method of generating fundamental frequency (F 0) contours of various speaking styles from text was developed. Instead of directly predicting F 0 values, the method predicts command values of the F 0 contour generation process model. Because of the model constraint, the resulting F 0 contour keeps certain naturalness even when the prediction is done incorrectly. The method includes a scheme of automatic extraction of the model commands, which is necessary to prepare the training corpuses for various speaking styles. By introducing constraints on phrase command locations, a better extraction was realized, led to a better performance of the method. Speech synthesis was conducted using HMM speech synthesizer for calm speech and three types of emotional speech. The perceptual experiment showed the designated emotions could be well conveyed with the F 0 contours generated by the developed method. 1.

To recognize non-native speech, larger acoustic/linguistic distortions must be handled adequately in acoustic modeling, language modeling, lexical modeling, and/or decoding strategy. In this paper, a novel method to enhance MLLR adaptation of acoustic models for non-native speech recognition is proposed. In the case of native speech recognition, MLLR speaker adaptation was successfully introduced because it enables efficient adaptation with a small number of adaptation data by using a regression tree of Gaussian mixtures of HMMs. However, as for non-native speech, most of the cases, the regression tree built from the baseline HMMs does not match with pronunciation proficiency of a speaker. This paper provides a solution for this problem, where the speaker’s proficiency is automatically estimated and the tree suited for the proficiency is built, which can be viewed as proficiency adaptation. Recognition experiments show that MLLR with the new tree raises the averaged error reduction rate up to about 30 % from the baseline MLLR performance of approximately 20 %.

The generation process model of the fundamental frequency contours (F0 contours) of speech is known to be capable of generating F0 contours quite close to observed ones. The extraction of model parameters from an observed contour, however, requires an iterative process starting from a set of initial parameter values. In order to guarantee a rapid convergence to an optimum solution, the values should be appropriate ones. We already have developed a method of automatically extracting these from given F0 contours, and applied it to Japanese sentences with good results. The method is based on approximating an observed contour by a continuous curve differentiable everywhere. In the present paper, it was applied to English utterances. Experiments were conducted for 4 native speakers’ utterances with 14.5% and 17.5% of average miss and false alarm rates for the accent commands, and 35.7% and 15.5% for the phrase commands.

A new method was developed for the separation of source and transfer function characteristics of speech sounds, with an aim of utilizing it to “flexible ” speech synthesis. The method is based on representing source waveform by an HMM, and transfer function by the AR process (AR-HMM model). As compared to methods based on ARX model, where a parametric representation is assumed for source waveform, a better separation is possible. By introducing a process of recursively deleting real poles of AR filters, which represent source waveform features, and including them into HMM source waveform, the resulting AR filters may correctly represent transfer function features. Experiments were conducted for Japanese vowel sounds in continuous speech, and the results were compared with those by conventional LP analysis and AR-HMM model analysis without recursive process. After representing obtained source and transfer function features respectively as DFT cepstrum and LPC cepstrum, variations of cepstrum parameters for each vowel sound were compared for the three analysis methods. The smallest variations were obtained by the proposed method, indicating that the proposed method can separate source and transfer function features well, and, thus, has potential ability of generating good quality of speech when applied to “flexible” speech synthesis. 1.

Introduces a model-constrained, data-driven method for generating fundamental frequency contours in Japanese text-to-speech synthesis. In the training phase, the parameters of a command-response F/sub 0/ contour generation model are learned by a prediction module, which can be a neural network or a set of binary regression trees. The input features consist of linguistic information related to accentual phrases that can be automatically derived from text, such as the position of the accentual phrase in the utterance, number of morae, accent type, and parts-of-speech. In the synthesis phase, the prediction module is used to generate appropriate values of model parameters. The use of the parametric model restricts the degrees of freedom of the problem, facilitating data-driven learning. Experimental results show that the method makes it possible to generate quite natural F/sub 0/ contours with a relatively small training database.

A mode-constrained corpus-based synthesis strategy was developed for fundamental frequency (F 0 ) contours of Japanese sentences.In the training phase, the relationship between linguistic factors and the command values (amplitudes and locations) of F 0 contour generation process model was learned for a prediction module; a neural network in the current paper.Input parameters consist of linguistic information related to accentual phrases that can be automatically driven from text, such as the position of the accentual phrase in the utterance, number of morae, accent type, and morphological information.In the synthesis phase, the prediction module is used to generate command values of the model.The synthesis method was also realized based on multiple linear regression analysis to examine how each input parameter contributes to the F 0 contour generation.The use of the parametric model restricts the degrees of freedom of the mapping between linguistic and prosodic features, and thus enables to generate appropriate values even with limited training data.Experimental results showed that the method could generate F 0 contours quite close to those by the rulebased method.

In order to construct a CALL (Computer Aided Language Learning) system that can teach learners accent and intonation of Japanese, it's necessary to automatically identify accent types and intonation types in sentence utterances.For this purpose, several acoustic (prosodic) features of speech were investigated taking their effects on human perception into account.For the accent type identification method, the use of average values of F0 in mora and target values of F0 in mora final was evaluated in CV and VC units.Average values of VC units and target values of CV units showed better performance in the identification task.As for the intonation identification, several acoustic features were investigated to represent 6 types of sentence final tones, each conveying different information of intention and perceptual impression.The proposed acoustic features for relative duration and sentence final pitch change showed good correspondence to perceptual features.

A new method of pruning redundant synthesis unit instances in a large-scale synthesis database was proposed based on weighted vector quantization (WVQ). WVQ takes relative importance of each instance into account when clustering the similar instances using vector quantization (VQ) technique. The proposed method was compared with two conventional pruning methods through objective and subjective evaluations of synthetic speech quality: one to simply limit maximum number of instance, and the other based on normal VQ-based clustering. For the same reduction rate of instance number, the proposed method showed the best performance. The synthetic speech with reduction rate 50% sounded with no perceptible degradation as compared to that without instance reduction.

An efficient search function based on topic estimation was integrated to our spoken dialogue system for academic document information retrieval. The following two points were mainly studied: 1) to properly categorize documents (to be retrieved) into related topics, and 2) to facilitate retrieval process using topic knowledge. For the first point, a method was developed to calculate recursively the relevance scores of retrieval words and documents for topics. Effects of the recursive process were proved through experimental results; better classification of retrieval words and documents into topics was realized. As for the second point, retrieval range was limited into topics estimated from retrieval words. It was shown through experiments of retrieval task solving that necessary number of dialogue turns (therefore, period of dialogue) could be largely reduced by the range limitation; a smooth retrieval process was proved to be realized using topic knowledge.

We have been developing a reliable method of prosodic word boundary detection for Japanese continuous speech based on the statistical modeling of mora transitions of fundamental frequency contours of prosodic words. Modifications in the codebook sizes and in the HMM topologies improved the boundary detection performance. When using mora boundary information obtainable from the phoneme recognition process, the detection rates were reached around 73% with 12.5% insertion errors for speaker-open experiments. This method was then integrated to a continuous speech recognition system with unlimited vocabulary. The integrated system conducts the recognition process in two stages: the first stage is to detect mora boundaries without prosodic information and the second stage is to increase the mora recognition rate using prosodic word boundary information. Slight improvements in mora recognition rates were observed both in speaker-closed and -open experiments.

Vibrato is an important factor in the production of good quality singing of western classical songs. Vibrato is characterized by a temporal modulation of the singing pitch, as well as of the spectral structure. We are trying to synthesize singing using the large database, selection synthesis scheme. Simple waveform concatenation-modification synthesis can produce unnatural sounding vibrato sounds. A method to synthesize more natural vibrato sounds is described. This method works by ensuring that the phase of the vibrato in the original (source database) unit, used for synthesis, is consistent with the vibrato phase of the synthesized (target) vibrato sound. By doing so, the required modification of the original unit's prosody can be minimized, and the production of the characteristic vibrato spectral modulations can be improved.

We present in this paper a new Chinese lexical tone recognition approach based on our pitch anchoring hypothesis, which suggests that the tone offset of the preceding lexical tone and the tone onset of the succeeding lexical tone serve as anchor points for the pitch heights of the onset and offset of the sandwiched lexical tone. The new approach exploits relative F0 heights between neighboring tones as important discriminating features for the lexical tones. Experimental results revealed that the new approach could increase the tone recognition accuracy greatly: above 10% (from 75.3% to 85.5%) compared with the conventional one.

Investigation was conducted on how prosodic features of emotional speech changed depending on emotion levels. The analysis results on fundamental frequency (F0) contours and speech rates implied that humans have several ways to express emotions and use them rather randomly. Investigation was also conducted on what acoustic features were important to express emotions. Perceptual experiments using synthetic speech with copied acoustic features of target speech indicated importance of the segmental features other than the prosodic features. Especially, a high importance was observed in the case of happiness. 1.

Fig.1. Two utterances “Sorewa oQtodesu ” (fast speech) and “Sorewa otodesu ” (slow speech) of a same speaker. The duration of the long phone /Qt / in the upper utterance is shorter than the short phone /t / in the lower utterance, showing the danger of using absolute values.

Information of phone relationships is regarded as acting an important role in speech recognition. It has been successfully exploited in many speaker adaptation approaches. In this paper, we propose a new approach, named Phone Pair Model (PPM) re-scoring, to utilize phone relationships for speaker-adaptive speech recognition. PPM re-scoring approach does not really adapt model parameters to a new speaker. It just uses some pre-registered phones ' samples from the speaker being recognized, to re-calculate the likelihood of phones that has been calculated on conventional phone HMMs, resulting in a more correct recognition result. Additionally, it can deal with not only inter-speaker acoustic variations but also intra-speaker acoustic variations adequately. Results of two recognition experiments, one using phone HMMs only and the other incorporating phone HMMs with the PPMs, showed that even by using only a few vowel samples as the pre-registered phones, PP-M re-scoring approach brought an increase in recognition rate. 1.

Although flexible control of acoustic features is possible in formant-based speech synthesizers, their development requires precise estimation of parameters related to vocal tract and source. This requirement is difficult to satisfy and often results in limiting quality of the synthesized speech. The difficulty is derived from the fact that estimation of the parameters is a nonlinear problem. Therefore, the completely automatic estimation of the parameters is quite difficult and some approximations or manual modifications of parameters with aprioriknowledge are required in the development. In this study, mainly to make the estimation more efficient and/or to assist developers doing the manual modifications of parameters, a formant-based analysissynthesis system is build. The system introduces pitchsynchronous acoustic analysis to reduce fluctuation of the estimated parameters. Experiments show that quality of synthetic speech of Japanese /r / sounds is significantly improved by using the proposed system. 1.

In this paper, we propose a novel implementation of a minimax decision rule for continuous density hidden Markov-model-based robust speech recognition. By combining the idea of the minimax decision rule with a normal Viterbi search, we derive a recursive minimax search algorithm, where the minimax decision rule is repetitively applied to determine the partial paths during the search procedure. Because of the intrinsic nature of a recursive search, the proposed method can be easily extended to perform continuous speech recognition. Experimental results on Japanese isolated digits and TIDIGITS, where the mismatch between training and testing conditions is caused by additive white Gaussian noise, show the viability and efficiency of the proposed minimax search algorithm.

A new method for prosodic word boundary detection in continuous speech was developed based on the statistical modeling of moraic transitions of fundamental frequency (F/sub 0/) contours, formerly proposed by the authors. In the developed method, F/sub 0/ contours of prosodic words were modeled separately according to the accent types. An input utterance was matched against the models and was divided into constituent prosodic words. By doing so, prosodic word boundaries can be obtained. The method was first applied to the boundary detection experiments of the ATR continuous speech corpus. With mora boundary locations given in the corpus, total detection rate reached 91.5%. Then the method was integrated into a continuous speech recognition scheme with unlimited vocabulary. A few percentage improvement was observed in mora recognition for the above corpus. Although all the experiments were done in closed conditions due to the corpus availability, the results indicated the usefulness of the proposed method.

The aim of this text-to-speech synthesis system is to convert the Estonian orthographic text to an orthoepically correct and natural-sounding spoken text for a wide range of practical application (especially for visual and speech disabled persons).

We study a category of robust speech recognition problem in which mismatches exist between training and testing conditions, and no accurate knowledge of the mismatch mechanism is available. The only available information is the test data along with a set of pretrained Gaussian mixture continuous density hidden Markov models (CDHMMs). We investigate the problem from the viewpoint of Bayesian prediction. A simple prior distribution, namely constrained uniform distribution, is adopted to characterize the uncertainty of the mean vectors of the CDHMMs. Two methods, namely a model compensation technique based on Bayesian predictive density and a robust decision strategy called Viterbi Bayesian predictive classification are studied. The proposed methods are compared with the conventional Viterbi decoding algorithm in speaker-independent recognition experiments on isolated digits and TI connected digit strings (TIDTGITS), where the mismatches between training and testing conditions are caused by: (1) additive Gaussian white noise, (2) each of 25 types of actual additive ambient noises, and (3) gender difference. The experimental results show that the adopted prior distribution and the proposed techniques help to improve the performance robustness under the examined mismatch conditions.

We extend our previously proposed Viterbi Bayesian predictive classification (VBPC) algorithm to accommodate a new class of prior probability density function (PDF) for continuous density hidden Markov model (CDHMM) based robust speech recognition. The initial prior PDF of CDHMM is assumed to be a finite mixture of natural conjugate prior PDF's of its complete-data density. With the new observation data, the true posterior PDF is approximated by the same type of finite mixture PDF's which retain the required most significant terms in the true posterior density according to their contribution to the corresponding predictive density. Then the updated mixture PDF is used to improve the VBPC performance. The experimental results on a speaker-independent recognition task of isolated Japanese digits confirm the viability and the usefulness of the proposed technique.

Experiments on accent type recognition and syntactic boundary detection of Japanese speech were conducted based on the statistical modeling of voice fundamental frequency contours formerly proposed by the authors. In the proposed modeling, fundamental frequency contours are segmented into moraic units to generate moraic contours, which are further represented by discrete codes. After modeling the accent types and syntactic boundaries, their recognition/detection was done for ATR speech corpus. For the accent type recognition, 4-mora words were used for the training and testing, and recognition rates of around 74% were obtained for speaker open experiments. For syntactic boundary detection, the detectability of accent phrase boundaries was tested for sentence speech. Although the experiments were conducted only for the closed condition due to the availability of speech corpus, the result indicated the usefulness of separating the boundary model into two depending on whether the boundary is accompanied by a pause or not.

We have formerly proposed a statistical model of moraic transitions of fundamental frequency (F0 ) contours and showed its effectiveness for prosodic boundary detection and accent type recognition. This model represented F0 contours of prosodic words to simultaneously detect and recognize prosodic word boundaries and accent types. This paper proposes a method where prosodic word F0 contours are modeled separately according to their accent types and presence/absence of succeeding pauses. An utterance is regarded as a sequence of prosodic words under a simple grammar. Each moraic F0 contour is represented by a pair of codes; the original shape code and the newly introduced delta code representing the degree of F0 change between the mora in question and its preceding mora. Compared with earlier results, the boundary detection rate inproves from 87.7 % to 91.5 %. Accent type recognition rate reached 76.0 % (type 1 accent discrimination) . 1. INTRODUCTION Prosodic features of speech are u...

We propose a method to generate a database that contains a parametric representation of F0 contours associated with linguistic and acoustic information, to be used by data-driven Japanese text-to-speech (TTS) systems. The configuration of the database includes recorded speech, F0 contours and their parametric labels, phonetic transcription with durations, and other linguistic information such as orthographic transcription, part-of-speech (POS) tags, and accent types. All information that is not available by dictionary lookup is obtained automatically. In this paper, we propose a method to automatically obtain parametric labels that describe F0 contours based on a superpositional model. Preliminary tests on a small data set show that the method can find the parametric representation of F0 contours with acceptable accuracy, and that accuracy can be improved by introducing additional linguistic information. 1. INTRODUCTION Generating prosodic features from text is one of the most diffic...

In this paper, we investigate a new Bayesian predictive classification (BPC) approach to realize robust speech recognition when there exist mismatches between training and test conditions but no accurate knowledge of the mismatch mechanism is available. A specific approximate BPC algorithm called Viterbi BPC (VBPC) is proposed for both isolated word and continuous speech recognition. The proposed VBPC algorithm is compared with conventional Viterbi decoding algorithm on speaker-independent isolated digit and connected digit string (TIDIGITS) recognition tasks. The experimental results show that VBPC can considerably improve robustness when mismatches exist between training and testing conditions.

We developed a CALL (computer-aided language learning) system for teaching the pronunciation of Japanese long vowels, the mora nasal and mora obstruents to nonnative speakers of Japanese. Long vowels and short vowels are spectrally almost identical but their phone durations differ significantly. Similar conditions exist between mora nasals and non-mora nasals, and between mora and non-mora obstruents. Our system uses speech recognition to measure the durations of each phone and compares them with distributions of native speakers while correcting for different speech rates. Results show that learners quickly capture the relevant duration cues. The amount of learning time spent on acquiring these durational skills is well within the time constraints of TJSL (teaching Japanese as a second language) curricula.

In the past, we attempted to use a multilayer perceptron neural network as a means to prevent those unknown language inputs from being misidentified as one of the target languages in language identification system. However, the use of multilayer perceptron neural network could not utilize the temporal information from the utterances. Results show that with the use of phonemic unigram as input features to a recurrent neural network of Jordan architecture, a 3 target language identification rate of 98.1% can be achieved. By setting the output thresholds to 0.6 to reject 2 more unknown languages, a lower overall rate of 85.9% is obtained.

A phrase boundary detection experiment was realised using a previously proposed method for the use of prosodic features in automatic speech recognition. The method consists of comparing the observed fundamental frequency contour of an utterance with the contour generated by a model for each recognition hypothesis, using a scheme of partial analysis-by-synthesis. Allowing l-mora discrepancies, the detection rate reached 95%. To cope with the high variability in the errors obtained by the partial analysis-by-synthesis even for correct hypotheses, further experiments were carried out in two ways: smoothing the observed contour with a low-pass filter to reduce the effects of microprosody, and adapting the initial conditions of the model-generated contours to the speaker. Experiments were also conducted applying the method at non-phrasal boundaries with the objective of obtaining clues that permit the differentiation between phrase boundaries and non-phrasal boundaries.

The number of languages in the world is much larger than the number of target languages that current language identication systems can handle.Therefore, we propose here the use of a multilayer perceptron neural network as a means to prevent those unknown language inputs from being misidentied as one of the target languages.We consider not only the target language identication rate but also the unknown language rejection rate.Results reveal that with the use of phonemic unigram as the input features to the neural network, a target language identication rate of 93.5% can be achieved for 3 languages.By varying the thresholds of the outputs, good unknown language rejection rate can also be obtained at the expense of lower identication rate.

In this paper, a computer assisted language training system, focusing on speech input and output, is described.The system is intended to help students of foreign language (typically Japanese or English) to improve their pronunciation, with an emphasis on prosodic features of speech.The system incorporates a combination of speech processing techniques, in order to analyze the input speech, and to produce eective speech feedback.The system is implemented on a Unix PC, with audio I/O capability, in a window e n vironment.

Major syntactic boundaries are often accompanied by a rise in the phrase component of the fundamental frequency (F0) contour.Detecting such rises, therefore, can be signicantly helpful to the speech recognition process.We developed a method to detect syntactic boundaries with phrasecomponent rise (henceforth, phrase boundaries), based on the compression of the accent component of the F0 contour (in logarithmic scale), using a low-pass lter.In this method, F0 contours are viewed as signals in the time domain, which can be roughly separated into phrase and accent components due to their dierent frequency contents.Phrase boundaries are detected whenever a signicant rise occurs in the derivative of the ltered F0 contour.(The concepts of phrase and accent components can be found in [1]).The method managed to detect about 77% of manually detectable phrase boundaries, though with a relatively high insertion rate.The insertion rate can be reduced by using the partial AbS method, proposed by the authors [7].

Although pitch extraction schemes based on the short-time autocorrelation function offer reliable results for most of the speech signal, the autocorrelation peaks indicating fundamental period fluctuate with frame position, causing occasional pitch extraction errors. In order to reduce these errors, a scheme using a new definition of normalized short-time autocorrelation function is proposed. One of the major advantages of the definition over conventional ones is that the frame length changes in proportion to the time lag, and, therefore, the input speech can be analyzed without any knowledge of the fundamental frequency range of the speaker. Two methods are proposed for the normalization, and the one compensating for variations of the short-time power of the waveform is shown to offer better results. A system for pitch extraction is constructed on a workstation, and the validity of the proposed scheme is demonstrated by experiments using the connected speech of male and female announcers.>

A text-to-speech conversion system for Japanese, developed for the purpose of producing high-quality speech output, is presented. It consists of four processing stages: (1) linguistic processing, (2) phonological processing, (3) control parameter generation, and (4) speech waveform generation. An overview of the whole system is presented. The innovations introduced into the second and the fourth stages, i.e. rules for generating prosodic symbols from the linguistic information in the second stage and the configuration of a new type of terminal analog speech synthesizer designed as the fourth stage, are described. The validity of the approach is confirmed by the improvements both in the prosodic and in the segmental qualities of synthesized speech.>

A method of path-limitation is proposed for the efficient DP matching of spoken words. Results of a preliminary recognition experiment show that this method can reduce the recognition errors as well. A method for effective speaker normalization is proposed. This method is based on the transformation of the relative positions of the five vowels of a given speaker in the parameter space. As a result of transformation, the dimension of the parameter space is reduced by a factor of four, yielding a great advantage in the computation time for matching. A study on the realization of a spoken-word recognition device using systolic array processors is described. A cyclic configuration that permits real-time matching of 600 templates with a single LSI chip is proposed.>

The causes of errors in conventional pitch extraction methods can be classified into two categories: 1) extrinsic factors that come from the analysis methods, such as imperfect signal representation due to inappropriate shape, width, and placement of the analysis window, etc., and 2) intrinsic factors that reside in the speech signal itself, such as the occurrence of a strong harmonic component or a sub-harmonic component, etc. In fact, fixed frame size and frame shift, adopted in most of the conventional systems, are responsible for a considerable part of their gross pitch errors. In this paper we combine the use of running waveform analysis, an exponential analysis window, and a variable frame shift to cope with errors of the first category. Various new methods are also introduced to cope with causes of errors of the second category. The validity of the proposed methods is confirmed by experiments using speech materials from both male and female speakers.

Announcements and narrations broadcast over radio and television have been serving in recent years as the most effective means for spreading the spoken version of the common Japanese, especially in regards to intonation. No formal studies, however, seem to have been conducted on the objective analysis and formulation of the rules of intonation of the common Japanese. The present work investigates the acoustical and perceptual characteristics of the intonation of the common Japanese used by radio and television announcers, and attempts to derive rules of intonation for efficient transmission of spoken messages. The results are expected to be of value both for high-quality speech synthesis and for scientific and objective teaching of the spoken version of the common Japanese.

Critical re-examination of the premises of conventional systems for continuous speech recognition has led to a study of human processes of speech perception. It was found that deletion of a syllable is often not noticed by a listenter, suggesting that the basic unit of continuous speech perception is larger than the syllable. Further experiments are described and discussed on the size of actual units of perception, effects of syntactic roles, unit, organization and access to the mental lexicon, effects of context, as well as effects of repeated listening. A system for automatic recognition of continuous speech is then proposed on the basis of these results and considerations, utilizing both words and syllables as units of recognition, and using both local and global coherence in reducing word candidates as well as modifying or renewing word and syllable templates.

A new optical processing system for spectrum analysis of speech was proposed. The main components of this system is an optical processor and a micro-computer. The processor consists of a He-Ne laser, optical lenses, and photographic film plates. Since the optical signal processing is inherently parallel for two-dimensional signals, time-varying spectra of one-dimensional signal of speech can be obtained without shifting a window along the time axis. Based on the results of a computer simulation for the spectrum analysis of speech, the optical processor was designed and constructed. Various /V/, /VV/ and /CV/ utterances were analyzed by using the optical processing system. The close agreement between the frequency spectra obtained by the system and those obtained by the simulation confirms that the system operates as expected. Non-linear time warping, indispensable for spoken word recognition, was shown to be accomplished by controlling the transmittance function of the windowing plate.

In a text-to-speech conversion system, the prosodic features of speech should be synthesized strictly by rules. In order to construct a set of rules to generate the symbols for the prosodic features, such as the pause duration, the phrase command magnitude, and the accent command amplitude, speech samples of Japanese texts were analyzed quantitatively using the model for fundamental frequency contour generation. The analysis made clear the relationships between the prosodic features and the linguistic information, such as the accent type of a word, the syntactic structure of a sentence, and the discourse structure of a text. Based on these relationships, rules were constructed which generate the prosodic symbols from the linguistic information. Text-to-speech conversion was conducted using these rules. The naturalness of the intonation of the synthesized speech indicated the validity of the rules.

A system is proposed for automatic speech recognition using syllable templates. In this system, input speech signal is analyzed and matched against syllable templates and converted into parameters characterizing each candidate syllable. Word recognition is based on an overall likelihood measure calculated for each item stored in the lexicon. A method is also developed for the optimization of syllable templates. The validity of the proposed method was tested in a preliminary recognition experiment in which a lexicon consisting of 1000 city names was used to recognize utterances of 100 city names by a female speaker. The rate of correct recognition was 96.5%.

A number of utterance samples of complex sentences of Japanese were analyzed using the model for the process of fundamental frequency contour generation, with special emphasis on the relationship between a fundamental frequency contour and its underlying lexical, syntax and semantic information. Phrase commands of the model were found to be roughly classified into three groups according to the level of corresponding node of the syntactic tree of a sentence, while accent commands were found to be roughly classified into two groups according to the type of accentuation. A set of rules was constructed for generating fundamental frequency contours of complex sentences of Japanese from linguistic information. Perceptual tests of naturalness of intonation using synthetic speech with rule-generated fundamental frequency contours indicated the validity of the rules.

A model is presented for the analysis and synthesis of F0-contours of spoken sentences. It is based on a quantitative formulation of the process whereby the logarithmic fundamental frequency is controlled by the phrase and accent commands, being respectively related to the syntactic and lexical information of a sentence. Analysis of natural utterances revealed that the model can generate close approximations to observed F0-contours. Requirements for the accuracy of model parameters were determined by listening tests of synthetic utterances. These results indicate the model's utility in speech synthesis.