In the present study, we quantify entrainment characteristics of conversation with the aim of automatic assessment of the severity of autism spectrum disorder (ASD). We focus on pairs of utterances immediately before and after turn-takings, which have prosodic/acoustic similarities.The clinical severity of ASD is estimated by the bottleneck features obtained by an hourglass-shaped deep neural network (DNN) in the neural entrainment distance (NED) method used to measure the degree of entrainment. The DNN is firstly pre-trained using a large conversation corpus in various daily situations and then fine-tuned with conversations during the Autism Diagnostic Observation Schedule (ADOS) assessment. Absolute difference vectors are calculated from the bottleneck feature vectors between a pair of utterances. Centroid and variance of the absolute difference vectors are combined with the speech features discovered in our previous study in order to estimate the scores of ASD severity.Consequently, the estimated scores significantly correlate with the actual observed ADOS ‘Reciprocity' scores with a coefficient of 0.70. This result shows the effective use of finetuning technique with data of typically developed individuals and, furthermore, reveals social communication deficits in ASD individuals represented by utterances adjacent to turn-takings.

In a recent conference paper, we have reported a rhythm transcription method based on a merged-output hidden Markov model (HMM) that explicitly describes the multiple-voice structure of polyphonic music. This model solves a major problem of conventional methods that could not properly describe the nature of multiple voices as in polyrhythmic scores or in the phenomenon of loose synchrony between voices. In this paper, we present a complete description of the proposed model and develop an inference technique, which is valid for any merged-output HMMs, for which output probabilities depend on past events. We also examine the influence of the architecture and parameters of the method in terms of accuracies of rhythm transcription and voice separation and perform comparative evaluations with six other algorithms. Using MIDI recordings of classical piano pieces, we found that the proposed model outperformed other methods by more than 12 points in the accuracy for polyrhythmic performances and performed almost as good as the best one for non-polyrhythmic performances. This reveals the state-of-the-art methods of rhythm transcription for the first time in the literature. Publicly available source codes are also provided for future comparisons.

This paper discusses real-time alignment of audio signals of music performance to the corresponding score (a.k.a. score following) which can handle tempo changes, errors and arbitrary repeats and/or skips (repeats/skips) in performances. This type of score following is particularly useful in automatic accompaniment for practices and rehearsals, where errors and repeats/skips are often made. Simple extensions of the algorithms previously proposed in the literature are not applicable in these situations for scores of practical length due to the problem of large computational complexity. To cope with this problem, we present two hidden Markov models of monophonic performance with errors and arbitrary repeats/skips, and derive efficient score-following algorithms with an assumption that the prior probability distributions of score positions before and after repeats/skips are independent from each other. We confirmed real-time operation of the algorithms with music scores of practical length (around 10000 notes) on a modern laptop and their tracking ability to the input performance within 0.7 s on average after repeats/skips in clarinet performance data. Further improvements and extension for polyphonic signals are also discussed.

This paper describes a method to separate a monaural music signal into harmonic components e.g., a guitar and percussive components, e.g., a snare drum. Separation of these two components is a useful preprocessing for many music information retrieval applications, and in addition, it can be used as a new kind of music equalizer in itself, which enables a music listener to adjust the ratio of the volume of the guitar and the drum freely by themselves. Because of these potential applications, there have been many attempts to develop such a technique, especially in the last decade. However, some of the state-of-the-art techniques have a drawback that they are based on costly operations, such as the multiplications of large-sized matrix, Monte Carlo method, etc., which may constitute barriers to the practical use on some small computers such as smart phones. In this paper, an efficient method that does not depend on these costly operations is described. In formulating the methods, the authors basically assumed only the “anisotropic smoothness” of music spectrogram, which can be one of the minimalistic model that reflects the natures of these instruments. To be specific, the authors just assumed that harmonic instruments are smooth in time, while the percussive instruments are smooth in frequency on a music spectrogram. In this paper, on the basis of the assumption, source separation methods are formulated as optimization problems that optimize the “anisotropic smoothness” under some conditions. Because of the simplicity of the model, the derived algorithms are quite simple. Experimental results show that the methods were effective compared to a state-of-the-art technique, and the computation time was much shorter than an existing method; specifically, it can process a three-minute song in around 4-20 seconds on a laptop PC.

We propose a novel singing voice enhancement technique for monaural music audio signals, which is a quite challenging problem. Many singing voice enhancement techniques have been proposed recently. However, our approach is based on a quite different idea from these existing methods. We focused on the fluctuation of a singing voice and considered to detect it by exploiting two differently resolved spectrograms, one has rich temporal resolution and poor frequency resolution, while the other has rich frequency resolution and poor temporal resolution. On such two spectrograms, the shapes of fluctuating components are quite different. Based on this idea, we propose a singing voice enhancement technique that we call two-stage harmonic/percussive sound separation (HPSS). In this paper, we describe the details of two-stage HPSS and evaluate the performance of the method. The experimental results show that SDR, a commonly-used criterion on the task, was improved by around 4 dB, which is a considerably higher level than existing methods. In addition, we also evaluated the performance of the method as a preprocessing for melody estimation in music. The experimental results show that our singing voice enhancement technique considerably improved the performance of a simple pitch estimation technique. These results prove the effectiveness of the proposed method.

We have previously proposed a generative model of speech F0 contours, based on the discrete-time version of the Fujisaki model (a model of the mechanisim for controlling F0s through laryngeal muscles). One advantage of this model is that it allows us to apply statistical methods to estimate the Fujisakimodel parameters from speech F0 contours. This paper proposes a new generative model of speech F0 contours incorporating a vocabulary model of intonation patterns. A parameter inference algorithm for the present model is derived. We quantitatively evaluated the performance of our parameter inference algorithm.

This paper introduces our ongoing work on generative modeling of speech fundamental frequency (F0) contours for estimating prosodic features from raw speech data. The present F0 contour model is formulated by translating the Fujisaki model, a well-founded mathematical model representing the control mechanism of vocal fold vibration, into a probabilistic model described as a discrete-time stochastic process. The motivation behind this formulation is two fold. One is to derive a general parameter estimation framework for the Fujisaki model, allowing for the introduction of powerful statistical methods. The other is to construct an automatically trainable version of the Fujisaki model so that in future it can be used to develop a statistical speaking style conversion system or incorporated into existing text-to-speech synthesis systems to improve the naturalness and intelligibility of computer-generated speech. We also briefly introduce a generative model of F0 contours of singing voice developed under the same spirit. Index Terms: speech F0 contour, Fujisaki model, generative model, hidden Markov model, EM algorithm

Symbolic pitch modeling is a way of incorporating knowledge about relations between pitches into the process of analyzing musical information or signals. In this paper, we propose a family of probabilistic symbolic polyphonic pitch models, which account for both the “horizontal” and the “vertical” pitch structure. These models are formulated as linear or log-linear interpolations of up to five sub-models, each of which is responsible for modeling a different type of relation. The ability of the models to predict symbolic pitch data is evaluated in terms of their cross-entropy, and of a newly proposed “contextual cross-entropy” measure. Their performance is then measured on synthesized polyphonic audio signals in terms of the accuracy of multiple pitch estimation in combination with a Nonnegative Matrix Factorization-based acoustic model. In both experiments, the log-linear combination of at least one “vertical” (e.g., harmony) and one “horizontal” (e.g., note duration) sub-model outperformed a pitch-dependent Bernoulli prior by more than 60% in relative cross-entropy and 3% in absolute multiple pitch estimation accuracy. This work provides a proof of concept of the usefulness of model interpolation, which may be used for improved symbolic modeling of other aspects of music in the future.

Acoustic event detection systems supporting heterogeneous sets of events face the problem of having to characterize them when they have different acoustic properties (transient, stationary, both, etc.), observing this fact even within the acoustic event itself. Moreover, managing large feature vectors with features characterizing different properties of the signal is always difficult. This paper introduces the usage of spectro-temporal fluctuation features in a tandem connectionist approach, modified to generate posterior features separately for each fluctuation scale and then combine the streams to be fed to a classic GMM-HMM model. The experiments explore scale and event wise performance, as well as different stream combination methods, and show that the proposed method outperforms the GMM-HMM baseline as well as recent proposals in the CHIL 2007 evaluation campaign's related acoustic event detection tasks.

Music spectrograms typically have many structural regularities that can be exploited to help solve the problem of decomposing a given spectrogram into distinct musically meaningful components. In this paper, we introduce new variants of the non-negative matrix factorization concept that incorporate music-specific constraints.

This paper proposes model-based non-negative matrix factorization (NMF) for estimating basis spectra and activations, detecting note onsets and offsets, and determining beat locations, simultaneously. Multipitch analysis is a process of detecting the pitch and onset of each note from a musical signal. Conventional NMF-based approaches often lead to unsatisfactory results very possibly due to the lack of musically meaningful constraints. As music is highly structured in terms of the temporal regularity underlying the onset occurrences of notes, we use this rhythmic structure to constrain NMF by parametrically modeling each note activation with a Gaussian mixture and derive an algorithm for iteratively updating model parameters. It is experimentally shown that the proposed model outperforms the standard NMF algorithms as regards onset detection rate.

In this paper, user-guided source separation based on independent vector analysis is presented. In this framework, temporal power variations of sources can be tuned by a user. The information is exploited as prior distributions of source activities in independent vector analysis with time-varying Gaussian model, and source signals are separated by maximum a posteriori (MAP) estimation. Experimental evaluations show the source activity tuning is much effective to improve the separation performance in hard mixing conditions such as long reverberation or level mismatch of sources.

In this paper, we explore several algorithms to find the best performing algorithm for harmonic and percussive sound separation (HPSS) based on anisotropic continuity of spectrogram through comparative evaluation of their experimental performance. Separating harmonic and percussive sounds is useful as a preprocessor for many music analysis purposes including chord estimation, rhythm analysis, and other music information retrieval tasks. We have introduced a method called “Harmonic/Percussive Sound Separation” (HPSS), that decomposes a music signal into two components by separating the spectrogram into horizontally-continuous and vertically-continuous components, which roughly correspond to harmonic and percussive sounds, respectively. Many possible ways exist to realize the HPSS algorithm based on this concept while it has been unknown which algorithm performs best. This paper describes the details of five different HPSS algorithms and compares their performances over real music signals.

This paper proposes a stochastic model of speech F0 contours, based on the stochastic formulation of the Fujisaki model. Our motivation for the stochastic formulation is twofold. Firstly, it allows us to derive a well-behaved algorithm for estimating the Fujisaki model parameters from a raw F0 contour. Secondly, it will open the door to incorporating the well-founded F0 contour model into various statistical speech processing problems. We quantitatively evaluated the performance of our method in terms of an Fujisaki-model parameter estimation accuracy using real speech data. Experimental results revealed that our method was superior to a state-of-the-art Fujisaki model parameter extractor. Index Terms: speech F0 contours, statistical model, Fujisaki model, hidden Markov model, EM algorithm

Current speech recognition systems, for example, typically use Gaussian mixture models (GMMs), to estimate the observation (or emission) probabilities of hidden Markov models (HMMs), and GMMs are generative models that have only one layer of latent variables. Instead of developing more powerful models, most of the research effort has gone into finding better ways of estimating the GMM parameters so that error rates are decreased or the margin between different classes is increased. The same observation holds for natural language processing (NLP) in which maximum entropy (MaxEnt) models and conditional random fields (CRFs) have been popular for the last decade. Both of these approaches use shallow models whose success largely depends on the use of carefully handcrafted features.

This paper considers the blind separation of the harmonic and percussive components of multichannel music signals. We model the contribution of each source to all mixture channels in the time-frequency domain via a spatial covariance matrix, which encodes its spatial characteristics, and a scalar spectral variance, which represents its spectral structure. We then exploit the spatial continuity and the different spectral continuity structures of harmonic and percussive components as prior information to derive maximum a posteriori (MAP) estimates of the parameters using the expectation-maximization (EM) algorithm. Experimental results over professional musical mixtures show the effectiveness of the proposed approach.

We propose a new statistical model, named Hierarchical Topic Trajectory Model (HTTM), for acquiring a dynamically changing topic model that represents the relationship between video frames and associated text labels. Model parameter estimation, annotation and retrieval can be executed within a unified framework with a few computation. It is also easy to add new modals such as audio signal and geotags. Preliminary experiments on video annotation task with manually annotated video dataset indicate that our proposed method can improve the annotation accuracy.

This paper presents a nonparametric Bayesian extension of non-negative matrix factorization (NMF) for music signal analysis. Instrument sounds often exhibit non-stationary spectral characteristics. We introduce infinite-state spectral bases into NMF to represent time-varying spectra in polyphonic music signals. We describe our extension of NMF with infinite-state spectral bases generated by the Dirichlet process in a statistical framework, derive an efficient optimization algorithm based on collapsed variational inference, and validate the framework on audio data.

Multipitch estimation techniques are widely used for music transcription and acquisition of musical data from digital signals. In this paper, we propose a flexible harmonic temporal timbre model to decompose the spectral energy of the signal in the time-frequency domain into individual pitched notes. Each note is modeled with a 2-dimensional Gaussian mixture. Unlike previous approaches, the proposed model is able to represent not only the harmonic partials but also the inharmonic attack of each note. We derive an Expectation-Maximization (EM) algorithm to estimate the parameters of this model and illustrate the higher performance of the proposed algorithm than NMF algorithm and HTC algorithm for the task of multipitch estimation over synthetic and real-world data.

Context-dependent HMMs are commonly used in speech recognition. Parameter sharing needed for this model can be realized by two methods: context clustering or tied-mixture. In speech recognition, the former is reported to be more precise. However, there is some difficulty in applying context clustering to handwritten word recognition, since the distribution of each character is typically a mixture of different distributions, such as block-printed, cursive, etc. For this reason, successful results reported so far are limited to the tied-mixture approach. To deal with this problem, we propose a novel parameter tying method ``Partial Tied-Mixture", where the Gaussian Mixture Model (GMM) consists of a portion of all Gaussians. Furthermore, we derive a method to concurrently optimize context clustering and GMM. Experiments on the CEDAR database show that the proposed method outperforms tied-mixture both in terms of precision and computational cost.

Observation of speech spectrum leads to the fact that speech has a specific spectral fluctuation pattern both along time and frequency. In this paper, we integrate the usage of this nature in a multi-feature approach for voice activity detection. The effect of separating such specific spectral fluctuation using multi-stage HPSS (Harmonic-Percussive Sound Separation) has been analyzed over conventional features in voice activity detection, reducing frame-wise detection error by up to 78%, depending on the SNR conditions and noise type. The multi-feature approach has been tested using Hidden Markov Models to model the features stream as a sequence, which has out-performed standard and similar VAD proposals in utterance-based tests intended for automatic speech recognition.

This paper describes novel methods for diffuse noise suppression using crystal-shaped microphone arrays. The two-stage processing of the observed signals by the Minimum Variance Distortionless Response (MVDR) beamformer and the subsequent Wiener post-filter is effective for diffuse noise suppression and gives the linear minimum mean square error (LMMSE) estimator of the target signal. It is essential in this framework to accurately estimate the short-time power spectrum and the steering vectors of the target signal from the noisy observations. Our methods diagonalize the spatial noise covariance matrix and utilizes the denoised off-diagonal entries of the spatial covariance matrix to accurately estimate the short-time power spectrum and the steering vectors of the target signal. We employ crystal arrays, certain classes of crystal-shaped array geometries, which make it possible to diagonalize the unknown noise covariance matrix by a constant unitary matrix regardless of its value as long as noise meets an isotropy condition. It is shown through experiments with simulated and real environmental noise that the proposed methods outperform previous methods substantially for real world noise and in the presence of reverberation.

This paper discusses a new approach for clustering sequences of bar-long percussive and bass-line patterns in audio music collections and its application to genre classification. Many musical genres and styles are characterized by two kinds of distinct representative patterns, i.e., percussive patterns and bass-line patterns. So far, in most automatic genre classification systems, rhythmic and bass melody information has not been effectively used. In order to extract bar-long unit rhythmic patterns for a music collection, we propose a clustering method based on one-pass dynamic programming andk-means clustering. For clustering bass-line patterns, a method based onk-means clustering capable of handling pitch-shifting is proposed. After extracting these two fundamental kinds of patterns for each style/genre, feature vectors which are suitable for representing information about the patterns are proposed for supervised learning. Experimental results show that the automatically calculated rhythmic pattern information and bass pattern information can be used to effectively classify musical genre/style and improve upon current approaches based on timbral features.

This paper describes a new design of the Wiener post-filter for diffuse noise suppression. The Wiener post-filter is well-known as an effective post-processing of the minimum variance distortionless response beamformer, and its output is the optimal estimate of the target signal in the sense of the minimum mean square error. It is essential to accurately estimate the target power spectrum from the observed signals contaminated by noise when designing the Wiener post-filter. In our method, it is estimated from the imaginary parts of the inter-channel observation cross-spectra, under the assumption that the inter-channel noise cross-spectra are real-valued. The post-filter is designed using the estimate and this design is shown to be effective even for a small-sized array through experiments using simulated and real environmental noise.

In this paper, we propose a new method of blind source separation (BSS) for music signals. Our method has the following characteristics: 1) the method is a combination of the sparseness-based model of source signals and the factorized basis model in nonnegative matrix factorization (NMF), 2) it is assumed that only one basis which structure source signals is active at each time-frequency bin of the observed signals, in order to degrade the degree of freedom, 3) parameter estimation algorithm is based on the EM algorithm regarding the index of the only one active basis as the hidden variable. We develop the formulation at a different point from NMF and show source separation performance in some simulation experiments.

In this paper, we present a simple iterative algorithm for range-difference (RD) based localization. The RD-based localization is a kind of nonlinear optimization problem and generally it has no closed-form solution. Through auxiliary function approach, we derive iterative update rules without any tuning parameters, which just consists of 1) averaging source-sensor distances, 2) averaging the source positions estimated by updating source-sensor distance on each sensor with the source-direction fixed. Due to the resemblance of the iterative averaging to k-means clustering, we call it r-means localization. The convergence of the algorithm is guaranteed. The acceleration of the convergence is also investigated.

Estimation of melody line in homophonic music audio signals is a challenging subject of study. Some of the difficulties are derived from presence of accompanying components. To overcome those difficulties, we propose a method to enhance melodic components in music audio signals. The enhancement algorithm uses fluctuation and shortness of melodic components, which we call temporal-variability. We also discuss a melody tracking algorithm, which can be simple thanks to the preprocessing. In this paper, we describe the enhancement method and tracking method, and show the experimental results that supports the efficiency of our methods.

This paper discusses an approach for the feature extraction for audio mood classification which is an important and tough problem in the field of music information retrieval (MIR). In this task the timbral information has been widely used, however many musical moods are characterized not only by timbral information but also by musical scale and temporal features such as rhythm patterns and bass-line patterns. In particular, modern music pieces mostly have certain fixed rhythm and bass-line patterns, and these patterns can characterize the impression of songs. We have proposed the extraction of rhythm and bass-line patterns, and these unit pattern analysis are combined with statistical feature extraction for mood classification. Experimental results show that the automatically calculated unit pattern information can be used to effectively classify musical mood.

We discuss an HMM-based method for detecting the chord sequence from musical acoustic signals using percussion-suppressed, Fourier-transformed chroma and delta-chroma features. To reduce the interference often caused by percussive sounds in popular music, we use Harmonic/Percussive Sound Separation (HPSS) technique to suppress percussive sounds and to emphasize harmonic sound components. We also use the Fourier transform of chroma to approximately diagonalize the covariance matrix of feature parameters so as to reduce the number of model parameters without degrading performance. It is shown that HMM with the new features yields higher recognition rates (the best in MIREX 2008 audio chord detection task) than that with conventional features.

This paper discusses about effective speech characterization for off-line voice activity detection (VAD), which is an important step prior to speech data mining. Five different natures of speech are examined; energy, spectral shape, periodicity, phonetic variation, and spectral fluctuation, the latter observed from a new point of view. Specific spectral fluctuation patterns of speech have been analyzed using multi-stage Harmonic/Percussive Sound Separation algorithm. We compared the performance of the features, and various combinations, to evaluate their robustness in multiple noise environments. The combined approach outperformed the baseline of CENSREC-1-C evaluation framework. The results suggest that the proposed feature extraction approach can improve state of the art VAD methods.

Although considerable effort has been devoted to both fundamental frequency (F0) and spectral envelope estimation in the field of speech processing, the problem of determiningF0and spectral envelopes has largely been tackled independently. IfF0were known in advance, then the spectral envelope could be estimated very reliably. On the other hand, if the spectral envelope were known in advance, then we could obtain a reliableF0estimate.F0and the spectral envelope, each of which is a prerequisite of the other, should thus be estimated jointly rather than independently in succession. On this basis, we develop a parametric speech spectrum model that allows us to estimate theF0and spectral envelope simultaneously. We confirmed experimentally the significant advantage of this joint estimation approach for bothF0estimation and spectral envelope estimation.

This paper presents a new sparse representation for acoustic signals which is based on a mixing model defined in the complex-spectrum domain (where additivity holds), and allows us to extract recurrent patterns of magnitude spectra that underlie observed complex spectra and the phase estimates of constituent signals. An efficient iterative algorithm is derived, which reduces to the multiplicative update algorithm for non-negative matrix factorization developed by Lee under a particular condition.

This paper discusses an approach to extract constituent percussive bar-long patterns in a music piece given as acoustic signal and to analyze the music structure with a map of constituent rhythmic patterns. Possible applications include music genre classification, music information retrieval (MIR) and music modification such as replacing rhythmic patterns with others. We propose a mathematical method based on One-pass DP algorithm and k-means clustering to extract unit percussive rhythmic patterns. As the result of identifying and localization the unit patterns in the entire piece, we obtained a music structure in the form of a map of rhythmic patterns.

Many musical genres and styles are characterized by distinct representative rhythmic patterns. In most automatic genre classification systems global statistical features based on timbral dynamics such as mel-frequency cepstral coefficients (MFCC) are utilized but so far rhythmic information has not so effectively been used. In order to extract bar-long unit rhythmic patterns for a music collection we propose a clustering method based on one-pass dynamic programming and k-means clustering. After extracting the fundamental rhythmic patterns for each style/genre a pattern occurrence histogram is calculated and used as a feature vector for supervised learning. Experimental results show that the automatically calculated rhythmic pattern information can be used to effectively classify musical genre/style and improve upon current approaches based on timbral features.

We present noise robust automatic speech recognition (ASR) using sparseness-based underdetermined blind source separation (BSS) technique. As a representative underdetermined BSS method, we utilized time-frequency masking in this paper. Although time-frequency masking is able to separate target speech from interferences effectively, one should consider two problems. One is that masking does not work well in noisy or reverberant environment. Another is that masking itself might cause some distortion of the target speech. For the former, we apply our time-frequency masking method [7] which can separate the target signal robustly even in noisy and reverberant environment. Next, investigating the distortion caused by time-frequency masking, we reveal following facts through experiments: 1) soft mask is better than binary mask in terms of recognition performance and 2) cepstral mean normalization (CMN) reduces the distortion, especially for that caused by soft mask. At the end, we evaluate the recognition performance of our method in noisy and reverberant real environment. Index Terms: time-frequency mask, speech sparseness, blind source separation, stereo-input, robust ASR

This paper describes a new framework for extracting the target signal in diffuse noise environments. We utilize crystal arrays, a certain class of symmetrical microphone arrays with crystal-like geometries, which enable interchannel decorrelation of isotropic noise without knowing the value of its covariance matrix. We refer to this decorrelation as blind noise decorrelation. Using an improved estimation of the signal power spectrum obtained by the blind noise decorrelation, the multichannel Wiener filter is properly implemented, which is the optimal estimator of the target signal in the minimum mean square error sense. Simulated experiments have shown the effectiveness of the proposed method.

We introduce in this paper an auxiliary function approach to parameter estimation of the constrained sinusoidal model, which enables us to derive a complex-spectrum-domain EM-like multiple F0estimation algorithm. Through simulations, we evaluated the performance of the presented method in the ability to avoid locally optimal solutions. We implemented a monaural speech separation system based on the presented method and confirmed its performance on compound signals of real speech.

In this paper, we discuss a new approach named Harmonic-Temporal-Timbral Clustering (HTTC) for the analysis of single-channel audio signal of multi-instrument polyphonic music to estimate the pitch, onset timing, power and duration of all the acoustic events and to classify them into timbre categories simultaneously. Each acoustic event is modeled by a harmonic structure and a smooth envelope both represented by Gaussian mixtures. Based on the similarity between these spectro-temporal structures, timbres are clustered to form timbre categories. The entire process is mathematically formulated as a minimization problem for the I-divergence between the HTTC parametric model and the observed spectrogram of the music audio signal to simultaneously update harmonic, temporal and timbral model parameters through the EM algorithm. Some experimental results are presented to discuss the performance of the algorithm.

Newborns must learn to structure incoming acoustic information into segments, words, phrases, etc., before they can start to learn language. This process is thought to rely on modulation structure of the speech waveform induced by segmental or prosodic regularities within the speech heard by the infant. Here, we investigate the process by which the initial acoustic processing required by modulation analysis can itself be tuned by exposure to the regularities of speech. Starting from the classic definition of modulation, as applied within channels of the peripheral filter, we formulate a mathematical framework in which the structure of initial spectral filtering is adapted for modulation analysis. Our working hypothesis is that the human ear and brain are adapted to the analysis of modulation, via a data-driven learning process on the scale of development (or possibly evolution). Simulation results are presented and a comparison with filterbanks classically used in signal processing is done.

In this paper, we evaluate a method for on-line handwritten Kanji character recognition by describing the structure of Kanji using stochastic context-free grammar (SCFG), and extend it in order to recognize Kanji strings. In this method, we turn attention to the hierarchical structure of Kanji characters which consist of character-parts and strokes, and consider all character patterns or strings to be generated from SCFG with stochastic stroke shape and position relationship between character-parts. Describing Kanji with a few stroke shape and relative position labels, the method enables efficient training and thus robust recognition. We evaluated the recognition performance on several domains of Kanji, and on Kanji strings consist of 2 or 3 characters and gained the recognition rate of 99.29 - 97.40% for characters and 90.80% for strings.

This paper introduces a new music signal processing method to extract multiple fundamental frequencies, which we call specmurt analysis. In contrast with cepstrum which is the inverse Fourier transform of log-scaled power spectrum with linear frequency, specmurt is defined as the inverse Fourier transform of linear power spectrum with log-scaled frequency. Assuming that all tones in a polyphonic sound have a common harmonic pattern, the sound spectrum can be regarded as a sum of linearly stretched common harmonic structures along frequency. In the log-frequency domain, it is formulated as the convolution of a common harmonic structure and the distribution density of the fundamental frequencies of multiple tones. The fundamental frequency distribution can be found by deconvolving the observed spectrum with the assumed common harmonic structure, where the common harmonic structure is given heuristically or quasi-optimized with an iterative algorithm. The efficiency of specmurt analysis is experimentally demonstrated through generation of a piano-roll-like display from a polyphonic music signal and automatic sound-to-MIDI conversion. Multipitch estimation accuracy is evaluated over several polyphonic music signals and compared with manually annotated MIDI data.

We discuss automatic music transcription from audio input to music score by integrating our probabilistic approaches to multipitch spectral analysis, rhythm recognition and tempo estimation. In spectral analysis, acoustic energies in spectrogram are clustered into acoustic objects (i.e., music notes) with our method called harmonic-temporal-structured clustering (HTC) utilizing EM algorithm over a structured Gaussian mixture with constraints of harmonic structure and temporal smoothness. After onset and offset timings are found from separated energies of music notes through note power envelope modeling to obtain the piano-roll representation, the rhythm and tempo are simultaneously recognized and estimated in terms of maximum posterior probability given a probabilistic note duration models with HMM (hidden Markov model) and probabilistic "rhythm vocabulary." Variable tempo is also modeled by a smooth analytic curve. Rhythm recognition and tempo estimation is alternately performed to iteratively maximize the joint posterior probability. Experimental results are also shown.

We present in this paper a novel F0contour estimation method based on a parametric description of the wavelet power spectrum of speech that accounts for its structure simultaneously in time and frequency directions. We model the speech spectrum as a sequence of spectral clusters governed by a smooth common F0contour expressed as a spline curve. The harmonic and temporal structure of these clusters and their common F0contour are estimated simultaneously. Through experimental comparisons with existing methods, we show that our algorithm is competitive on clean single-speaker speech, and that it outperforms existing methods both in the presence of noise and for the estimation of multiple F0contours of cochannel concurrent speech.

This paper discusses model-based rhythm and tempo analysis of music data in the MIDI format. The data is assumed to be obtained from a module performing multi-pitch analysis of music acoustic signals inside an automatic transcription system. In performed music, observed note lengths and local tempo fluctuate from the nominal note lengths and long-term tempo. Applying the framework of continuous speech recognition to rhythm recognition, we take a probabilistic top-down approach on the joint estimation of rhythm and tempo from the performed onset events in MIDI data. Short-term rhythm patterns are extracted from existing music samples and form a "rhythm vocabulary." Local tempo is represented by a smooth curve. The entire problem is formulated as an integrated optimization problem to maximize a posterior probability, which can be solved by an iterative algorithm which alternately estimates rhythm and tempo. Evaluation of the algorithm through various experiments is also presented.

This paper presents a new approach to online recognition of handwritten Kanji characters focusing on their hierarchical structure. Stochastic context-free grammar (SCFG) is introduced to represent the Kanji character generating process in combination with Hidden Markov Models (HMM) representing Kanji substrokes and to improve the recognition accuracy of important and frequently used Kanji characters in which inter-stroke relative positions play important roles. Combining the stroke likelihood and the relative-position likelihood between character-parts in the parsing process is expected to compensate their ambiguities. By modeling relative positions and share the models across distinct Kanji categories, a small training data can yield effective results and enables us to recognize Kanji simply by defining the SCFG rules to represent their structures without training data. Experimental results on an online handwritten Kanji database from JAIST (Japan Advanced Institute of Science and Technology) showed significant improvements in the recognition rates of some important Kanji with relatively fewer strokes and also showed little difference between the trained- and the non-trained Kanji in recognition rates.

In this paper, we propose a novel system to localize a sound source in any 2D directions using only two microphones. In our system, the two microphones are asymmetrically placed on a sphere, thus, (1) the diffraction by the sphere and the asymmetrical arrangement of the microphones yield the localization cue including the front-back judgment, and (2) unlike the dummy head system, no previous measurements are necessary due to the analytical representation of the sphere diffraction. To deal with reverberation or ambient noises, we consider the maximum likelihood estimation of the direction of arrival with a diffused noise model on a sphere. We present a real system that we built through the investigation of the optimal microphone arrangement for speech, and give experimental results in real environment.

This paper proposes a multipitch analyzer called the harmonic temporal structured clustering (HTC) method, that jointly estimates pitch, intensity, onset, duration, etc., of each underlying source in a multipitch audio signal. HTC decomposes the energy patterns diffused in time-frequency space, i.e., the power spectrum time series, into distinct clusters such that each has originated from a single source. The problem is equivalent to approximating the observed power spectrum time series by superimposed HTC source models, whose parameters are associated with the acoustic features that we wish to extract. The update equations of the HTC are explicitly derived by formulating the HTC source model with a Gaussian kernel representation. We verified through experiments the potential of the HTC method

This paper proposes a novel $F_{0}$ contour estimation algorithm based on a precise parametric description of the voiced parts of speech derived from the power spectrum. The algorithm is able to perform in a wide variety of noisy environments as well as to estimate the $F_{0}$ s of cochannel concurrent speech. The speech spectrum is modeled as a sequence of spectral clusters governed by a common $F_{0}$ contour expressed as a spline curve. These clusters are obtained by an unsupervised 2-D time-frequency clustering of the power density using a new formulation of the EM algorithm, and their common $F_{0}$ contour is estimated at the same time. A smooth $F_{0}$ contour is extracted for the whole utterance, linking together its voiced parts. A noise model is used to cope with nonharmonic background noise, which would otherwise interfere with the clustering of the harmonic portions of speech. We evaluate our algorithm in comparison with existing methods on several tasks, and show 1) that it is competitive on clean single-speaker speech, 2) that it outperforms existing methods in the presence of noise, and 3) that it outperforms existing methods for the estimation of multiple $F_{0}$ contours of cochannel concurrent speech.

In environment with considerably long reverberation time, each frame of speech is affected by energy components from the preceding frames. Therefore, to adapt parameters of a state of HMM, it becomes necessary to consider these frames, and compute their contributions to current state. However, these speech frames preceding to a state of HMM are not known during adaptation of the models. In this paper, we propose to use preceding states as units of preceding speech segments, estimate their contributions to current state in maximum likelihood manner, and adapt models by accounting their contributions. When clean models were adapted by proposed method for a speaker-dependent isolated word recognition task, word accuracy of the system typically increased from 67.6% to 83.2%, and from 44.8% to 72.5%, for channel distorted speech simulated by linear convolution of clean speech and impulse responses with reverberation time (T60) of 310 ms and 780 ms, respectively

We have been working on a new speech analyzer based on a parametric representation of speech governed by the F0 parameter, towards practical human-machine interfaces. As a precise estimation of the frequency response of the vocal tract from a real speech signal requires the power of each component of the harmonic structure to be accurately estimated, one hopes to have a high-precision estimation of F0. At the same time, under the empirical constraint that speech spectral envelopes are usually smooth in the power domain, half pitch errors can be significantly avoided. Therefore, F0 and the envelope should be estimated jointly rather than separately through an optimal estimation of the spectral envelope and the spectral fine structure. In this article, we introduce a new speech analysis method using a spectral model with a composite function of envelope and fine structure models. Index Terms: parametric speech analyzer, speech synthesis, pitch estimation, spectral envelope estimation.

The paper describes a novel approach for audio stream segregation of a multi-pitch music signal. We propose a parameter-constrained time-frequency spectrum model expressing both a harmonic spectral structure and a temporal curve of the power envelope with Gaussian kernels. MAP estimation of the model parameters using the EM algorithm provides fundamental frequency, onset and offset time, spectral envelope and power envelope of every underlying audio stream. Our proposed method showed high accuracy in a pitch name estimation task of several pieces of real music performance data.

In environment with considerably long reverberation time, each frame of speech is affected by reflected energy components from the preceding frames. Therefore to adapt model parameters of a state, it becomes necessary to consider these frames, and compute their contributions to current state. However, these clean speech frames preceding to a state of HMM are not known during adaptation of the models. This paper describes a method to estimate the preceding frames for a state in HMM, by splitting the state into a number of substates. The estimated sequence of frames can then be used to find reflected energy component for the state and compensate its parameters. The effectiveness of the method was confirmed by the experimental results on an isolated-word recognition task.

A method for the separation of harmonic structures of cochannel input concurrent sounds is described. A model for multiple harmonic structures is constructed with a mixture of tied Gaussian mixtures, from which a single harmonic structure is modeled. Our algorithm enables estimation of both the number and the shape of the underlying harmonic structures, based on a maximum likelihood estimation of the model parameters using the EM algorithm and an information criterion. It operates without restriction on the number of mixed sounds and varieties of sound sources, and extracts accurate fundamental frequencies continuously with simple procedures in the spectral domain. Experiments showed high performance of the algorithm for both simultaneous speech and polyphonic music.

Abstract This paper describes a multi-pitch tracking algorithmof 1-channel simultaneous multiple speech. The algo-rithm selectively carries out the two alternative processesat each frame: frame-independent-process and frame-dependent-process. The former is the one we have previ-ously proposed[6], that gives good estimates of the num-ber of speakers and F 0 s with a single-frame-processing.The latter corresponds to the topic mainly described inthis paper, that recursively tracks F 0 s using nonlinearKalman filtering. We tested our algorithm on simulta-neous speech signal data and showed higher performancethan when the frame-independent-process was only used. 1. Introduction 1-channel multi-pitch estimation technique may con-tribute to various applications, such as spontaneous dia-logue speech recognition, that allows competitive speech,noise robust speech recognition, especially where noiseis a harmonic signal(e.g., telephone ring, back groundmusic, etc.), and also many music applications. How-ever, multi-pitch estimation of non-stationary signalsis hardly simple due to the complex factors such asspectral overlap, poor frequency resolution and spec-tral widening in short-time analysis, etc. Various ap-proaches concerning to this problem have convention-ally been attempted [1, 2, 3], while two important taskshave been left unsolved. Firstly, there has been no ro-bust way of estimating the number of speakers, andmost of the methods were obliged to assume for sim-plicity that the number is known a priori. Secondly, thedouble/half(harmonics/subharmonics) pitch error has stillbeen one of the most critical problem where convinc-ing solutions are not yet proposed. One may say bothproblems share the same difficulty of defining physicallyor mathematically proper criteria. Until now, we haveproposed a GMM(Gaussian mixture model)-based multi-pitch estimation algorithm that works as a single-frame-processing and gives solutions to the two tasks statedabove according to the information criterion[6]. This al-gorithm has not yet taken into account any time depen-dency property, that would ensure the improvements in

This paper presents a technique for estimating HMM model parameters for noisy speech from given clean speech HMM and noise HMM. The model parameters are estimated by approximating the non-linear function governing the relationship between speech and noise, by a Lagrange polynomial, and thus enabling the distribution of corrupted speech parameters to have a closed form. The method is computationally efficient, and the experimental results showed significant improvement in recognition performance of noisy speech with this approach. Typically, word accuracy increased from 9.2% with clean model to 82.8% with the model composed by the proposed method as compared to 45.4% with the model composed by PMC Log-normal approximation, on an isolated word recognition task for exhibition hall noise added at 10 dB SNR.

We propose a novel principle based on Complex Spectrum Circle Centroid (CSCC) for restoring complex spectrum of the target signal from multiple microphone input signals in a noisy environment. If noise arrives at multiple microphones with different time delays relative to the target signal, the observed noisy signals lie on a circle in the complex spectrum plane from which the target signal is restored by finding the centroid of the circle. Unlike most of existing methods for noise reduction such as ICA, AMNOR and beamforming, this nonlinear operation is applicable to any type of noise including non-stationary, moving, signal-correlated, nonplanar, and spoken noises, without identifying the noise direction and training parameters.

This paper describes a method of generating a Kanji hierarchical structured dictionary for stroke-number and stroke-order free handwriting recognition based on sub-stroke HMM. In stroke-based methods, a large number of stroke-order variations can be easily expressed by just adding different stroke sequences to the dictionary and it is not necessary to train new reference patterns. The hierarchical structured dictionary has an advantage that thousands of stroke-order variations of Kanji characters can be produced using a small number of stroke-order rules defining Kanji parts. Moreover, the recognition speed is fast since common sequences are shared in a substroke network, even if the total number of stroke-order combinations becomes enormous practically. In experiments, 300 different stroke-order rules of Kanji parts were statistically chosen by using 60 writers' handwritings of 1016 educational Kanji characters. By adding these new stroke-order rules to the dictionary, about 9000 variations of different stroke-orders were generated for 2965 JIS 1st level Kanji characters. As a result, we successfully improved the recognition accuracy from 82.6% to 90.2% for stroke-order free handwritings.

This study discusses the subject of training data selection for neural networks using back propagation. We have made only one assumption that there are no overlapping of training data belonging to different classes, in other words the training data is linearly/semi-linearly separable . Training data is analyzed and the data that affect the learning process are selected based on the idea of Critical points. The proposed method is applied to a classification problem where the task is to recognize the characters A,C and B,D. The experimental results show that in case of batch mode the proposed method takes almost 1/7 of real and 1/10 of user training time required for conventional method. On the other hand in case of online mode the proposed method takes 1/3 of training epochs, 1/9 of real and 1/20 of user and 1/3 system time required for the conventional method. The classification rate of training and testing data are the same as it is with the conventional method.

A new Jacobian approach that linearly decomposes the composite of additive noise, multiplicative noise (channel transfer function) and speaker's vocal tract length, and adapts the acoustic model parameters simultaneously to these factors is proposed in this paper. Due to the fact that these factors non-linearly degrade the observed features for speech recognition, existing approaches fail to adapt the acoustic models adequately. Approximating the nonlinear operation by a linear model enables to employ the least square error estimation of the factors and adapt the acoustic model parameters with small amount of speech samples. Speech recognition experiments on ATR isolated word database demonstrate significant reduction of error rates, which supports the effectiveness of the proposed scheme.

Proposes a class of hidden Markov model (HMM) called multiple-regression HMM (MR-HMM) that utilizes auxiliary features such as fundamental frequency (F/sub 0/) and speaking styles that affect spectral parameters to better model the acoustic features of phonemes. Though such auxiliary features are considered to be the factors that degrade the performance of speech recognizers, the proposed MR-HMM adapts its model parameters, i.e. mean vectors of output probability distributions, depending on these auxiliary information to improve the recognition accuracy. Formulation for parameter reestimation of MR-HMM based on the EM algorithm is given in the paper. Experiments of speaker-dependent isolated word recognition demonstrated that MR-HMMs using F/sub 0/ based auxiliary features reduced the error rates by more than 20% compared with the conventional HMMs.

A new method is proposed for online handwriting recognition of Kanji characters. The method employs substroke HMM as minimum units to constitute Japanese Kanji characters and utilizes the direction of pen motion. The main motivation is to fully utilize the continuous speech recognition algorithm by relating sentence speech to Kanji character phonemes to substrokes, and grammar to Kanji structure. The proposed system consists input feature analysis, substroke HMM, a character structure dictionary and a decoder. The present approach has the following advantages over the conventional methods that employ whole character HMM. 1) Much smaller memory requirement for dictionary and models. 2) Fast recognition by employing efficient substroke network search. 3) Capability of recognizing characters not included in the training data if defined as a sequence of substrokes in the dictionary. 4) Capability of recognizing characters written by various different stroke orders with multiple definitions per one character in the dictionary. 5) Easiness in HMM adaptation to the user with a few sample character data.

A new class of Support Vector Machine (SVM) that is applica- ble to sequential-pattern recognition such as speech recognition is developed by incorporating an idea of non-linear time alignment into the kernel function. Since the time-alignment operation of sequential pattern is embedded in the new kernel function, stan- dard SVM training and classification algorithms can be employed without further modifications. The proposed SVM (DTAK-SVM) is evaluated in speaker-dependent speech recognition experiments of hand-segmented phoneme recognition. Preliminary experimen- tal results show comparable recognition performance with hidden Markov models (HMMs). 1 Introduction Support Vector Machine (SVM) [1] is one of the latest and most successful statistical pattern classifier that utilizes a kernel technique [2, 3]. The basic form of SVM classifier which classifies an input vector x

We propose a new class of hidden Markov model (HMM) called asynchronous-transition HMM (AT-HMM). Opposed to conventional HMMs where hidden state transition occurs simultaneously to all features, the new class of HMM allows state transitions asynchronized between individual features to better model asynchronous timings of acoustic feature changes. In this paper, we focus on a particular class of AT-HMM with sequential constraints based on a novel concept of "state tying along time". To maximize the advantage of the new model, we also introduce a feature-wise state tying technique. Speaker-dependent speech recognition experiments demonstrated error reduction rates more than 30% and 50% in phoneme and isolated word recognitions, respectively, compared with conventional HMMs.

A sharable software repository for Japanese LVCSR (Large Vocabulary Continuous Speech Recognition) is introduced. It is designed as a baseline platform for research and developed by researchers of different academic institutes under a governmental support. The repository consists of a recognition engine (Julius), Japanese acoustic models and statistical language models as well as Japanese morphological analysis tools. These modules can be easily integrated and replaced under a plug-and-play framework, which makes it possible to fairly evaluate components and to develop specific application systems. Assessment of these modules and systems in a 20000-word dictation task is reported. The software repository is freely available to the public.

We propose a novel method for clustering allophones called Feature-Dependent Allophone Clustering (FD-AC) that determines feature-dependent HMM topology automatically. Existing methods for allophone clustering are based on parameter sharing between the allophone models that resemble each other in behaviors of feature vector sequences. However, all the features of the vector sequences may not necessarily have a common allophone clustering structures It is considered that the vector sequences can be better modeled by allocating the optimal allophone clustering structure to each feature. In this paper, we propose Feature-Dependent Successive State Splitting (FD-SSS) as an implementation of FD-AC. In speaker-dependent continuous phoneme recognition experiments, HMMs created by FD-SSS reduced the error rates by about 10% compared with the conventional HMMs that have a common allophone clustering structure for all the features.

An extension of Jacobian Adaptation (JA) of HMMs for degraded speech recognition is presented in which appropriate set of initial models is selected from a number of initial-model sets designed for different noise environments. Based on the first order Taylor series approximation in the acoustic feature domain, JA adapts the acoustic model parameters trained in the initial noise environment A to the new environment B much faster than PMC that creates the acoustic models for the target environment from scratch. Despite the advantage of JA to PMC, JA has a theoretical limitation that the change of acoustic parameters from the environment A to B should be small in order that the linear approximation holds. To extend the coverage of JA, the ideas of multiple sets of initial models and their automatic selection scheme are discussed. Speaker-dependent isolated-word recognition experiments are carried out to evaluate the proposed method. 1.

This paper proposes two-step generation of a variable-length class-based language model that integrates local and global constraints. In the first-step, an initial class set is recursively designed using local constraints. Word elements for each class are determined using Kullback divergence and total entropy. In the second step, the word classes are recursively and words are iteratively recreated, by grouping consecutive words to generate longer units and by splitting the initial classes into finer classes. These operations in the second step are carried out selectively, taking into account local and global constraints on the basis of a minimum entropy criterion. Experiments showed that the perplexity of the proposed initial class set is superior to that of the conventional part-of-speech class, and the perplexity of the variable-word-length model consequently becomes lower. Furthermore, this two-step model generation approach greatly reduces the training time.

Unsupervised speaker adaptation plays an important role in "batch dictation", the aim of which is to automatically transcribe large amounts of recorded dictation using speech recognition. In the case of unsupervised speaker adaptation which uses recognition results of target speech as the means of supervision, erroneous recognition results degrade the quality of the adapted acoustic models. This paper presents a new supervision selection method. By using this method, correction of the first candidate is judged based on the likelihood ratio of the first and the second candidates. This method eliminates erroneous recognition results and corresponding speech data from the adaptive training data. We implemented this method in the iterative unsupervised speaker adaptation procedure. It is shown that the recognition errors are drastically reduced by 50% in a practical application of batch-style speech-to-text conversion of recorded dictation of Japanese medical diagnoses compared with speaker-independent recognition.

This paper describes a Jacobian approach to fast adaptation of acoustic models to noisy environments. Acoustic models under a noise assumption are compensated by Jacobian matrices with the difference between assumed and observed noise cepstra. Detailed mathematical formulation and algorithm derivation are presented. Experiments showed that when a small amount of training data is given, this approach outperforms the existing approaches (such as PMC and NOVO) for composing a model from speech and noise models. It drastically reduces computational cost by replacing the complicated computation of model composition by simple matrix arithmetic and enables real-time environmental noise adaptation. Combination with spectrum subtraction is also discussed.

This paper proposes a new type of acoustic model called the discrete mixture HMM (DMHMM). As large scale speech databases have been constructed for speaker-independent HMMs, continuous mixture HMMs (CMHMMs) are needed to increase the number of mixture components in order to represent complex distributions. This leads to a high computational cost for calculating the output probabilities. The DMHMM represents the feature parameter space by using the mixtures of multivariate distributions in the same way as the diagonal covariance CMHMM. Instead of using Gaussian mixtures to represent the feature distributions in each dimension, the DMHMM uses the mixtures of the discrete distributions based on scalar quantization (SQ). Since the discrete distribution has a higher degree-of-freedom in terms of representation, the DMHMM is advantageous in representing the feature distributions efficiently with fewer mixture components. In isolated word recognition experiments for telephone speech, we have found that the DMHMM outperformed the CMHMMs when those models had the same number of mixture components.

This paper proposes a new scheme of the tied-structure for constraining an HMM structure to increase training efficiency and recognition robustness. In conventional tied-structure approaches, tied parameters (or distributions, states, allophones) share the same value for decreasing model complexity. In the new framework, the tied parameters are correlated to each other rather than share the same value. To establish the appropriate correlation between model parameters, a speaker-independent initial model is trained using multiple sets of speaker-dependent data. The transfer vectors from the initial mean vectors of Gaussian distributions to the trained mean vectors are clustered to obtain sets of mean vectors that are mutually correlated across speakers. This kind of speaker-independent parameter-correlated structure can yield a lower degree-of-freedom for the model compared to the baseline speaker-independent model. Using the model with the parameter-correlated structure, speaker adaptation experiments are performed to demonstrate the higher training efficiency of the model compared to conventional speaker adaptation techniques without the correlation structure.

This paper describes an efficient method of attaining the highest level of recognition performance yet achieved in minimum classification error (MCE) training for a small amount of data. This method combines MCE and vector-field-smoothed Bayesian learning called MAP/VFS. In the proposed method, the training capability of MCE in robust acoustic modeling can be significantly enhanced with MAP/VFS. In the method, MCE training is performed using an initial model trained through MAP/VFS. The same data are used in both training. For speaker adaptation using 50-word training data, the error reduction rate drastically rises to 47% compared with 16.5% when using only MCE. This high rate, in which 39% is due to MAP, an additional 4% is due to VFS, and a further improvement of 4% is due to MCE, can be attained by enhancing MCE training capability with MAP/VFS.

Describes an automatic batch-style dictation paradigm in which the entire dictated speech is fully utilized for speaker adaptation and is recognized using the speaker adaptation results. The key point is that the same speech data is used both for recognition as the target and for speaker adaptation. Two steps, speech recognition and speaker adaptation which uses the recognition results as means of supervision, are iterated to maximize the advantage of closed-data speaker adaptation. Recognition errors are reduced by 37% in a practical application of batch-style speech-to-text conversion of recorded dictation of Japanese medical diagnoses compared to speaker-independent recognition. To select only reliable recognition results, a supervision improvement procedure is used, by which erroneous recognition results can be eliminated from the supervision. In this procedure, 59-74% of the data are extracted from the tentative recognition results, and their reliability is 89-93%. This procedure also reduces recognition errors by 45%.

This paper describes a Viterbi search algorithm for continuous speech recognition using context-dependent phone models under the constraint defined by a context-free grammar (CFG).It is based on a frame synchronous LR parser which dynamically generates a finite state network (FSN) from the CFG with an efficient path merging mechanism.Full context-dependency (intra-and interword context) is taken into account in the likelihood calculation process.This paper first describes the algorithm and the processing mechanism, then compares the experimental results of our algorithm and the conventional tree-based HMM-LR speech recognition algorithm which uses HMMs and an LR parser in phone-synchronous processing.The experiments show that our algorithm runs faster than the conventional HMM-LR algorithm with an equivalent recognition accuracy.

This paper describes an algorithm for reducing the amount of arithmetic operations in the likelihood computation of continuous mixture HMM (CMHMM) with diagonal covariance matrices while retaining high performance. The key points are the use of the scalar quantization of the input observation vector components and table look-up. These make multiplication, squaring and division operations entirely unnecessary in the whole HMM computation (i.e., output probability calculation and trellis/Viterbi computation). It is experimentally proved in an large-vocabulary isolated word recognition task that scalar quantization into no less than 16 levels does not cause significant degradation in the speech recognition performance. Scalar quantization is also utilized in the computation truncation for unlikely distributions; the total number of distribution likelihood computations can be reduced by 66% with only a slight performance degradation. This "multiplication-free" HMM algorithm has high potentiality in speech recognition applications on personal computers.

One of the problems with context-dependent HMMs is that a large number of model parameters should be estimated using a limited amount of training data. Parameters that have the same property should be tied in order to represent acoustic models efficiently. This paper proposes four-level tied-structure for phoneme models. The four levels include 1) model level, 2) state level, 3) distribution level, and 4) feature parameter level. Although some techniques have been proposed for the first three levels, feature parameter tying in the fourth level is newly proposed in this paper. We found that feature parameter tying makes it possible to represent 1,600 mean vectors of multivariate Gaussian mixture HMMs by using the combination of 16 representative mean values in each dimension. Experimental results show that feature parameter tying reduces the amount of calculation required for recognition without significant degrading performance. Furthermore, we found that feature parameter tying is also effective for model training.

The paper presents a fast and incremental speaker adaptation method called MAP/VFS, which combines maximum a posteriori (MAP) estimation, or in other words Bayesian learning, with vector field smoothing (VFS). The point is that MAP is an intra-class training scheme while VFS is an inter-class smoothing technique. This is a basic technique for on-line adaptation which will be important in constructing a practical speech recognition system. Speaker adaptation speed of the incremental MAP is experimentally shown to be significantly accelerated by the use of VFS in word-by-word adaptation. The recognition performance of MAP is consistently improved and stabilized by VFS. The word error reduction rate achieved in incrementally adapting a few words of sample data is about 22%.

The paper proposes a tree-structured speaker clustering algorithm and discusses its application to fast speaker adaptation. By tracing the clustering tree from top to bottom, adaptation is performed step-by-step from global to local individuality of speech. This adaptation method employs successive branch selection in the speaker clustering tree rather than parameter training and hence achieves fast adaptation using only a small amount of training data. This speaker adaptation method was applied to a hidden Markov network (HMnet) and evaluated in Japanese phoneme and phrase recognition experiments, in which it significantly outperformed speaker-independent recognition methods. In the phrase recognition experiments, the method reduced the error rate by 26.6% using three phrase utterances (approximately 2.7 seconds).>

The paper proposes an unsupervised speaker adaptation method using "all-phoneme ergodic hidden Markov network" that combines allophonic (context-dependent phone) acoustic models with stochastic language constraints. Hidden Markov networks (HMnet) for allophone modeling and allophonic bigram probabilities derived from a large text database are combined to yield a single large ergodic HMM which represents arbitrary speech signals in a particular language so that the model parameters can be re-estimated using text-unknown speech samples with the Baum-Welch algorithm. Combined with the vector field smoothing (VFS) technique, unsupervised speaker adaptation can be effectively performed. This method experimentally gave fairly better performances compared with the authors' previous unsupervised adaptation method using conventional phonetic HMMs and phoneme bigram probabilities.>

This paper describes the continuous speech recognition subsystem ATREUS which is used as the speech input stage in the experimental speech translation system ASURA. The speech recognition algorithm is SSS-LR/VFS which consists of context-dependent phone models (HMnet), a generalized LR parser, and vector field smoothing for speaker / environment adaptation.

The authors propose a novel method of incorporating pitch information into a hidden Markov model (HMM) phoneme recognizer by exploiting the correlation between pitch and spectral parameters, e.g. cepstrum. Pitch patterns are not used explicitly; instead, spectral parameters are normalized framewise according to the pitch value. Evidence is given to show that the use of pitch information consistently improves the recognition performance. Experiments with 24 phoneme labels showed that the phoneme error rate for fast continuous speech could be improved by about 10%.>

The authors propose an algorithm, successive state splitting (SSS), for simultaneously finding an optimal set of phoneme context classes, an optimal topology, and optimal parameters for hidden Markov models (HMMs) commonly using a maximum likelihood criterion. With this algorithm, a hidden Markov network (HM-Net), which is an efficient representation of phoneme-context-dependent HMMs, can be generated automatically. The authors implemented this algorithm, and tested it on the recognition of six Japanese consonants ( mod b mod , mod d mod , mod g mod , mod m mod , mod n mod and mod N mod ). The HM-Net gave better recognition results with a lower number of total output probability density distributions than conventional phoneme-context-independent mixture Gaussian density HMMs.>

A phoneme filter neural network (PFN) approach to vowel recognition is described. The PFN is a multilayer neural network with fewer hidden units than input units prepared for each of the phoneme categories. Each network is trained as identity mapping by speech data belonging to one phoneme category. In the recognition process, the similarity between the input data and output data is computed for each network. The results of an experiment involving the Japanese vowel recognition task showed that the PFN recognition rates for the top two or more choices are higher than those of a conventional three-layer neural network and the PFN outputs represented candidate likelihoods. It was also confirmed that the PFN has a mapping ability and recognition performance superior to those of the linear K-L transformation method because of the nonlinearity of the PFN.>

The authors propose a phoneme recognition method using pairwise discriminant time-delay neural networks (PD-TDNNs) and show its high recognition performance with experimental results for /b, d, g, m, n, N/. They tested this method on a phoneme recognition task for 18 Japanese consonants (/g, d, g, p, t, k, m, n, N, s, sh, h, z, ch, ts, r, w, y/) and considered its basic performance in detail. In tests on continuous speech using PD-TDNNs trained with the 18 consonant phoneme data in isolated word utterances, the authors obtained a first candidate recognition rate of 76.8%, and 92.9% for the cumulative recognition rate up to third candidates.>

A Japanese continuous-speech recognition system is described which is based on phonetic hidden Markov models (HMMs) combined with two levels of grammatical representations: an intraphase transition network grammar and an interphase dependency grammar. A joint score, combining acoustic likelihood and linguistic certainty factors derived from phonetic HMMs and two levels of grammar, is maximized to obtain the optimal recognition results of sentences. Two efficient algorithms, bidirectional network parsing and breadth-first dependency parsing, are deviced to optimize the joint score globally. The system attains a phrase recognition rate of 80.8% with the intraphase parser only and 86.8% with both the intraphase and interphase parsers, where the perplexity of the phrase syntax is 40. This result shows the effectiveness of the two-level grammar approach.>

A general principle is proposed to solve problems in context-dependent phoneme segment (or subword unit) based speech recognition, namely, how to choose the set of units and how to estimate the context effect missing in the training data. A phoneme environment clustering algorithm, which automatically selects an optimal set of allophones and estimates the missing context, is presented. This algorithm additionally gives the means to analyze coarticulation effects automatically and quantitatively. The problem is formulated as a clustering technique in phoneme environment space to approximate the mapping function from phoneme environment space to phoneme pattern space by a limited number of centroid patterns based on a distortion measure defined on the phoneme pattern space. The algorithm is tested for phoneme recognition and word recognition, and results are discussed.>

This paper reveals that an elegant duality exists between Linear Prediction (LPC) and Composite Sinusoidal Modeling (CSM) from the viewpoint of orthogonal polynomial theory. Both LPC and CSM are formulated as orthogonal polynomial theory with variablesz = e^{j\omega}(LPC) andx = \cos\omega(CSM) where the "inner product" is defined by the integral of a pair of functions of frequency ω weighted by the spectral density function of the signal. This viewpoint leads to the duality of LPC and CSM and reveals the correspondence between parameters in the LPC and CSM domains. Conventional LPC theory (including PARCOR) is shown to explain only half of the theory presented here. The theory of mutual conversion between LPC and CSM yields a new interpretation of LSP (line spectrum pairs) and an alternative algorithm for LSP analysis. Fundamental properties of LSP are derived.