OBJECTIVE: Nocturnal hypoglycemia is a known challenge for people with type 1 diabetes, especially for physically active individuals or those on multiple daily injections. We developed an evidential neural network (ENN) to predict at bedtime the probability and timing of nocturnal hypoglycemia (0-4 vs 4-8 h after bedtime) based on several glucose metrics and physical activity patterns. We utilized these predictions in silico to prescribe bedtime carbohydrates with a Smart Snack intervention specific to the predicted minimum nocturnal glucose and timing of nocturnal hypoglycemia. MATERIALS AND METHODS: We leveraged free-living datasets collected from 366 individuals from the T1DEXI Study and Glooko. Inputs to the ENN used to model nocturnal hypoglycemia were derived from demographic information, continuous glucose monitoring, and physical activity data. We assessed the accuracy of the ENN using area under the receiver operating curve, and the clinical impact of the Smart Snack intervention through simulations. RESULTS: The ENN achieved an area under the receiver operating curve of 0.80 and 0.71 to predict nocturnal hypoglycemic events during 0-4 and 4-8 h after bedtime, respectively, outperforming all evaluated baseline methods. Use of the Smart Snack intervention reduced probability of nocturnal hypoglycemia from 23.9 ± 14.1% to 14.0 ± 13.3% and duration from 7.4 ± 7.0% to 2.4 ± 3.3% in silico. DISCUSSION: Our findings indicate that the ENN-based Smart Snack intervention has the potential to significantly reduce the frequency and duration of nocturnal hypoglycemic events. CONCLUSION: A decision support system that combines prediction of minimum nocturnal glucose and proactive recommendations for bedtime carbohydrate intake might effectively prevent nocturnal hypoglycemia and reduce the burden of glycemic self-management.

Missing values with mixed data types is a common problem in a large number of machine learning applications such as processing of surveys and in different medical applications. Recently, Gaussian copula models have been suggested as a means of performing imputation of missing values using a probabilistic framework. While the present Gaussian copula models have shown to yield state of the art performance, they have two limitations: they are based on an approximation that is fast but may be imprecise and they do not support unordered multinomial variables. We address the first limitation using direct and arbitrarily precise approximations both for model estimation and imputation by using randomized quasi-Monte Carlo procedures. The method we provide has lower errors for the estimated model parameters and the imputed values, compared to previously proposed methods. We also extend the previous Gaussian copula models to include unordered multinomial variables in addition to the present support of ordinal, binary, and continuous variables.

OBJECTIVE: We provide an e-Science perspective on the workflow from risk factor discovery and classification of disease to evaluation of personalized intervention programs. As case studies, we use personalized prostate and breast cancer screenings. MATERIALS AND METHODS: We describe an e-Science initiative in Sweden, e-Science for Cancer Prevention and Control (eCPC), which supports biomarker discovery and offers decision support for personalized intervention strategies. The generic eCPC contribution is a workflow with 4 nodes applied iteratively, and the concept of e-Science signifies systematic use of tools from the mathematical, statistical, data, and computer sciences. RESULTS: The eCPC workflow is illustrated through 2 case studies. For prostate cancer, an in-house personalized screening tool, the Stockholm-3 model (S3M), is presented as an alternative to prostate-specific antigen testing alone. S3M is evaluated in a trial setting and plans for rollout in the population are discussed. For breast cancer, new biomarkers based on breast density and molecular profiles are developed and the US multicenter Women Informed to Screen Depending on Measures (WISDOM) trial is referred to for evaluation. While current eCPC data management uses a traditional data warehouse model, we discuss eCPC-developed features of a coherent data integration platform. DISCUSSION AND CONCLUSION: E-Science tools are a key part of an evidence-based process for personalized medicine. This paper provides a structured workflow from data and models to evaluation of new personalized intervention strategies. The importance of multidisciplinary collaboration is emphasized. Importantly, the generic concepts of the suggested eCPC workflow are transferrable to other disease domains, although each disease will require tailored solutions.

Microsimulation with stochastic life histories is an important tool in the development of public policies. In this article, we use microsimulation to evaluate policies for prostate cancer testing. We implemented the microsimulations as an R package, with pre- and post-processing in R and with the simulations written in C++. Calibrating a microsimulation model with a large population can be computationally expensive. To address this issue, we investigated four forms of parallelism: (i) shared memory parallelism using R; (ii) shared memory parallelism using OpenMP at the C++ level; (iii) distributed memory parallelism using R; and (iv) a hybrid shared/distributed memory parallelism using OpenMP at the C++ level and MPI at the R level. The close coupling between R and C++ offered advantages for ease of software dissemination and the use of high-level R parallelisation methods. However, this combination brought challenges when trying to use shared memory parallelism at the C++ level: the performance gained by hybrid OpenMP/MPI came at the cost of significant re-factoring of the existing code. As a case study, we implemented a prostate cancer model in the microsimulation package. We used this model to investigate whether prostate cancer testing with specific re-testing protocols would reduce harms and maintain any mortality benefit from prostate-specific antigen testing. We showed that four-yearly testing would have a comparable effectiveness and a marked decrease in costs compared with two-yearly testing and current testing. In summary, we developed a microsimulation package in R and assessed the cost-effectiveness of prostate cancer testing. We were able to scale up the microsimulations using a combination of R and C++, however care was required when using shared memory parallelism at the C++ level.

Earlier studies have shown that certain emotional characteristics are best observed at different analysis-frame lengths. When features of multiple modalities are extracted, it is reasonable to believe that different temporal lengths would better model the underlying characteristics that result from different emotions. In this study, we examine the use of such differing timescales in constructing emotion classifiers. A novel fusion method is introduced that utilizes the outputs of individual classifiers that are trained using multi-dimensional inputs with multiple temporal lengths. We used the IEMOCAP database which contains audiovisual information of 10 subjects in dyadic interaction settings. The classification task was performed over three emotional dimensions: valence, activation, and dominance. The results demonstrate the utility of the multimodal-multitemporal approach. Statistically significant improvements in accuracy are seen for in all three dimensions when compared with unimodal-unitemporal classifiers.

Paralinguistic cues in children’s speech convey the child’s affective state and can serve as important markers for the early detection of autism spectrum disorder (ASD). In this paper, we detect paralinguistic events, such as laughter and fussing/crying, along with toddlers’ speech from the Multi-modal Dyadic Behavior Dataset (MMDB). We use both spectral and prosodic acoustic features selected using a combination of filter and wrapperbased methods. The classification accuracy using a support vector machine with a linear kernel for detecting laughter in children’s speech was 77.87% and that for fussing/crying was 79.37%. A tertiary classification scheme for detecting laughter, fussing/crying, and speech yielded an accuracy of 69.73%. To test for the generalization of the approach for detecting fussing/crying, we used recordings from the Strange Situation protocol, which is used to observe attachment behavior between an infant and a parent. Using a cross-corpus testing set for detecting fussing/crying, we obtained a detection accuracy of 71.6%. These results indicate that the selected acoustic features are capable of discriminating children’s laughter, fussing/crying, and speech and the algorithms generalize well to a dataset consisting of paralinguistic cues of a different age group, infants (12 18 months of age), gathered in a different context.

We introduce a new problem domain for activity recognition: the analysis of children's social and communicative behaviors based on video and audio data. We specifically target interactions between children aged 1-2 years and an adult. Such interactions arise naturally in the diagnosis and treatment of developmental disorders such as autism. We introduce a new publicly-available dataset containing over 160 sessions of a 3-5 minute child-adult interaction. In each session, the adult examiner followed a semi-structured play interaction protocol which was designed to elicit a broad range of social behaviors. We identify the key technical challenges in analyzing these behaviors, and describe methods for decoding the interactions. We present experimental results that demonstrate the potential of the dataset to drive interesting research questions, and show preliminary results for multi-modal activity recognition.

Laughter is an important para-linguistic cue that can be useful in gauging the affective state of the speaker. In this paper, we present an approach to detecting laughter in children’s speech using acoustic features in the spectral and prosodic domains. Feature selection was performed using the information gain-based technique and a speaker-independent validation using a support vector machine (SVM), an accuracy of 94.43% was observed, which was a 12.48% absolute improvement over the baseline result of 81.95%. For us to explore generalization properties, the models of speech and laughter were tested on a completely different database of adult-child interactions known as the Multimodal Dyadic Behavior Dataset (MDBD). The accuracy using the earlier trained models was 70.58%. Even though the children in this database were toddlers (less than three years old), the results suggest that the predictive power of the selected features generalizes well to different forms of children’s laughter.

We present a novel, maximum likelihood framework for automatic spike-sorting based on a joint statistical model of action potential waveform shape and inter-spike interval durations of cortical neuronal firing clusters. We derive an expression for the joint likelihood of the set of observed waveforms and neuronal firing times and hidden neuronal labels. We then use an iterative unsupervised procedure for simultaneous clustering and parameter estimation to find the maximum-likelihood sequence of neuronal labels. We evaluate our method on the WaveClus artificial data-set with 2483 firing events, and obtain a significant improvement in clustering accuracy over the waveform-only EM-GMM baseline in high noise conditions.

This paper presents an extension of Probabilistic Space Maps (PS-MAPS) to adapt clean acoustic models to a noisy environment. In the presence of noise, the relationship between noisy and clean speech features (MFCC's, HDLA, etc.) is either nonlinear or unknown. Given the relationship between features, traditional methods try to linearize it using approximations. These methods cannot be used for systems where the mapping model for clean and noisy features is missing. Given sufficient training data, PS-MAPS provides an excellent framework for extracting and modeling this relationship. The PS-MAP based approach to model adaptation is completely data driven. Experiments were performed on Aurora 2 dataset to evaluate the effectiveness of the algorithm.

In this paper we present a probabilistic algorithm that extracts a mapping between two subspaces by representing each subspace as a collection of states. An arbitrary increase in number of states results in over-fitting the training data without exploring the underlying structure of the map. This paper suggests a method to impose sparsity constraints on the state map by using entropic priors. This probabilistic model is applied to the problem of artificial bandwidth expansion that involves estimating the missing frequency components (3.7 - 8 kHz and 0 - 0.3 kHz) of speech given the narrowband speech signal (0.3 - 3.7 kHz).

This paper presents a novel method for estimating formant frequencies and bandwidths based on an underlying vocal tract model. A novel statistical model for vocal tract cross-sectional areas is developed which allows computation of full likelihood functions. Modifications to the basic particle filter algorithm have also been developed to help combat both diversity depletion and convergence problems. The performance of the method is evaluated against hand labeled formant database (L. Deng et al., 2003).

This paper discusses the challenges of building information retrieval applications that operate on large amounts of voice/audio data. Various problems and issues are presented along with proposed solutions. A set of techniques based on a phonetic keyword spotting approach is presented, together with examples of concrete applications that solve real-life problems.

Comb filters are popular in the speech enhancement field because of their ability to suppress noise in the voiced speech spectrum without degrading the speech quality. Adaptive comb filters (ACF) can suppress significant amounts of noise in voiced speech using the quasiperiodic properties of the voiced speech sounds. However, they rely on accurate voicing and pitch epoch detection, which is challenging at low SNRs and in nonstationary noise conditions. Here, we propose using a non-acoustic auxiliary sensor for detecting the pitch epochs and voicing. Experiment results with the ACF filter showed practically no noise reduction when noisy speech is used for pitch epoch and voicing detection. However, when the auxiliary sensor signal is used, significant improvements are obtained. Therefore, it is shown that the ACF system becomes useful, at least under the considered noise conditions, only if the auxiliary data is available. In addition to the dramatic gain with the auxiliary sensors, performance of the ACF system is further boosted by using it in tandem with a frequency-domain ACF system proposed here. Objective measures, spectrogram analysis, and subjective listening test results clearly show substantial improvement with the tandem system compared to the time-domain ACF system.

Automatic Speech Attribute Transcription (ASAT), an ITR project sponsored under the NSF grant (IIS-04-27113), is a cross-institute effort involving Georgia Institute of Technology, The Ohio State University, University of California at Berkeley, and Rutgers University. This project approaches speech recognition from a more linguistic perspective: unlike traditional ASR systems, humans detect acoustic and auditory cues, weigh and combine them to form theories, and then process these cognitive hypotheses until linguistically and pragmatically consistent speech understanding is achieved. A major goal of the ASAT paradigm is to develop a detection-based approach to automatic speech recognition (ASR) based on attribute detection and knowledge integration. We report on progress of the ASAT project, present a sharable platform for community collaboration, and highlight areas of potential interdisciplinary ASR research. Index Terms: attributes, events, features, detection, speech recognition, speech attribute transcription, utterance verification

Feature fusion is a popular approach for improving the accuracy of speech recognition systems in noisy environments. Although the feature fusion method performs well for high and moderate SNRs, its performance degrades rapidly at low SNRs. Moreover, auxiliary features that are robust to acoustic noise may sometimes be unreliable, because of sensor misplacements etc, which can further degrade the performance. Furthermore, noisy sensor signals may exist not only in the test data but also in the training data. Here, the feature fusion method is combined with a missing data technique to improve noise-robustness at low SNRs. An auxiliary feature is used both for feature fusion and for detecting the unreliable speech frames. Noisy auxiliary features are addressed using a missing data approach both in the decoding and training systems. In the experiments, substantial improvements in the feature fusion method are obtained especially at low SNRs. The proposed missing data-based training strategy is also shown to improve the accuracy significantly.

In this paper, a speech signal estimation framework involving Kalman filters for use as a front-end to the Aurora-2 speech recognition task is presented. Kalman-filter based speech estimation algorithms assume autoregressive (AR) models for the speech and the noise signals. In this paper, the parameters of the AR models are estimated using a expectation-maximization approach. The key to the success of the proposed algorithm is the constraint on the AR model parameters corresponding to the speech signal to belong to a codebook trained on AR parameters obtained from clean speech signals. Aurora-2 noise-robust speech recognition experiments are performed to demonstrate the success of the codebook-constrained Kalman filter in improving speech recognition accuracy in noisy environments. Results with both clean and multi-conditional training are provided to show the improvements in the recognition accuracy compared to the base-line system where no pre-processing is employed

Event detection is a fundamental yet critical component in automatic speech recognition (ASR) systems that attempt to extract knowledge-based features at the front-end level. In this context, it is common practice to design the detectors inside well-known frameworks based on artificial neural network (ANN) or support vector machine (SVM). In the case of ANN, speech scientists often design their detector architecture relying on conventional feed-forward multi-layer perceptron (MLP) with sigmoidal activation function. The aim of this paper is to introduce other ANN architectures inside the context of detection-based ASR. In particular, a bank of feed-forward MLPs using sinusoidal activation functions is set up to address the event detection problem. Experimental results demonstrate the effectiveness of this ANN design for speech attribute detectors

This paper proposes a new application of Gaussian mixture modeling (GMM) for neural cell type classification between globus pallidus externus (GPe) and globus pallidus internus (GPi). Our work is motivated by the results of previous research in which different neural cell types could be identified by their discharge patterns. It is critical for surgeons to distinguish between these two cell types to identify brain nuclei during the procedure known as a pallidotomy, a treatment for Parkinson's disease. Currently, skilled surgeons rely on discharge patterns converted to sound. In this study, performance evaluations are conducted based on a labeled database recorded during previous neural surgeries. The GMM achieves better than 92% correct recognition using 10-second segments, which demonstrates the best performing classifier to date.

This paper presents the framework for an ultra low bit rate speech vocoder. The system is based on a recognition-synthesis paradigm in which a single ergodic hidden Markov model (EHMM) is used to capture the statistical characterizations of speech in a flexible manner capable of limiting the effects of recognition errors. Because predetermined speech units are not used, this system has the advantage of not requiring a transcription for the training data set. By incorporating a mixed excitation scheme based on an improved MELP formulation into the EHMM, additional gains in quality and speaker characterization are achieved at no cost to the bit rate.

Noise mitigation systems for speech coding and recognition have primarily focused on spectral subtraction techniques due to their well understood behavior and computational simplicity. As computation complexity becomes a smaller constraint, understanding the characteristics of different estimation schemes becomes more important. The merits of two algorithms based on direct estimation of the linear prediction spectrum of a speech signal are explored. These algorithms are maximum likelihood (ML) and minimum mean square error estimation (MMSE) of the autoregressive speech spectrum. The MMSE algorithm is able to improve objective quality effectively at low SNRs while also improving the speech recognition accuracy by 20-30% on the Aurora2 test set at the cost of requiring two orders of magnitude more operations than the ML method. Because of these improvements, autoregressive based algorithms should be considered in the future for noise robust speech processing tasks.

This paper presents a novel extension to vector quantization referred to as extended cluster information (ECI). In this method the decoder retains more general statistics about the VQ clusters found during codebook training than the single prototypical point of conventional VQ systems. Typically this information is unnecessary, however if the items being compressed are feature space vectors used as input to a statistical pattern classification system, the extra probabilistic information can be used during the classification as in Bayes predictive classification (BPC) to improve recognition results. To demonstrate ECI-VQ, a simple experiment is described where the Aurora2 distributed speech recognition front end is altered to provide more aggressive mel frequency cepstral coefficient (MFCC) compression. As the bit-rate drops, the corresponding recognition performance suffers. It is then shown that using ECI-VQ as the input to an uncertain observation (UO) speech recognizer, a number of errors due to compression can be corrected with no extra cost in bit-rate.

An automated glottal waveform estimation algorithm is presented that improves on a previous manual glottal extraction technique which produced excellent glottal waveform estimates. The algorithm uses only basic approximations of glottal closure regions and successive iterations to find the best candidate for a glottal waveform estimate within a speech frame. Visual comparisons of the glottal waveform estimates created by the algorithm and those generated from the use of glottal closure information provided by an electroglottograph (EGG) reveal that the algorithm produced virtually identical estimates.

The increasing amount of music being stored in digital formats calls for increasingly more creative methods for music information retrieval. One subset of music retrieval methods relies on storing a melody in a pitch contour representation. Most often, this contour information is generated either from symbolic format (MIDI) or from raw audio after a pitch transcription step. We propose a method of extracting pitch contour information from musical audio without an intermediate transcription step by combining a musically-tuned constant Q transform with crosscorrelation. When tested on a database of 520 monophonic music recordings, our method generates pitch contours from raw audio data with up to 98% accuracy.

Removal of ambient noise from a single-channel audio signal is becoming an increasingly important problem due to the proliferation of portable communication devices. Furthermore, in applications such as wireless telephony and phonetic data mining, it is desired that noise suppression be robust to changing noise conditions and that processing take place in real time or faster. This paper proposes an adaptive noise suppression system that mitigates or eliminates processing artifacts common to Wiener filtering without decreasing speech recognition performance. Results of one implementation of such a structure demonstrate significant improvements in both perceptual quality and speech recognition performance under noisy conditions.

Speech not only conveys the linguistic information, but also characterizes the talker's identity and therefore can be used in personal authentication. While most of the speech information is contained in the acoustic channel, the lip movement during speech production also provides useful information. In this paper we investigate the effectiveness of visual speech features in a speaker veri£cation task. We £rst present the visual front-end of the automatic speechreading system. We then develop a recognition engine to train and recognize sequences of visual parameters. The experimental results based on the XM2VTS database [I] demonstrate that visual information is highly effective in reducing both false acceptance and false rejection rates in speaker veri£cation tasks.

The amount of digitized music stored on personal computers and available on the Internet is growing at a rapid rate. To address the access problem that this creates, we explore adapting HMM-based wordspotting techniques from speech recognition to create a system for melody-based retrieval of songs from a database of digitized music stored in a musically-unstructured format. In this paper, we present the construction of this melody spotter and evaluate its performance when trained under different feature vectors including a musical scale-based subset of the FFT and two Mel-scale based features. The results show the success of this system under the scale-based features when presented with both perfect melody queries and queries perturbed by minor errors.

The most obvious difference between normal and whispered speech is the excitation. However, there are other significant spectral differences between these two modes of speech. In particular, the formant locations are raised in whispers because of increased coupling between the vocal tract and the trachea. In addition, the noise excitation increases the variance of spectral estimates such as linear prediction. In order to reconstruct quality phonated speech from whispers, it is necessary to remove the bias in the formants and reduce the variance of spectrum. Methods for estimating the linear prediction spectrum from whispered speech are proposed. These include modification of the formants in the line spectrum domain and smoothing of the noisy linear prediction spectra.

Mainstream automatic speech recognition has focused almost exclusively on the acoustic signal. The performance of these systems degrades considerably in the real world in the presence of noise. On the other hand, most human listeners, both hearing-impaired and normal hearing, make use of visual information to improve speech perception in acoustically hostile environments. Motivated by humans' ability to lipread, the visual component is considered to yield information that is not always present in the acoustic signal and enables improved accuracy over totally acoustic systems, especially in noisy environments. In this paper, we investigate the usefulness of visual information in speech recognition. We first present a method for automatically locating and extracting visual speech features from a talking person in color video sequences. We then develop a recognition engine to train and recognize sequences of visual parameters for the purpose of speech recognition. We particularly explore the impact of various combinations of visual features on the recognition accuracy. We conclude that the inner lip contour features together with the information about the visibility of the tongue and teeth significantly improve the performance over using outer contour only features in both speaker dependent and speaker independent recognition tasks.

Extending automatic speech recognition (ASR) to the visual modality has been shown to increase recognition accuracy greatly and improve system robustness over purely acoustic systems, especially in acoustically hostile environments. An important aspect of designing such systems is how to incorporate the visual component into the acoustic speech recognizer to achieve optimal performance. We investigate methods of integrating the audio and visual modalities within HMM-based classification models. We examine existing integration schemes and propose the use of a coupled hidden Markov model (CHMM) to exploit audio-visual interaction. Our experimental results demonstrate that the CHMM consistently outperforms other integration models for a large range of acoustic noise levels and suggest that it better captures temporal correlations between the two streams of information.

A method for modifying formant locations and bandwidths directly in the line spectrum domain is developed. This method is based on first-order approximations between different representations of the linear prediction spectrum and does not require finding the roots of the prediction polynomial. In addition, this method is less sensitive to pole interaction problems associated with direct pole modification.

An algorithm to blindly compensate zero-memory nonlinear distortions of speech waveforms is derived and analyzed. This method finds a maximum-likelihood estimate of the distortion without a priori knowledge of the microphone characteristics by using the expectation-maximization algorithm. The autoregressive signal model coefficients are solved jointly with the nonlinearity estimate created by an extended Kalman filter. Also, a new family of nonlinear functions is developed for use with this algorithm, although the method can estimate the shape of any parametric zero-memory nonlinearity. These nonlinear distortions can degrade the speech recognition rates, yet lower the perceptual quality only slightly. The compensation algorithm improves automatic speech recognition of distorted speech for a variety of such nonlinearities.

This paper describes a method for the unsupervised and gender-independent estimation of the average human vocal tract length from the speech waveform, and reports results obtained on Fant's (1960) X-ray vowel data as well as results from experiments performed on multiple sentence utterances of 86 male and 78 female TIMIT speakers, including correlation analyses between the vocal tract length estimates and given body heights. The investigated error criteria that make non-iterative, closed-form estimator solutions possible are all found to achieve good speaker clustering potential for both male and female subgroups.

The sinusoidal transform (ST) provides a sparse representation for speech signals by utilizing several psychoacoustic phenomena. It is well suited to applications in signal enhancement because the signal is represented in a parametric manner that is easy to manipulate. The multi-resolution sinusoidal transform (MRST) has the additional advantage that it is both particularly well suited to typical speech signals and well matched to the human auditory system. The currently reported work discusses the removal of noise from a noisy signal by applying an adaptive Wiener filter to the MRST parameters and then conditioning the parameters to eliminate "musical noise". In informal tests MRST based noise reduction was found to reduce background noise significantly better than traditional Wiener filtering and to virtually eliminate the "musical noise" often associated with Wiener filtering.

A compression scheme for diverse speech and audio signals is proposed. In this scheme, signals are analyzed with a 2-band QMF filter bank followed by the application of a modulated lapped biorthogonal transform (MLBT) to each of the filter bank channels. Subsequent encoding of transform coefficients is performed using Laplacian optimized scalar and vector quantizers, whose rates are determined by an estimated noise threshold, i.e., masking threshold. Listening tests show that the coder achieves a quality at 32 kbits/s that is preferred over the ITU G.722 coder at 64 kbits/s, for speech, music, and more diverse signals consisting of speech in the presence of eventful background sounds. Both the delay of the coder, at 40 ms, and the level of complexity are moderate.

Previous work has shown that connectionist learning systems can simulate important aspects of the categorization of speech sounds by human and animal listeners.Training is on representations of synthetic, exemplar voiced and unvoiced stop consonants passed through a computational model of the auditory periphery.In this work, we use the modern inductive inference technique of support vector machines (SVMs) as the learning system.Visualization of the SVM's weight vector reveals what has been learned about the voiced/unvoiced distinction.

Subjective testing of speaker recognizability is an intricate, time consuming and very expensive process, but using objectively measurable descriptors to augment the subjective speaker recognizability tests could result in increased efficiency and reliability. This paper describes our investigation into the relevancy of a set of objective descriptors to human perception of speaker identity through multidimensional scaling (MDS) of subjective speaker pair similarity judgments. The evaluated objective descriptors can achieve same/different detection error rates as low as 4.13% for male speaker pairs, and 8.17% for female speaker pairs, with only 3 seconds of speech. Five descriptors related to glottal, vocal tract and prosodic features were found to have significant correlations with the perceptual dimensions of the MDS solutions.

This paper discusses the application of hidden Markov models (HMMs) to solve the translational and rotational invariant automatic target recognition (TRIATR) problem associated with SAR imagery. This approach is based on a cascade of these stages: preprocessing, feature extraction and selection, and classification. Preprocessing and feature extraction and selection involve successive applications of extraction operations from measurements of the Radon transform of target chips. The features which are invariant to changes in rotation, position and shifts, although not to changes in scale are optimized through the use of feature selection techniques. The classification stage successively takes as its inputs the multidimensional multiple observation sequences, parameterizes them statistically using continuous density models to capture target and background appearance variability, and thus results in the TRIATR-HMMs. Experimental results have demonstrated that the recognition rate is as high as 99% over both the training set and the testing set.

Although sinusoidal models have been demonstrated to be capable of high-quality musical instrument synthesis, speech modification, and speech synthesis, little exploration of the application of these models to the synthesis of singing voice has been undertaken. We propose a system framework similar to that employed in concatenation-based text-to-speech synthesizers, and describe its extension to the synthesis of singing voice. The power and flexibility of the sinusoidal model used in the waveform synthesis portion of the system enables high-quality, computationally-efficient synthesis and the incorporation of musical qualities such as vibrato and spectral tilt variation. Modeling of segmental phonetic characteristics is achieved by employing a "unit selection" procedure that selects sinusoidally-modeled segments from an inventory of singing voice data collected from a human vocalist. The system, called LYRICOS, is capable of synthesizing very natural-sounding singing that maintains the characteristics and perceived identity of the analyzed vocalist.

With the more widespread use of lower bit rate speech coders, the evaluation of speaker recognizability becomes a major issue to be addressed as well as the evaluation of overall voice quality. Furthermore, subjective quality evaluation of speech coders may produce different results depending on the voice character of the speakers used in the evaluation process. It follows naturally that methods and procedures to characterize speakers perceptually must be devised. We report on an enhanced set of objective descriptors of the speech waveform, assessing the reliability of their measurements as well as their merit in discriminating utterances from different speakers. Of the 45 measures presented, 35 have less than 10% RMS measurement error, and 25 of those have less than 5%.

This paper presents a new high-quality, variable-rate vocoder in which the average bit-rate is parametrically controllable. The new vocoder is intended for use with data-voice simultaneous channel (DVSC) applications, in which the speech data is transmitted simultaneously with video and other types of data. The vocoder presented in this paper achieves state-of-the-art quality at several different bit-rates between 5.5 kbps and 10 kbps. Further, it achieves this performance at acceptable levels of complexity and delay.

Although sinusoidal models have been shown to be useful for time-scale and pitch modification of voiced speech, objectionable artifacts often arise when such models are applied to unvoiced speech. This article presents a sinusoidal model-based speech modification algorithm that preserves the natural character of unvoiced speech sounds after pitch and time-scale modification, eliminating commonly encountered artifacts. This advance is accomplished via a perceptually motivated modulation of the sinusoidal component phases that mitigates artifacts in the reconstructed signal after time-scale and pitch modification.

In this paper, an algorithm for the concatenation of speech signal segments taken from disjoint utterances is presented. The algorithm is based on the analysis-by-synthesis/overlap-add (ABS/OLA) sinusoidal model, which is capable of performing high quality pitch- and time-scale modification of both speech and music signals. With the incorporation of concatenation and smoothing techniques, the model is capable of smoothing the transitions between separately-analyzed speech segments by matching the time- and frequency-domain characteristics of the signals at their boundaries. The application of these techniques in a text-to-speech system based on concatenation of diphone sinusoidal models is also presented.

We describe an optimal discriminant feature based pattern recognition system with two-layer feedforward neural networks. This system can simplify the structure of the covariance matrix to facilitate visual interpretation of cluster graphs and to assess clustering performance. It also can enable one class to be well concentrated and an other maximally spread over the space, while keeping the inter-cluster distance as large as possible. The performance of our 2-D optimal discriminant feature-based neural network system does not degrade compared with a multidimensional feature-based neural network system, while the computational complexity and the storage requirements are significantly improved. This technique has been applied and has shown accurate classification (98% in probability) of digital waveforms obtained from digitized radiation detector outputs. To the best of our knowledge, application of such a technique is novel.

Automatic speaker identification and verification systems can achieve impressive performance levels but the determination of what physical features of the speech waveform contribute to the perceptual characterization of speakers remains a topic of research interest. We present and investigate a variety of features objectively extracted from the speech waveform to characterize perceptual speaker differences. The features belong to three broad classes of glottal, vocal tract and prosodic measurements. They are first evaluated in the context of a speaker distinguishability task by two measures: (i) an invariant criterion of cluster scattering, and, (ii) the error rate achieved by a maximum likelihood same-or-different-speaker classification. Principal components analysis indicates an underlying dimensionality of 6 out of the 20 objective descriptors.

This paper describes a model of speech production based on solving for acoustic wave propagation in the vocal tract using a finite-difference time-domain (FDTD) technique. This FDTD technique was first developed by Yee (1966) and utilizes a discretization scheme in which pressure and velocity components are interleaved in both space and time. The specific implementation of this model of speech production, including discretization of the coupled acoustic wave equations, boundary conditions, stability criteria, values of model constants, and method of excitation, are presented. The accuracy of the model is verified by comparing the FDTD results to the theoretically expected results for a well-known acoustics problem. The FDTD model of speech production has been used in a variety of experiments, and several results, including those that compare the use of several common glottal models as excitation, are presented.

We have addressed the problem of analyzing the digital pulse waveforms of radiation detector outputs. With the availability of extremely high-speed A/D conversion with good resolution, it is now possible to look more deeply at the waveform shapes than is currently done. In our studies, a new technique of unsupervised pattern recognition has been applied which has demonstrated accurate classification (98.33% in probability) of digital pulse waveforms. To the best of our knowledge, application of such a technique is novel. The preliminary results of this system, which show clearly improved measurement conditions, are therefore very promising.

This paper reports the implementation of high-quality synthesis of speech with varying speaking styles using the Klatt (1980) synthesizer. This research is based on previously-reported research that determined that the glottal waveforms of various styles of speech are significantly and identifiably different. Given the parameter tracks that control the synthesis of a normal version of an utterance, those parameters that control known acoustic correlates of speaking style are varied appropriately, relative to normal, to synthesize styled speech. In addition to varying the parameters that control the glottal waveshape, phoneme duration, phoneme intensity, and pitch contour are also varied appropriately. Listening tests that demonstrate that the synthetic speech is perceptibly and appropriately styled, and that the synthetic speech is natural-sounding, were performed, and the results are presented.

This paper presents an architecture for generating a high-speed data pattern with precise edge placement (resolution) by using the matched delay technique. The technique involves passing clock and data signals through arrays of matched delay elements in such a way that the data rate and resolution of the generated data stream are controlled by the difference of these matched delays. This difference can be made much smaller than an absolute gate delay. Since the resolution of conventional designs is determined by these absolute delays, the matched delay technique yields a much finer resolution as well as higher speeds than traditional methods. The matched delay technique lends itself to high-precision and high-speed applications such as fast network interfaces or test pattern generators. This paper also describes a matched delay data generator submitted for fabrication in a MOSIS 1.2 /spl mu/m CMOS technology. Simulations indicate that data signals with on-chip bit rates of 833 Mb/s and resolutions of 100 ps can be generated.

A spoken language system combines speech recognition, natural language processing and human interface technology. It functions by recognizing the person's words, interpreting the sequence of words to obtain a meaning in terms of the application, and providing an appropriate response back to the user. Potential applications of spoken language systems range from simple tasks, such as retrieving information from an existing database (traffic reports, airline schedules), to interactive problem solving tasks involving complex planning and reasoning (travel planning, traffic routing), to support for multilingual interactions. We examine eight key areas in which basic research is needed to produce spoken language systems: (1) robust speech recognition; (2) automatic training and adaptation; (3) spontaneous speech; (4) dialogue models; (5) natural language response generation; (6) speech synthesis and speech generation; (7) multilingual systems; and (8) interactive multimodal systems. In each area, we identify key research challenges, the infrastructure needed to support research, and the expected benefits. We conclude by reviewing the need for multidisciplinary research, for development of shared corpora and related resources, for computational support and far rapid communication among researchers. The successful development of this technology will increase accessibility of computers to a wide range of users, will facilitate multinational communication and trade, and will create new research specialties and jobs in this rapidly expanding area.>

Studies have shown that depending on speaker task and environmental conditions, recognizers are sensitive to noisy stressful environments. The focus of the study is to achieve robust recognition in diverse environmental conditions through the formulation of feature enhancement and stress equalization algorithms under the framework of source generator theory. The generator framework is considered as a means of modeling production variation under stressful speaking conditions. A multi-dimensional stress equalization procedure is formulated that produces recognition features less sensitive to varying factors caused by stress. A feature enhancement algorithm is employed based on iterative techniques previously derived for enhancement of speech in varying background noise environments. Combined stress equalization and feature enhancement reduces average word error rates across 10 noisy stressful conditions by -38.7% (e.g., noisy loud, angry, and Lombard effect stress conditions, etc.). The results suggest that the combination of a flexible source generator framework to address stressed speaking conditions, and a feature enhancement algorithm that adapts based on speech-specific constraints, can be effective in reducing the consequences of stress and noise for robust automatic recognition.>

Investigates a projection-based likelihood measure that significantly improves automatic speech recognition performance in the presence of additive broadband noise. The measure was developed by modifying likelihood scores in continuous Gaussian density hidden Markov models (HMMs), resulting in the weighted projection measure (WPM). Experimental results using the proposed measure are reported for several performance factors: different cepstral-based parameters, normal and multistyle speech, and various noise signals, including white, jittering white, and broadband colored noise. In all cases, significant improvements in speaker-dependent, isolated word recognition were achieved using the WPM instead of the standard Gaussian likelihood measure (weighted Euclidean distance (WED)). As an example, at a SNR of 5 dB, the WPM resulted in improvement in recognition accuracy from 19.4 to 80.6% compared with the standard WED for the DFT mel-cepstral representation.

In this study, a cepstral likelihood measure based on the projection operation is incorporated into a mixture density hidden Markov model (HMM) scheme to improve recognition in the presence of additive noise. The case in which the models are determined only under noise-free conditions is addressed. A background discussion and a derivation of the measure are provided. Recognition experiments are presented showing the usefulness of the proposed measure over the standard Gaussian measure (weighted Euclidean distance) for speaker independent, isolated word recognition in noise. It was found that the proposed mixture weighted projection measure significantly improved performance in several noise types, including white, jittering white, and colored noise. As an example, at an SNR of 10-dB white noise, recognition improved from only 38.4% correct using the Gaussian measure to 83.6% using the developed measure.>

A continuation of the authors' previous research (1990) in the area of statistical analysis and modeling of glottal waveforms of emotionally stressed speech is presented. A simple glottal model is defined which, by varying four parameters, can be used to model the glottal excitation of many styles of speech. An algorithm which automatically decouples these model glottal waveforms from the vocal tract information in a given speech segment is presented. Several applications of decoupling the model glottal waveforms are discussed. In particular, a method of automatic formant tracking of stressed speech which is superior to traditional formant tracking procedures is described. Also, a method of modifying the style of speech by removing some of the effects of stress on the speech waveform is presented.>

The use of a projection-based cepstral measure for speech recognition in noise is investigated. Interpretations of the measure's spectral and perceptual significance are given, along with its application to cepstral, melcepstral, and delta parameters. It is shown how the projection measure can be incorporated into a continuous density hidden Markov model system in the form of a weighted measure. Both the case where these densities are unimodal Gaussians and mixtures of Gaussians are addressed. In recognition experiments, the weighted projection measure significantly outperformed the standard weighted Euclidean or Gaussian distance measure.>

It is shown that the structure of a large ergodic hidden Markov model (HMM) can be decomposed into a set of substructures representing the English phonemes. The large HMM, which is trained using a standard forward-backward algorithm, is shown to be organized in a way that reflects the phonetic nature of speech. It is shown that the states of the HMM can be classified in terms of a set of broad phonetic classes and that the spectra associated with the states are related to each state's use in the phonetic models. The phonetic models are shown to have internal structures reflecting the acoustic nature of the individual phonemes. The large HMMs used in this study are trained using the continuous speech multi-speaker TIMIT database employing a continuous observation density training algorithm. On a subset of the database, with 80 male speakers used for training and a separate set of 24 speakers reserved for testing, the phonetic recognition system achieved a 52% recognition rate with 14% insertions.>

The problem of compensation for a particular form of sensorineural hearing impairment known as recruitment of loudness is explored. The technique used is based on a digital sinusoidal analysis/synthesis model of speech. Time-varying, frequency-dependent nonlinearities are applied between the analysis and synthesis modules to perform the compensation. This system differs from traditional multichannel filtering techniques by incorporating a model of the interaction of sinusoidal maskers in both normal-hearing and hearing-impaired individuals. Such phenomena as spread of masking, elevated thresholds, reduced dynamic range, and other psychoacoustic properties are taken into account in the compensation. A system of simultaneous equations is solved to determine the proper gains for each sinusoid. The net effect is a combination between multichannel amplitude compression and automatic gain control since the compressive gains calculated separately for each frame of speech automatically adjust to the level of the speech components in that frame. A simplified version of the compensation algorithm has been implemented in near real-time using a Texas Instruments TMS320C30-based microprocessor board within a personal computer.>

The directed divergence, which is a measure based on the discrimination information between two signal classes, is investigated. A simplified expression for computing the directed divergence is derived for comparing two Gaussian autoregressive processes such as those found in speech. This expression alleviates both the computational cost (reduced by two thirds) and the numerical problems encountered in computing the directed divergence. In addition, the simplified expression is compared with the Itakura-Saito distance (which asymptotically approaches the directed divergence). Although the expressions for these two distances closely resemble each other, only moderate correlations between the two were found on a set of actual speech data.>

Results from a study of glottal waveforms derived from 11 different styles of stressed speech are described. The general goal of this research is to investigate the differences in the glottal excitation across specific speaking styles. Using a glottal inverse filtering method tailored specifically for the speech under study, an analysis of glottal waveforms from voiced speech spoken under various conditions is performed. The extracted waveforms are parameterized using six descriptors, and statistical analysis of the resulting database allows very good characterization of each style with respect to these parameters. Specifically, each style is shown to have a unique profile which will allow high performance in a stress style identification task.>

The problem of speech recognition in noisy, stressful environments is addressed. The main contribution is the achievement of robust recognition in diverse environmental conditions through the formulation of a series of speech-enhancement and stress-compensation preprocessing algorithms. These preprocessors produce speech or recognition features less sensitive to varying factors caused by stress and noise. Results from four recognition scenarios based on such preprocessing are reported. Neutral, stressful, noisy neutral, and noisy stressful speech styles are considered. Noise reduction is based on constrained iterative speech enhancement. Stress compensation algorithms are based on formant location, bandwidth, and intensity.>

Continuous speech from one male American speaker is described by a single, large, hidden Markov model. The problems of model parameter estimation, speech-state decoding, and speech-observation decoding are presented under a unified conceptual framework built around a maximum-likelihood approach. The model has 64 states and 1024 observations, and is assumed to be fully connected. This model possesses not only speech analysis but also speech synthesis capabilities. Very-low-bit-rate speech coding is made possible by the fact that very little information is needed to encode speech state sequences. Intelligible speech was produced at 1.68 bits/frame, whereas a 2 bits/frame pitched-excited LPC vector quantizer did not produce intelligible speech.>

A set of iterative speech enhancement techniques using spectral constraints is extended and evaluated. The approaches apply inter- and intraframe spectral constraints to ensure optimum speech quality across all classes of speech. Constraints are applied on the basis of the presence of perceptually important speech characteristics found during the enhancement procedure. Results show improvement over past techniques for additive white noise distortions. Three points are addressed in the present study. First, a convenient and consistent terminating point for the iterative technique is presented which was previously unavailable. Second, the techniques have been generalized to allow for slowly varying, colored noise. Finally, a comparative evaluation has been performed to determine their usefulness as preprocessors for recognition in extremely noisy environments in the vicinity of 0 dB SNR.>

Traditional hidden Markov model speech recognition is generally based on a set of parameters (often LPC related) which are extracted at discrete intervals. Such an analysis necessitates use of a discrete-trial hidden Markov model in which the underlying states can only change at intervals related to the frame rate of the analysis. The exact locations of the analysis windows used can influence the front-end outputs and as a result can cause confusion between words differing in short-duration consonants. In the current study, an alternate method which does not require segmentation is proposed, and a simple version is implemented. The discrete trial hidden Markov model algorithms are adapted to this framework leading to significantly improved recognition performance.

A new and improved iterative speech enhancement technique based on spectral constraints is presented in this paper. The iterative technique, originally formulated by Lim and Oppenheim, attempts to solve for the maximum likelihood estimate of a speech waveform in additive white noise. The new approach applies inter- and intra-frame spectral constraints to ensure convergence to reasonable values and hence improve speech quality. An extremely efficient technique for applying these constraints is in the use of line spectral pair (LSP) coefficients. The inter-frame constraints ensures more speech-like formant trajectories than those found in the unconstrained approach. Results from speech degraded by additive white Gaussian noise show noticeable quality improvement.

A new type of very low bit rate speech coder based on a global Discrete Hidden Markov Model (DHMM) of continuous speech for a single speaker is presented here. Several important issues of the training, coding, and decoding procedures are discussed for a 64-state, 1024-observation model. Such a framework is useful in reducing the redundancy in a 10-bit classical Vector Quantizer (VQ), and could lead to a DHMM coder with a bit rate comparable to that of a Segment Vocoder (SV) or a Matrix Quantizer (MQ). This is achieved not only by modelling the long term non-stationarity and the inter-frame time dependencies of the speech, but also by efficiently representing a different kind of information such as vocal tract structure and linguistic patterns.

In multipulse LPC speech coding, there are a number of important issues which need to be addressed. The purpose of this paper is to report some of our work related to two of these issues. The first pertains to how one measures the quality of the synthetic speech for a given set of multipulse parameters. In this paper, we present a general set of frequency domain weighting functions which may incorporate various spectral characterizations of the speech. The second issue relates to how one determines the optimal set of pulse positions for a given set of of LPC filter coefficients. Current methods involve a sequential procedure, which is suboptimal compared with simultaneous positioning of pulses throughout an entire analysis frame. We therefore discuss an algorithm for simultaneous pulse positioning based on Pisarenko's harmonic decomposition.

Multiple pulse excited linear predictive coding (MPLPC) has recently received a great deal of attention in the literature as an attractive means of speech coding at data rates below 10 Kbits/second. The existing approaches to MPLPC analysis arrive at the parameters for an all-pole model by minimizing the mean squared modeling error before attempting to find a set of pulses to excite the model. The strategy proposed here selects the all-pole parameters to concentrate the model excitation in a finite number of locations. The goal is then to produce a maximally pulse-like residual as a result of the all-pole parameter estimation.

Two methods of encoding speech for tactile displays were compared in experiments which measured the discriminability of tactile representations of short speech segments. One display represented the short-term speech spectrum in time-swept mode and used vibration amplitude to encode spectral amplitude. The other represented the vocal-tract shape and length estimates as derived from LPC analysis of the speech waveform and used the number of active vibrators to encode the logarithm of cross sectional area. Both displays were applied to the thigh via a 12 × 12 matrix of vibrators. The vibrators were driven at 250 Hz during voiced segments and by random noise during unvoiced segments. Overall results show a slight superiority for the spectral display in vowel discrimination. The results of detailed analyses suggest that many of the detailed characteristics of the tactile patterns were poorly discriminated. Better models of tactile perception must be formulated before speech data compression techniques can be used effectively.