The real-world performance of slot filling and intent detection task generally degrades due to transcription errors generated by speech recognition engine. The insertion, deletion, and mis-recognition errors from speech recognizer's front-end cause the mis-interpretation and mis-alignment of the language understanding models. In this work, we propose a new jointly trained model of intent detection and slot filling with consideration of speech recognition errors. The attention-based encoder-decoder recurrent neural network first decodes the intent information from an utterance, and then corrects errors in the word sequence, if any, before extracting the slot information. The triple joint training framework maximizes the probability of a correct understanding given an input utterance. Our experimental results showed that the proposed model obtained 2.87% absolute gain over the joint model without ASR error correction for slot filling and 0.73% absolute error rate reduction for intent detection.

This paper focuses on stochastic modeling and energy dis-aggregation based on conditional random fields (CRFs) using real-world energy consumption data. Firstly, energy-consumption activities modeling aims at understanding and identifying energy-consumption activities using behavior models based on observed energy signals. Our ultimate goal is to suggest ways to modify human behavior activities in order to conserve energy by optimizing the use of energy. Preliminary analysis of energy consumption data clearly shows the potential effectiveness of activity behavior changes on the changing energy consumption behavior. Secondly energy disaggregation aims at breaking up the total energy signal into its component appliances. This is very useful since it can provide home owners with feedback about the way they use electrical energy, and can also motivate users to conserve significant amounts of energy. In the current study, we focus on activity/event disaggregation using the total energy-consumption signal.

This paper studies various aspects of child vocalization as captured in a newly established parallel corpus of sixteen 18–31 months old US and Shanghainese toddlers. The recordings were acquired in 16-hour sessions during an ‘ordinary’ day in the child’s natural environment and manually labeled. The vocalization characteristics are studied by means of phonotactic and prosodic analysis with emphasis on automatic processing. In the phonotactic domain, a Gaussian mixture model (GMM) tokenizer, a bank of phone recognizers, and formant tracking are used to analyze the movements in the acoustic-phonetic space. In the prosodic domain, pitch patterns, duration, and rhythm are analyzed. Besides strong individual-specific characteristics of the subjects in some of the domains considered, the two language groups show differences in the occupation of the F1–F2 formant space, choice of pitch pattern durations, and consistency in producing complex phonetic patterns. Index Terms: children vocalization, speech acquisition, phonotactic modeling, pitch patterns, rhythmicity parameters

In this paper, we propose a stochastic driver-behavior modeling framework which takes into account both individual and general driving characteristics as one aggregate model. Patterns of individual driving styles are modeled using Dirichlet process mixture model, a nonparametric Bayesian approach which automatically selects the optimal number of model components to fit sparse observations of each particular driver's behavior. In addition, general or background driving patterns are also captured with a Gaussian mixture model using a reasonably large amount of development observed data from several drivers. By combining both probability distributions, the aggregate driver-dependent model can better emphasize driving characteristics of each particular driver, while also backing off to exploit general driving behavior in cases of unmatched parameter spaces from individual training observations. The proposed driver-behavior model was employed to anticipate pedal-operation behavior during car-following maneuvers involving several drivers on the road. The experimental results showed advantages of the combined model over the adapted model previously proposed.

This paper describes the development of a self-coaching system to improve driving behavior by allowing drivers to review a record of their own driving activity. By employing stochastic driver-behavior modeling, the proposed system is able to detect a wide range of potentially hazardous situations, which conventional event data recorders are not able to capture, including those involving latent risks, of which drivers themselves are unaware. By utilizing these automatically detected hazardous situations, our web-based system offers a user-friendly interface for drivers to navigate and review each hazardous situation in detail (e.g., driving scenes are categorized into different types of hazardous situations and are displayed with corresponding multimodal driving signals). Furthermore, the system provides feedback on each risky driving behavior and suggests how users can safely respond to such situations. The proposed system establishes a cooperative relationship between the driver, the vehicle, and the driving environment, leading to the development of the next generation of safety systems and paving the way for an alternative form of driving education that could further reduce the number of fatal accidents. The system's potential benefits are demonstrated through preliminary extensive evaluation of an on-road experiment, showing that safe-driving behavior can be significantly improved when drivers use the proposed system.

In this paper, we present our recently developed stochastic driver-behavior model based on Gaussian mixture model (GMM) framework. The proposed driver-behavior modeling is employed to anticipate car-following behavior in terms of pedal control operations in response to the observable driving signals, such as the own vehicle velocity and the following distance to the leading vehicle. In addition, the proposed driver modeling allows adaptation scheme to enhance the model capability to better represent particular driving characteristics of interest (i.e., individual driving style) from the observed driving data themselves. Validation and comparison of the proposed driver-behavior models on realistic car-following data of several drivers showed the promising results. Furthermore, the adapted driver models showed consistent improvement over the unadapted driver models in both short-term and long-term predictions.

In this paper, we propose a new framework for discriminating the initial maneuver of a lane-crossing event from a driver correction event, which is the primary reason for false warnings of lane departure prediction systems (LDPSs). The proposed algorithm validates the beginning episode of the trajectory of driving signals, i.e., whether it will cause a lane-crossing event, by employing driver behavior models of the directional sequence of piecewise lateral slopes (DSPLS) representing lane-crossing and driver correction events. The framework utilizes only common driving signals and allows the adaptation scheme of driver behavior models to better represent individual driving characteristics. The experimental evaluation shows that the proposed DSPLS framework has a detection error with as low as a 17% equal error rate. Furthermore, the proposed algorithm reduces the false-warning rate of the original lane departure prediction system with less tradeoff for the correct prediction.

In this paper, we propose a new framework to discriminate the initial maneuver of lane-crossing event from driver-correction event, which is the primary reason for false warnings of the lane departure prediction systems. The proposed algorithm validates the beginning episode of the trajectory of driving signals whether it will cause a lane crossing event or not, by employing driver behavior models of directional sequence of piecewise lateral slopes (DSPLS) representing lane-crossing and driver-correction events. The framework utilizes only common driving signals, and allows adaptation scheme of driver behavior models to better represent individual driving characteristics. The experimental evaluation shows that the proposed DSPLS has detection error as low as 17% equal error rate. Furthermore, the proposed algorithm reduces the false alarm rate of the original lane departure prediction system from 38.8% to 6.1% with less trade-off for the prediction accuracy.

This paper describes our first step for advances in humanmachine interactive systems for in-vehicle environments of the UTDrive project. UTDrive is part of an on-going international collaboration to collect and research rich multi-modal data recorded for modeling behavior while the driver is interacting with speech-activated systems or performing other secondary tasks. A simultaneous second goal is to better understand speech characteristics of the driver undergoing additional cognitive load since dialog systems are generally not formulated for high task-stress environment (e.g., driving a vehicle). The corpus consists of audio, video, brake/gas pedal pressure, forward distance, GPS information, and CAN-Bus information. The resulting corpus, analysis, and modeling will contribute to more effective speech systems which are able to sense driver cognitive distraction/stress and adapt itself to the driver’s cognitive capacity and driving situations for improved safety while driving. Index Terms: in-vehicle speech system, driver distraction, multi-modal resource,driver behavior modeling, safety driving

In this paper, the problem of identifying in-set versus out-of-set speakers for limited training/test data durations is addressed. The recognition objective is to form a decision regarding an input speaker as being a legitimate member of a set of enrolled speakers or outside speakers. The general goal is to perform rapid speaker model construction from limited enrollment and test size resources for in-set testing for input audio streams. In-set detection can help ensure security and proper access to private information, as well as detecting and tracking input speakers. Areas of applications of these concepts include rapid speaker tagging and tracking for information retrieval, communication networks, personal device assistants, and location access. We propose an integrated system with emphasis on short-enrollment data (about 5 s of speech for each enrolled speaker) and test data (2-8 s) within a text-independent mode. We present a simple and yet powerful decision rule to accept or reject speakers using a discriminative vector in the decision score space, together with statistical hypothesis testing based on the conventional likelihood ratio test. Discriminative training is introduced to further improve system performance for both decision techniques, by employing minimum classification error and minimum verification error frameworks. Experiments are performed using three separate corpora. Using the YOHO speaker recognition database, the alternative decision rule achieves measurable improvement over the likelihood ratio test, and discriminative training consistently enhances overall system performance with relative improvements ranging from 11.26%-28.68%. A further extended evaluation using the TIMIT (CORPUS1) and actual noisy aircraft communications data (CORPUS2) shows measurable improvement over the traditional MAP based scheme using the likelihood ratio test (MAP-LRT), with average EERs of 9%-23% for TIMIT and 13%-32% for noisy aircraft communications. The results confirm that an effective in-set/out-of-set speaker recognition system can be formulated using discriminative training for rapid tagging of input speakers from limited training and test data sizes

This study addresses novel advances in English dialect/accent classification. A word-based modeling technique is proposed that is shown to outperform a large vocabulary continuous speech recognition (LVCSR)-based system with significantly less computational costs. The new algorithm, which is named Word-based Dialect Classification (WDC), converts the text-independent decision problem into a text-dependent decision problem and produces multiple combination decisions at the word level rather than making a single decision at the utterance level. The basic WDC algorithm also provides options for further modeling and decision strategy improvement. Two sets of classifiers are employed for WDC: a word classifier DW(k)and an utterance classifier Du. DW(k)is boosted via the AdaBoost algorithm directly in the probability space instead of the traditional feature space. Duis boosted via the dialect dependency information of the words. For a small training corpus, it is difficult to obtain a robust statistical model for each word and each dialect. Therefore, a context adapted training (CAT) algorithm is formulated, which adapts the universal phoneme Gaussian mixture models (GMMs) to dialect-dependent word hidden Markov models (HMMs) via linear regression. Three separate dialect corpora are used in the evaluations that include the Wall Street Journal (American and British English), NATO N4 (British, Canadian, Dutch, and German accent English), and IViE (eight British dialects). Significant improvement in dialect classification is achieved for all corpora tested

It is suggested that algorithms capable of estimating and characterizing accent knowledge would provide valuable information in the development of more effective speech systems such as speech recognition, speaker identification, audio stream tagging in spoken document retrieval, channel monitoring, or voice conversion. Accent knowledge could be used for selection of alternative pronunciations in a lexicon, engage adaptation for acoustic modeling, or provide information for biasing a language model in large vocabulary speech recognition. In this paper, we propose a text-independent automatic accent classification system using phone-based models. Algorithm formulation begins with a series of experiments focused on capturing the spectral evolution information as potential accent sensitive cues. Alternative subspace representations using principal component analysis and linear discriminant analysis with projected trajectories are considered. Finally, an experimental study is performed to compare the spectral trajectory model framework to a traditional hidden Markov model recognition framework using an accent sensitive word corpus. System evaluation is performed using a corpus representing five English speaker groups with native American English, and English spoken with Mandarin Chinese, French, Thai, and Turkish accents for both male and female speakers.

Advances in formulating spoken document retrieval for a new National Gallery of the Spoken Word (NGSW) are addressed. NGSW is the first large-scale repository of its kind, consisting of speeches, news broadcasts, and recordings from the 20th century. After presenting an overview of the audio stream content of the NGSW, with sample audio files from U.S. Presidents from 1893 to the present, an overall system diagram is proposed with a discussion of critical tasks associated with effective audio information retrieval. These include advanced audio segmentation, speech recognition model adaptation for acoustic background noise and speaker variability, and information retrieval using natural language processing for text query requests that include document and query expansion. For segmentation, a new evaluation criterion entitled fused error score (FES) is proposed, followed by application of the CompSeg segmentation scheme on DARPA Hub4 Broadcast News (30.5% relative improvement in FES) and NGSW data. Transcript generation is demonstrated for a six-decade portion of the NGSW corpus. Novel model adaptation using structure maximum likelihood eigenspace mapping shows a relative 21.7% improvement. Issues regarding copyright assessment and metadata construction are also addressed for the purposes of a sustainable audio collection of this magnitude. Advanced parameter-embedded watermarking is proposed with evaluations showing robustness to correlated noise attacks. Our experimental online system entitled "SpeechFind" is presented, which allows for audio retrieval from a portion of the NGSW corpus. Finally, a number of research challenges such as language modeling and lexicon for changing time periods, speaker trait and identification tracking, as well as new directions, are discussed in order to address the overall task of robust phrase searching in unrestricted audio corpora.

We study the problem of identifying in-set and out-of-set speakers. The goal is to identify whether an unknown input speaker belongs to either a group of in-set speakers or an unseen out-of-set group. A state-of-the-art GMM classifier, with universal background model (UBM) and standard likelihood ratio test, is used as our baseline system. We propose an alternative hypothesis testing method that employs neighborhood information with respect to each in-set speaker model in the model space based on the Kullback-Leibier divergence. The Bayes factor is used in the verification stage (accept/reject hypothesis). We evaluate the proposed procedure on a clean CORPUS 1 set, and a noisy CORPUS 2 set which contains session-to-session variability. Experiments show an improvement in equal error rate for the system even when in-set speaker models are acoustically close in the model space, and as the size of the in-set speaker group increases.

In this paper, we propose an approach to address the problem of text-independent open-set speaker identification. The in-set speakers are clustered into smaller subsets without merging speaker models. The Anti-Speaker or Background Model is then adapted for each subset which minimizes the identification errors of the pseudo impostors during the training stage. Score normalization is applied to align all the in-set speaker score distributions to share a single scale. Finally, confidence measure processing is used to identify in-set versus out-of-set speakers. Experiments with TIMIT and the CU-Accent corpora show an improvement in Equal Error Rate on the average of 20.28 and 8.35 over the baseline performance respectively. Finally, a probe experiment is also included that considers prosody for in-set speaker detection.

This paper presents recent results using statistics generated by a MMI-supervised vector quantizer as a measure of audio similarity. Such a measure has proved successful for talker identification, and the extension from speech to general audio, such as music, is straightforward. A classifier that distinguishes speech from music and non-vocal sounds is presented, as well as experimental results showing how perfect classification accuracy may be achieved on a small corpus using substantially less than two seconds per test audio file. The techniques a presented here may be extended to other applications and domains, such as audio retrieval-by-similarity, musical genre classification, and automatic segmentation of continuous audio. Introduction This paper presents a method of rapidly determining the characteristics of audio samples, using a supervised tree-based vector quantizer trained to maximise mutual information (MMI). Unlike other approaches based on perceptual criteria (Pfeiffer, F...

In this paper, we investigate the integration of two processing methods to improve speech quality for invehicle speech systems: multi-sensor beamforming and constrained iterative (Auto-LSP) speech enhancement. The intent is to establish an intelligent microphone array processing scheme in high noise environments by considering the effectiveness of a multi-sensor beamformer method and the Auto-LSP single channel speech enhancement method. The goal therefore is to design a system where the strengths of one method help compensate any potential weaknesses of the other. The noise cancellation method is an acoustic beamformer designed and constructed using a linear microphone array. The speech enhancement method is the constrained iterative Auto-LSP approach, previously considered for single channel enhancement. After establishing the combined processing scheme, evaluations are performed using speech and acoustic noise data collected in vehicles. Noise suppression levels by the beamformer is established for different road noise conditions. Quality improvement from the enhancement scheme is assessed using objective speech quality measures over a test speech corpus using TIMIT data. The results show that while beamforming alone can suppress background noise levels, the combination of beamforming and constrained enhancement can provide as much as a 63% improvement in objective quality, suggesting a potential single comprehensive solution for in-vehicle speech systems.