This paper describes a data collection methodology and emotion annotation of dyadic interactions between a human, a Pepper robot, a Google Home smart-speaker, or another human. The collected 16 hours of audio recordings were used to analyze the propensity to change someone’s opinions about ecological behavior regarding the type of conversational agent, the kind of nudges, and the speaker’s emotional state. We describe the statistics of data collection and annotation. We also report the first results, which showed that humans change their opinions on more questions with a human than with a device, even against mainstream ideas. We observe a correlation between a certain emotional state and the interlocutor and a human’s propensity to be influenced. We also reported the results of the studies that investigated the effect of human likeness on speech using our data.

The emotion detection technology to enhance human decision-making is an important research issue for real-world applications, but real-life emotion datasets are relatively rare and small. The experiments conducted in this paper use the CEMO, which was collected in a French emergency call center. Two pre-trained models based on speech and text were fine-tuned for speech emotion recognition. Using pre-trained Transformer encoders mitigates our data’s limited and sparse nature. This paper explores the different fusion strategies of these modality-specific models. In particular, fusions with and without cross-attention mechanisms were tested to gather the most relevant information from both the speech and text encoders. We show that multimodal fusion brings an absolute gain of 4-9% with respect to either single modality and that the Symmetric multi-headed cross-attention mechanism performed better than late classical fusion approaches. Our experiments also suggest that for the real-life CEMO corpus, the audio component encodes more emotive information than the textual one.

Affective computing develops systems, which recognize or influence aspects of human life related to emotion, including feelings and attitudes. Significant potential for both good and harm makes it ethically sensitive, and trying to strike sound balances is challenging. Common images of the issues invite oversimplification and offer a limited understanding of the moral consequences and ethical tensions. Considering the state-of-the-art shows how pervasive and complex they are. In many areas, the discipline can potentially bring ethically significant benefits and hence has a duty to try. They include making interactions with machines more effective and less stressful, diagnostic and therapeutic roles in emotion-related disorders, intelligent tutoring, and reducing isolation. However, the limits of recognition technology mean that actions are likely to be based on impoverished representations of people’s affective state, particularly with certain groups; systems are liable to arouse feelings that are positive, but not well grounded in reality, affectively engaging systems can become addictive and manipulative, and they confer dangerous power on those who control the technology. We offer an overview of those and other particular ethical issues, positive and negative, which arise from the current state of affective computing. It aims to reflect the complexities inherent in both the technology and current ethical discussions. Establishing appropriate responses is a challenge for society as a whole, not only the affective computing community.

In this paper, we present the methodology of corpus design that will be used to study the comparison of influence between linguistic nudges with positive or negative influences and three conversational agents: robot, smart speaker, and human. We recruited forty-nine participants to form six groups. The conversational agents first asked the participants about their willingness to adopt five ecological habits and invest time and money in ecological problems. The participants were then asked the same questions but preceded by one linguistic nudge with positive or negative influence. The comparison of standard deviation and mean metrics of differences between these two notes (before the nudge and after) showed that participants were mainly affected by nudges with positive influence, even though several nudges with negative influence decreased the average note. In addition, participants from all groups were willing to spend more money than time on ecological problems. In general, our experiment’s early results suggest that a machine agent can influence participants to the same degree as a human agent. A better understanding of the power of influence of different conversational machines and the potential of influence of nudges of different polarities will lead to the development of ethical norms of human-computer interactions.

Zero-shot speech emotion recognition (SER) endows machines with the ability of sensing unseen-emotional states in speech, compared with conventional SER endeavors on supervised cases. On addressing the zero-shot SER task, auditory affective descriptors (AADs) are typically employed to transfer affective knowledge from seen- to unseen-emotional states. However, it remains unknown which types of AADs can well describe emotional states in speech during the transfer. In this regard, we define and research on three types of AADs, namely, per-emotion semantic-embedding, per-emotion manually annotated, and per-sample manually annotated AADs, through zero-shot emotion recognition in speech. This leads to a systematic design including prototype- and annotation-based zero-shot SER modules, relying on the input from per-emotion and per-sample AADs, respectively. We then perform extensive experimental comparisons between human and machines’ AADs on the French emotional speech corpus CINEMO for positive-negative (PN) and within-negative (WN) tasks. The experimental results indicate that semantic-embedding prototypes from pretrained models can outperform manually annotated emotional dimensions in zero-shot SER. The results further demonstrate that it is possible for machines to understand and describe affective information in speech better than human beings, with the help of sufficient pretrained models.

Recognizing a speaker’s emotion from their speech can be a key element in emergency call centers. End-to-end deep learning systems for speech emotion recognition now achieve equivalent or even better results than conventional machine learning approaches. In this paper, in order to validate the performance of our neural network architecture for emotion recognition from speech, we first trained and tested it on the widely used corpus accessible by the community, IEMOCAP. We then used the same architecture with the real life corpus, CEMO, comprised of 440 dialogs (2h16m) from 485 speakers. The most frequent emotions expressed by callers in these real-life emergency dialogues are fear, anger and positive emotions such as relief. In the IEMOCAP general topic conversations, the most frequent emotions are sadness, anger and happiness. Using the same end-to-end deep learning architecture, an Unweighted Accuracy Recall (UA) of 63% is obtained on IEMOCAP and a UA of 45.6% on CEMO, each with 4 classes. Using only 2 classes (Anger, Neutral), the results for CEMO are 76.9% UA compared to 81.1% UA for IEMOCAP. We expect that these encouraging results with CEMO can be improved by combining the audio channel with the linguistic channel. Real-life emotions are clearly more complex than acted ones, mainly due to the large diversity of emotional expressions of speakers.

This paper addresses the problem of evaluating engagement of the human participant by combining verbal and nonverbal behaviour along with contextual information. This study will be carried out through four different corpora. Four different systems designed to explore essential and complementary aspects of the JOKER system in terms of paralinguistic/linguistic inputs were used for the data collection. An annotation scheme dedicated to the labeling of verbal and non-verbal behavior have been designed. From our experiment, engagement in HRI should be multifaceted.

Work on voice sciences over recent decades has led to a proliferation of acoustic parameters that are used quite selectively and are not always extracted in a similar fashion. With many independent teams working in different research areas, shared standards become an essential safeguard to ensure compliance with state-of-the-art methods allowing appropriate comparison of results across studies and potential integration and combination of extraction and recognition systems. In this paper we propose a basic standard acoustic parameter set for various areas of automatic voice analysis, such as paralinguistic or clinical speech analysis. In contrast to a large brute-force parameter set, we present a minimalistic set of voice parameters here. These were selected based on a) their potential to index affective physiological changes in voice production, b) their proven value in former studies as well as their automatic extractability, and c) their theoretical significance. The set is intended to provide a common baseline for evaluation of future research and eliminate differences caused by varying parameter sets or even different implementations of the same parameters. Our implementation is publicly available with the openSMILE toolkit. Comparative evaluations of the proposed feature set and large baseline feature sets of INTERSPEECH challenges show a high performance of the proposed set in relation to its size.

The search of a small acoustic feature set for emotion recognition faces three main challenges. Such a feature set must be robust to large diversity of contexts in real-life applications; model parameters must also be optimized for reduced subsets; finally, the result of feature selection must be evaluated in cross-corpus condition. The goal of the present study is to select a consensual set of acoustic features for valence recognition using classification and non-classification based feature ranking and cross-corpus experiments, and to optimize emotional models simultaneously. Five realistic corpora are used in this study: three of them were collected in the framework of the French project on robotics ROMEO, one is a game corpus (JEMO) and one is the well-known AIBO corpus. Combinations of features found with non-classification based methods (information gain and Gaussian mixture models with Bhattacharyya distance) through multi-corpora experiments are tested under cross-corpus conditions, simultaneously with SVM parameters optimization. Reducing the number of features goes in pair with optimizing model parameters. Experiments carried on randomly selected features from two acoustic feature sets show that a feature space reduction is needed to avoid over-fitting. Since a Grid search tends to find non-standard values with small feature sets, the authors propose a multi-corpus optimization method based on different corpora and acoustic feature subsets which ensures more stability. The results show that acoustic families selected with both feature ranking methods are not relevant in cross-corpus experiments. Promising results have been obtained with a small set of 24 voiced cepstral coefficients while this family was ranked in the 2nd and 5th positions with both ranking methods. The proposed optimization method is more robust than the usual Grid search for cross-corpus experiments with small feature sets.

Thanks to a remarkably great ability to show amusement and engagement, laughter is one of the most important social markers in human interactions. Laughing together can actually help to set up a positive atmosphere and favors the creation of new relationships. This paper presents a data collection of social interaction dialogs involving humor between a human participant and a robot. In this work, interaction scenarios have been designed in order to study social markers such as laughter. They have been implemented within two automatic systems developed in the Joker project: a social dialog system using paralinguistic cues and a task-based dialog system using linguistic content. One of the major contributions of this work is to provide a context to study human laughter produced during a human-robot interaction. The collected data will be used to build a generic intelligent user interface which provides a multimodal dialog system with social communication skills including humor and other informal socially oriented behaviors. This system will emphasize the fusion of verbal and non-verbal channels for emotional and social behavior perception, interaction and generation capabilities.

Understanding human behavioral and emotional cues occurring in interaction has become a major research interest due to the emergence of numerous applications such as in social robotics. While there is agreement across different theories that some behavioral signals are involved in communicating information, there is a lack of consensus regarding their specificity, their universality, and whether they convey emotions, affective, cognitive, mental states or all of those. Our goal in this study is to explore the relationship between behavioral and emotional cues extracted from speech (e.g., laughter, speech duration, negative emotions) with different communicative information about the human participant. This study is based on a corpus of audio/video data of humorous interactions between the nao{} robot and 37 human participants. Participants filled three questionnaires about their personality, sense of humor and mental states regarding the interaction. This work reveals the existence of many links between behavioral and emotional cues and the mental states reported by human participants through self-report questionnaires. However, we have not found a clear connection between reported mental states and participants profiles.

Databases of spontaneous multimodal expressions of affective states occurring during a task are few. This paper presents a protocol for eliciting stress in a public speaking task. Behaviors of 19 participants were recorded via a multimodal setup including speech, video of the facial expressions and body movements, balance via a force plate, and physiological measures. Questionnaires were used to assert emotional states, personality profiles and relevant coping behaviors to study how participants cope with stressful situations. Several subjective and objective performances were also evaluated. Results show a significant impact of the overall task and conditions on the participants' emotional activation. The possible future use of this new multimodal emotional corpus is described.

Speech production modifications are one of the many indications of stress in humans. A job interview simulation task permitted the collection of a multimodal corpus, including physiological data. Physiological cues of stress are reliable on long periods, and require invasive sensors. Human voice variations have been proved to be a non-invasive stress cue. In this paper, we focus on a frame-wise detection of stress on several window analysis sizes and analyze the behavior of different audio features classes. We trained our system on 19 subjects, and test it on 10 other subjects. Our best system obtains a 71.9 percent Unweighted Average Recall on 5s windows.

The video of our demonstrator presents a social interaction system for studying humor with the Aldebaran robot NAO. Our application records and analyzes audio and video stream to provide real-time feedback. Using this dialog system during show & tell sessions at Interspeech 2013, we have collected different kind of laughter (positive and negative) from 45 subjects. The participants were involved in a verbal exchange with NAO, including tongue-twisters games and jokes, as well as witty remarks and laughs from the robot. The conversation data captured is used here to study subject behaviors from various personalities and cultural backgrounds.

Negative emotions such as anger recognition in particular can deliver useful information to both the customer and the agent of Interactive Voice Response platforms. The state-of-the-art of emotion detection is characterized as not taking into account real-life emotion behavior but “realistic” induced emotion. This study is part of the French project Voxfactory (Cap Digital). The aim is to analyze the quality of the interactions collected in call centers by using the topics of the dialogs, but also informations on opinions and emotions. A corpus of 18 hours of real dialogs between agent and customer collected in a service of complaints of French company EDF (power supply) has been annotated with emotional labels. We describe experiments on detection of three emotional states during calls. Full speaker independent test set has been used in order to be closer to a real life situation. The novelty of this paper is the analysis of full conversations (including turns with low confidence in emotion annotation and noisy turns) and the impact on the detection score. The idea is to see how far we are from a system adapted to a real life situation.

In this article, the detection of real-life emotions is explored across different corpora featuring a similar task. Two emotional states (Anger and Neutral) are thus examined across three French corpora collected in call centers in different contexts (service complaints, Stock Exchange service and medical emergency). The effects of these differents contexts on the performance of acoustic-based emotion detection systems are investigated with two objectives in mind: improving the generalization power of existing models by handling these differences and trying to quantify the differences between corpora in order to easily compare them. The described experiments are based on the following approach: examining the detection performance when training on a corpus and testing on another one or when training on mixtures of the corpora.

International audience

In this paper we present an improved Sequential Forward Floating Search algorithm. Subsequently, extensive tests are carried out on a selection of French emotional language resources well suited for a first impression on general applicability. A detailed analysis is presented to test the various modifications suggested one-by-one. Our conclusion is that the modification in the forward step result in a considerable improvement in speed (∼80%) while no considerable and systematic loss in quality is experienced. The modifications in the backward step seem to have only significance when a higher number of features is achieved. The final clarification of this issue remains the task of future work. As a result we may suggest a quick feature selection algorithm, which is practically more suitable for the state of the art, larger corpora and wider feature-banks. Our quick SFFS is general: it can also be used in any other field of application. Index Terms: emotion detection, feature selection heuristics 1.

In this article, we describe experiments on the detection of three emotional states (Anger, Positive and Neutral) for two French corpora collected in call centers in different contexts (service complaints and medical emergency). These corpora have a high level of privacy. In order to be comparable with results obtained in the community we used the openEAR acoustic features extraction platform instead of our own library. One of our aims being the comparison of anger and positive emotions across corpora, we train models on one corpus and test it on the other to compare their similarities, then conversely. We will discuss the possible gain in generalization power.

It is not fully known how long it takes a human to reliably recognize emotion in speech from the beginning of a phrase. However, many technical applications demand for very quick system responses, e. g. to prepare different feedback alternatives before the end of a speaker turn in a dialog system. We therefore investigate this ‘gating paradigm ’ employing two spoken language resources in a cross- and combined manner with a focus on valence: we determine how quick a reliable estimate is obtainable and whether matching by models trained on the same length of speech prevails. In addition we analyze how individual feature groups by type and derived functionals respond and find considerably different behavior. The language resources have been chosen to cover for manually segmented and automatically segmented speech at the same time. In the result one second of speech is sufficient on the datasets considered. Index Terms: affective computing, automatic emotion recognition, incremental speech processing, gating paradigm

Most paralinguistic analysis tasks are lacking agreed-upon evaluation procedures and comparability, in contrast to more ‘traditional ’ disciplines in speech analysis. The INTERSPEECH 2010 Paralinguistic Challenge shall help overcome the usually low compatibility of results, by addressing three selected subchallenges. In the Age Sub-Challenge, the age of speakers has to be determined in four groups. In the Gender Sub-Challenge, a three-class classification task has to be solved and finally, the Affect Sub-Challenge asks for speakers ’ interest in ordinal representation. This paper introduces the conditions, the Challenge corpora “aGender ” and “TUM AVIC ” and standard feature sets that may be used. Further, baseline results are given.

In this paper, we report on classification results for emotional user states (4 classes, German database of children interacting with a pet robot).Starting with 5 emotion labels per word, we obtained chunks with different degrees of prototypicality.Six sites computed acoustic and linguistic features independently from each other.A total of 4232 features were pooled together and grouped into 10 low level descriptor types.For each of these groups separately and for all taken together, classification results using Support Vector Machines are reported for 150 features each with the highest individual Information Gain Ratio, for a scale of prototypicality.With both acoustic and linguistic features, we obtained a relative improvement of up to 27.6%, going from low to higher prototypicality.

Recent work on emotional speech processing has demonstrated the interest to consider the information conveyed by the emotional component in speech to enhance the understanding of human behaviors. But to date, there has been little integration of emotion detection systems in effective applications. The present research focuses on the development of a fear-type emotions recognition system to detect and analyze abnormal situations for surveillance applications. The Fear vs. Neutral classification gets a mean accuracy rate at 70.3%. It corresponds to quite optimistic results given the diversity of fear manifestations illustrated in the data. More specific acoustic models are built inside the fear class by considering the context of emergence of the emotional manifestations, i.e. the type of the threat during which they occur, and which has a strong influence on fear acoustic manifestations. The potential use of these models for a threat type recognition system is also investigated. Such information about the situation can indeed be useful for surveillance systems.

In this paper, we report on classification results for emotional user states (4 classes, German database of children interacting with a pet robot).Six sites computed acoustic and linguistic features independently from each other, following in part different strategies.A total of 4244 features were pooled together and grouped into 12 low level descriptor types and 6 functional types.For each of these groups, classification results using Support Vector Machines and Random Forests are reported for the full set of features, and for 150 features each with the highest individual Information Gain Ratio.The performance for the different groups varies mostly between ≈ 50% and ≈ 60%.

The emotion detection work reported here is part of a larger study aiming to model user behavior in real interactions. We already studied emotions in a real-life corpus with human-human dialogs on a financial task. We now make use of another corpus of real agent-caller spoken dialogs from a medical emergency call center in which emotion manifestations are much more complex, and extreme emotions are common. Our global aims are to define appropriate verbal labels for annotating real-life emotions, to annotate the dialogs, to validate the presence of emotions via perceptual tests and to find robust cues for emotion detection. Annotations have been done by two experts with twenty verbal classes organized in eight macro-classes. We retained for experiments in this paper four macro classes: Relief, Anger, Fear and Sadness. The relevant cues for detecting natural emotions are paralinguistic and linguistic. Two studies are reported in this paper: the first investigates automatic emotion detection using linguistic information, whereas the second investigates emotion detection with paralinguistic cues. On the medical corpus, preliminary experiments using lexical cues detect about 78% of the four labels showing very good detection for Relief (about 90%) and Fear (about 86%) emotions. Experiments using paralinguistic cues show about 60% of good detection, Fear being best detected. Index Terms: emotion detection, real-life emotion, lexical and paralinguistic cues

In the context of call centers, emotion detection is potentially important for customer care. Emotions in natural interaction are often blended. For example, in a Stock Exchange service centre, some customers are angry because they are afraid to lose money. A 100 agent-client dialog corpus has been annotated at the speaker turn level with one label among 5 emotions including fear and anger. In this paper, we report on our experiments of automatic emotion detection using acoustic cues with several classifiers. 73% correct detection was achieved in discriminating between negative and neutral emotions and 60% between anger and fear. An analysis of the confusions led us to question the validity of the initial single valued annotation scheme. It was found that customer emotional states can be a mixture of anger and fear. As a result a new annotation scheme is used allowing the selection of two verbal labels per segment.

In everyday life, speech is part of a multichannel system involved in conveying emotion. Understanding how it operates in that context requires suitable data, consisting of multimodal records of emotion drawn from everyday life. This paper reflects the experience of two teams active in collecting and labelling data of this type. It sets out the core reasons for pursuing a multimodal approach, reviews issues and problems for developing relevant databases, and indicates how we can move forward both in terms of data collection and approaches to labelling.

In contrast to most previous studies conducted on artificial data with archetypal emotions, this paper addresses some of the challenges faced when studying real-life non-basic emotions. A 10-hour dialog corpus recorded in a French Medical emergency call center has been studied. In this paper, our aim is twofold, to study emotion mixtures in natural data and to achieve high level of performances in real-life emotion detection. An annotation scheme using two labels per segment has been used for representing emotion mixtures. A closer study of these mixtures has been carried out, revealing the presence of conflictual valence emotions. A correct detection rate of about 82% was obtained between Negative and Positive emotions using paralinguistic cues without taking into account these conflictual blended emotions. A perceptive study and the confusions analysis obtained with automatic detection enable to validate the annotation scheme with two emotion labels per segment.

The present research focuses on the acquisition and annotation of vocal resources for emotion detection. We are interested in detecting emotions occurring in abnormal situations and particularly in detecting ”fear”. The present study considers a preliminary database of audiovisual sequences extracted from movie fictions. The sequences selected provide various manifestations of target emotions and are described with a multimodal annotation tool. We focus on audio cues in the annotation strategy and we use the video as support for validating the audio labels. The present article deals with the description of the methodology of data acquisition and annotation. The validation of annotation is realized via two perceptual paradigms in which the +/-video condition in stimuli presentation varies. We show the perceptual significance of the audio cues and the presence of target emotions.

Detecting emotions in the context of automated call center services can be helpful for following the evolution of the human-computer dialogues, enabling dynamic modification of the dialogue strategies and influencing the final outcome. The emotion detection work reported here is a part of larger study aiming to model user behavior in real interactions. We make use of a corpus of real agent-client spoken dialogues in which the manifestation of emotion is quite complex, and it is common to have shaded emotions since the interlocutors attempt to control the expression of their internal attitude. Our aims are to define appropriate emotions for call center services, to annotate the dialogues and to validate the presence of emotions via perceptual tests and to find robust cues for emotion detection. In contrast to research carried out with artificial data with simulated emotions, for real-life corpora the set of appropriate emotion labels must be determined. Two studies are reported: the first investigates automatic emotion detection using linguistic information, whereas the second concerns perceptual tests for identifying emotions as well as the prosodic and textual cues which signal them. About 11% of the utterances are annotated with non-neutral emotion labels. Preliminary experiments using lexical cues detect about 70% of these labels.

This paper reports on an analysis of prosodic cues for emotion characterization in 100 natural spoken dialogs recorded at a telephone customer service center. The corpus annotated with task-dependent emotion tags which were validated by a perceptual test. Two F0 range parameters, one at the sentence level and the other at the subsegment level, emerge as the most salient cues for emotion classification. These parameters can differentiate between negative emotion (irritation/anger, anxiety/fear) and neutral attitude and confirm trends illustrated by the perceptual experiment.

One central goal of the AMITIÉS multilingual human-computer dialogue project is to create a dialogue management system capable of engaging the user in human-like conversation in a specific domain. To that end, we have developed new methods for the manual annotation of spoken dialogue transcriptions from European financial call centers. We have modified the DAMSL dialogic schema to create a dialogue act taxonomy appropriate for customer services. To capture the semantics, we use a domain-independent framework populated with domain-specific lists. We have designed a new flexible, platform-independent annotation tool, XDMLTool, and annotated several hundred dialogues in French and English. Inter-annotator agreement was moderate. We are using these data to design our dialogue system, and we hope that they will help us to derive appropriate dialogue strategies for novel situations. 1.

In this paper, we describe the evaluation of the dialog management and response generation strategies being developed for retrieval of touristic information, selected as a common domain for the ARC AUPELF-B2 action. A large number of spoken dialog systems have been reported in the literature making use of different strategies for dialog management. Comparing and evaluating different strategies is a difficult task, which often remains unexplored, because in most cases evaluation approaches require a unified database structure and efficient integration of data from several disparate sources and forms. To avoid this problem, we implemented two dialog strategy versions within the same general platform. We investigate qualitative and quantitative criteria for evaluation of these dialog control strategies: in particular, by testing the efficiency of our system with and without automatic mechanisms for guiding the user via suggestive prompts. An evaluation phase has been carried out to assess...

In this paper we report on our ongoing work in developing spoken language corpora in the context of information access in two travel domain tasks, L'ATIS and MASK. The collection of spoken language corpora remains an important research area and represents a significant portion of work in the development of spoken language systems. The use of additional acoustic and language model training data has been shown to almost systematically improve performance in continuous speech recognition. Similarly, progress in spokenlanguage understanding is closely linked to the availability of spoken language corpora. We record subjects on a regular basis using development versions of the spoken language systems for both tasks, obtaining over 1000 queries/month from 20 subjects. To help assess our progress in system development, each subject since March'95 completes a questionnaire addressing the user-friendliness, reliability, ease-of-use of the MASK data collection system. INTRODUCTION The collecti...

Hybrid systems, using neural networks (NNs) and hidden Markov models (HMMs) are designed to take advantage of both methods; the pattern classification power of NNs and the temporal modelling structure of HMMs. This paper describes the use of expert sub-network modules of the type time delay neural network (TDNN) for phone recognition of continuous speech. The originality of the hybrid system developed is in combining the probabilities of the modular TDNN architecture with those of CDHMMs during the recognition phase. On three speakers of the DARPA RM speaker-dependent task, we show that these small TDNNs trained on phone ambiguities can improve word recognition performance of state-of-the-art CDHMMs. The TDNN implementation achieved a word error rate reduction of 15%. We discuss strategies for extending this approach from the DARPA RM speaker-dependent database to the larger DARPA RM speaker-independent database.>

The paper presents a hybrid continuous-speech recognition system that leads to improved results on the speaker dependent DARPA Resource Management task. This hybrid system, called the combined system, is based on a combination of normalized neural network output scores with hidden Markov model (HMM) emission probabilities. The neural network is trained under mean square error and the HMM is trained under maximum likelihood estimation. In theory, whatever criterion may be used, the same word error rate should be reached if enough training data is available. As this is never the case, the idea of combining two different criteria, each of them extracting complementary characteristics of the feature is interesting. A state-of-the-art HMM system will be combined with a time delay neural network (TDNN) integrated in a Viterbi framework. A hierarchical TDNN structure is described that splits training into subtasks corresponding to subsets of phonemes. This structure makes training of TDNNs on large-vocabulary tasks manageable on workstations. It will be shown that the combined system, despite the low accuracy of the hierarchical TDNN, achieves a word error rate reduction of 15% with respect to our state-of-the-art HMM system. This reduction is obtained with a 10% increase only in the number of parameters.

A comparison of several architectures of time-delayed neural networks (TDNN) as the preprocessing step for hidden Markov modeling (HMM) speaker-dependent continuous-speech recognition systems is presented. A modular TDNN architecture on the basis of acoustic-phonetic knowledge, where each sub-network is trained on a different subset of phonemes is defined. It allows the authors to define a hierarchical tree structure of sub-networks. This structure offers the possibility of proposing a framework to enlarge the number of outputs by defining context-dependent sub-networks. They also compare different methods for integrating TDNN in a HMM framework, a discrete and a continuous integration. For the speaker JWS-4 of the speaker-dependent DARPA RM1 database, with context independent phonemes, 21.3% word error rate are obtained without grammar, 4.6% with the DARPA word-pair grammar (perplexity of 60).>