In recent years, there has been a significant increase in interest in developing Spoken Language Understanding (SLU) systems. SLU involves extracting a list of semantic information from the speech signal. A major issue for SLU systems is the lack of sufficient amount of bi-modal (audio and textual semantic annotation) training data. Existing SLU resources are mainly available in high-resource languages such as English, Mandarin and French. However, one of the current challenges concerning low-resourced languages is data collection and annotation. In this work, we present a new freely available corpus, named TARIC-SLU, composed of railway transport conversations in Tunisian dialect that is continuously annotated in dialogue acts and slots. We describe the semantic model of the dataset, the data and experiments conducted to build ASR-based and SLU-based baseline models. To facilitate its use, a complete recipe, including data preparation, training and evaluation scripts, has been built and will be integrated to SpeechBrain, a popular open-source conversational AI toolkit based on PyTorch.

SpeechBrain is an open-source Conversational AI toolkit based on PyTorch, focused particularly on speech processing tasks such as speech recognition, speech enhancement, speaker recognition, text-to-speech, and much more. It promotes transparency and replicability by releasing both the pre-trained models and the complete recipes of code and algorithms required for training them. This paper presents SpeechBrain 1.0, a significant milestone in the evolution of the toolkit, which now has over 200 recipes for speech, audio, and language processing tasks, and more than 100 models available on Hugging Face. SpeechBrain 1.0 introduces new technologies to support diverse learning modalities, Large Language Model (LLM) integration, and advanced decoding strategies, along with novel models, tasks, and modalities. It also includes a new benchmark repository, offering researchers a unified platform for evaluating models across diverse tasks.

In this paper, we propose a novel end-to-end sequence-to-sequence spoken language understanding model using an attention mechanism. It reliably selects contextual acoustic features in order to hypothesize semantic contents. An initial architecture capable of extracting all pronounced words and concepts from acoustic spans is designed and tested. With a shallow fusion language model, this system reaches a 13.6 concept error rate (CER) and an 18.5 concept value error rate (CVER) on the French MEDIA corpus, achieving an absolute 2.8 points reduction compared to the state-of-the-art. Then, an original model is proposed for hypothesizing concepts and their values. This transduction reaches a 15.4 CER and a 21.6 CVER without any new type of context.

This paper presents a qualitative study of errors produced by an end-to-end spoken language understanding (SLU) system (speech signal to concepts) that reaches state of the art performance. Different studies are proposed to better understand the weaknesses of such systems: comparison to a classical pipeline SLU system, a study on the cause of concept deletions (the most frequent error), observation of a problem in the capability of the end-to-end SLU system to segment correctly concepts, analysis of the system behavior to process unseen concept/value pairs, analysis of the benefit of the curriculum-based transfer learning approach. Last, we proposed a way to compute embeddings of sub-sequences that seem to contain relevant information for future work.

This work investigates the embeddings for representing dialog history in spoken language understanding (SLU) systems. We focus on the scenario when the semantic information is extracted directly from the speech signal by means of a single end-to-end neural network model. We proposed to integrate dialogue history into an end-to-end signal-to-concept SLU system. The dialog history is represented in the form of dialog history embedding vectors (so-called h-vectors) and is provided as an additional information to end-to-end SLU models in order to improve the system performance. Three following types of h-vectors are proposed and experimentally evaluated in this paper: (1) supervised-all embeddings predicting bag-of-concepts expected in the answer of the user from the last dialog system response; (2) supervised-freq embeddings focusing on predicting only a selected set of semantic concept (corresponding to the most frequent errors in our experiments); and (3) unsupervised embeddings. Experiments on the MEDIA corpus for the semantic slot filling task demonstrate that the proposed h-vectors improve the model performance.

Machine learning (ML) and deep learning with deep neural networks (DNN), have drastically improved the performances of modern systems on numerous spoken language understanding (SLU) related tasks. Since most of current researches focus on new neural architectures to enhance the performances in realistic conditions, few recent works investigated the use of different algebras with neural networks (NN), to better represent the nature of the data being processed. To this extent, quaternion-valued neural networks (QNN) have shown better performances, and an important reduction of the number of neural parameters compared to traditional real-valued neural networks, when dealing with multidimensional signal. Nonetheless, the use of QNNs is strictly limited to quaternion input or output features. This article introduces a new unsupervised method based on a hybrid autoencoder (AE) called real-to-quaternion autoencoder (R2H), to extract a quaternion-valued input signal from any real-valued data, to be processed by QNNs. The experiments performed to identify the most related theme of a given telephone conversation from a customer care service (CCS), demonstrate that the R2H approach outperforms all the previously established models, either real- or quaternion-valued ones, in term of accuracy and with up to four times fewer neural parameters.

Recurrent neural networks (RNN) are at the core of modern automatic speech recognition (ASR) systems. In particular, long short-term memory (LSTM) recurrent neural networks have achieved state-of-the-art results in many speech recognition tasks, due to their efficient representation of long and short term dependencies in sequences of inter-dependent features. Nonetheless, internal dependencies within the element composing multidimensional features are weakly considered by traditional real-valued representations. We propose a novel quaternion long short-term memory (QL-STM) recurrent neural network that takes into account both the external relations between the features composing a sequence, and these internal latent structural dependencies with the quaternion algebra. QLSTMs are compared to LSTMs during a memory copy-task and a realistic application of speech recognition on the Wall Street Journal (WSJ) dataset. QLSTM reaches better performances during the two experiments with up to 2.8 times less learning parameters, leading to a more expressive representation of the information.

Deep neural networks (DNNs) and more precisely recurrent neural networks (RNNs) are at the core of modern automatic speech recognition systems, due to their efficiency to process input sequences. Recently, it has been shown that different input representations, based on multidimensional algebras, such as complex and quaternion numbers, are able to bring to neural networks a more natural, compressive and powerful representation of the input signal by outperforming common real-valued NNs. Indeed, quaternion-valued neural networks (QNNs) better learn both internal dependencies, such as the relation between the Mel-filter-bank value of a specific time frame and its time derivatives, and global dependencies, describing the relations that exist between time frames. Nonetheless, QNNs are limited to quaternion-valued input signals, and it is difficult to benefit from this powerful representation with real-valued input data. This paper proposes to tackle this weakness by introducing a real-to-quaternion encoder that allows QNNs to process any one dimensional input features, such as traditional Mel-filter-banks for automatic speech recognition.

Recently, the connectionist temporal classification (CTC) model coupled with recurrent (RNN) or convolutional neural networks (CNN), made it easier to train speech recognition systems in an end-to-end fashion. However in real-valued models, time frame components such as mel-filter-bank energies and the cepstral coefficients obtained from them, together with their first and second order derivatives, are processed as individual elements, while a natural alternative is to process such components as composed entities. We propose to group such elements in the form of quaternions and to process these quaternions using the established quaternion algebra. Quaternion numbers and quaternion neural networks have shown their efficiency to process multidimensional inputs as entities, to encode internal dependencies, and to solve many tasks with less learning parameters than real-valued models. This paper proposes to integrate multiple feature views in quaternion-valued convolutional neural network (QCNN), to be used for sequence-to-sequence mapping with the CTC model. Promising results are reported using simple QCNNs in phoneme recognition experiments with the TIMIT corpus. More precisely, QCNNs obtain a lower phoneme error rate (PER) with less learning parameters than a competing model based on real-valued CNNs.

Quaternion convolutional neural networks (QCNN) are powerful architectures to learn and model external dependencies that exist between neighbor features of an input vector, and internal latent dependencies within the feature. This paper proposes to evaluate the effectiveness of the QCNN on a realistic theme identification task of spoken telephone conversations between agents and customers from the call center of the Paris transportation system (RATP). We show that QCNNs are more suitable than real-valued CNN to process multidimensional data and to code internal dependencies. Indeed, real-valued CNNs deal with both internal and external relations at the same level since components of an entity are processed independently. Experimental evidence is provided that the proposed QCNN architecture always outperforms real-valued equivalent CNN models in the theme identification task of the DECODA corpus. It is also shown that QCNN accuracy results are the best achieved so far on this task, while reducing by a factor of 4 the number of model parameters.

This paper addresses the problem of automatic speech recognition (ASR) error detection and their use for improving spoken language understanding (SLU) systems. In this study, the SLU task consists in automatically extracting, from ASR transcriptions , semantic concepts and concept/values pairs in a e.g touristic information system. An approach is proposed for enriching the set of semantic labels with error specific labels and by using a recently proposed neural approach based on word embeddings to compute well calibrated ASR confidence measures. Experimental results are reported showing that it is possible to decrease significantly the Concept/Value Error Rate with a state of the art system, outperforming previously published results performance on the same experimental data. It also shown that combining an SLU approach based on conditional random fields with a neural encoder/decoder attention based architecture , it is possible to effectively identifying confidence islands and uncertain semantic output segments useful for deciding appropriate error handling actions by the dialogue manager strategy .

Automatic transcription of spoken documents is affected by automatic transcription errors that are especially frequent when speech is acquired in severe noisy conditions. Automatic speech recognition errors induce errors in the linguistic features used for a variety of natural language processing tasks. Recently, denoisng autoencoders (DAE) and stacked autoencoders (SAE) have been proposed with interesting results for acoustic feature denoising tasks. This paper deals with the recovery of corrupted linguistic features in spoken documents. Solutions based on DAEs and SAEs are considered and evaluated in a spoken conversation analysis task. In order to improve conversation theme classification accuracy, the possibility of combining abstractions obtained from manual and automatic transcription features is considered. As a result, two original representations of highly imperfect spoken documents are introduced. They are based on bottleneck features of a supervised autoencoder that takes advantage of both noisy and clean transcriptions to improve the robustness of error prone representations. Experimental results on a spoken conversation theme identification task show substantial accuracy improvements obtained with the proposed recovery of corrupted features.

International audience

Recently, machine learning methods have provided a broad spectrum of original and efficient algorithms based on Deep Neural Networks (DNN) to automatically predict an outcome with respect to a sequence of inputs. Recurrent hidden cells allow these DNN-based models to manage long-term dependencies such as Recurrent Neural Networks (RNN) and Long Short-Term Memory (LSTM). Nevertheless, these RNNs process a single input stream in one (LSTM) or two (Bidirectional LSTM) directions. But most of the information available nowadays is from multistreams or multimedia documents, and require RNNs to process these information synchronously during the training. This paper presents an original LSTM-based architecture, named Parallel LSTM (PLSTM), that carries out multiple parallel synchronized input sequences in order to predict a common output. The proposed PLSTM method could be used for parallel sequence classification purposes. The PLSTM approach is evaluated on an automatic telecast genre sequences classification task and compared with different state-of-the-art architectures. Results show that the proposed PLSTM method outperforms the baseline n-gram models as well as the state-of-the-art LSTM approach.

Machine Learning (ML) techniques have allowed a great performance improvement of different challenging Spoken Language Understanding (SLU) tasks. Among these methods, Neural Networks (NN), or Multilayer Perceptron (MLP), recently received a great interest from researchers due to their representation capability of complex internal structures in a low dimensional subspace. However, MLPs employ document representations based on basic word level or topic-based features. Therefore, these basic representations reveal little in way of document statistical structure by only considering words or topics contained in the document as a “bag-of-words”, ignoring relations between them. We propose to remedy this weakness by extending the complex features based on Quaternion algebra presented in [1] to neural networks called QMLP. This original QMLP approach is based on hyper-complex algebra to take into consideration features dependencies in documents. New document features, based on the document structure itself, used as input of the QMLP, are also investigated in this paper, in comparison to those initially proposed in [1]. Experiments made on a SLU task from a real framework of human spoken dialogues showed that our QMLP approach associated with the proposed document features outperforms other approaches, with an accuracy gain of 2% with respect to the MLP based on real numbers and more than 3% with respect to the first Quaternion-based features proposed in [1]. We finally demonstrated that less iterations are needed by our QMLP architecture to be efficient and to reach promising accuracies.

Many software development tasks require developers to quickly learn a subset of an Application Programming Interface (API). API learning resources are crucial for helping developers learn an API, but the knowledge relevant to a particular topic of interest may easily be scattered across different documents, which makes finding the necessary information more challenging. This paper proposes an approach to discovering tutorial sections that explain a given API type. At the core of our approach, we classify fragmented tutorial sections using supervised text classification based on linguistic and structural features. Experiments conducted on five tutorials show that our approach is able to discover sections explaining an API type with precision between 0.69 and 0.87 (depending on the tutorial) when trained and tested on the same tutorial. When trained and tested across tutorials, we obtained a precision between 0.74 and 0.94 and lower recall values.

Mapping text documents in an LDA-based topic-space is a classical way to extract high-level representation of text documents. Unfortunately, LDA is highly sensitive to hyper-parameters related to the number of classes, or word and topic distribution, and there is no systematic way to pre-estimate optimal configurations. Moreover, various hyper-parameter configurations offer complementary views on the document. In this paper, we propose a method based on a two-step process that, first, expands the representation space by using a set of topic spaces and, second, compacts the representation space by removing poorly relevant dimensions. These two steps are based respectively on multi-view LDA-based representation spaces and factor-analysis models. This model provides a view-independent representation of documents while extracting complementary information from a massive multi-view representation. Experiments are conducted on the DECODA conversation corpus and the Reuters-21578 textual dataset. Results show the efficiency of the proposed multiview compact representation paradigm. The proposed categorization system reaches an accuracy of 86.5% with automatic transcriptions of conversations from DECODA corpus and a Macro-F1 of 80% during a classification task of the well-known Reuters-21578 corpus, with a significant gain compared to the baseline (best single topic space configuration), as well as methods and document representations previously studied.

Mohamed Morchid, Mohamed Bouallegue, Richard Dufour, Georges Linarès, Driss Matrouf, Renato de Mori. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2014.

In this paper, we study the impact of dialogue representations and classification methods in the task of theme identification of telephone conversation services having highly imperfect automatic transcriptions. Two dialogue representations are firstly compared: the classical Term Frequency-Inverse Document Frequency with Gini purity criteria (TF-IDF-Gini) method and the Latent Dirichlet Allocation (LDA) approach. We then propose to study an original classification method that takes advantage of the LDA topic space representation, highlighted as the best dialogue representation. To do so, two assumptions about topic representation led us to choose a Gaussian process (GP) based method. This approach is compared with a Support Vector Machine (SVM) classification method. Results show that the GP approach is a better solution to deal with the multiple theme complexity of a dialogue, no matter the conditions studied (manual or automatic transcriptions). We finally discuss the impact of the topic space reduction on the classification accuracy.

International audience

International audience

The performance of Automatic Speech Recognition (ASR) systems drops dramatically when used in noisy environments. Speech analytics suffer from this poor quality of automatic transcriptions. In this paper, we seek to identify themes from dialogues of telephone conversation services using multiple topic-spaces estimated with a Latent Dirichlet Allocation (LDA) approach. This technique consists in estimating several topic models that offer different views of the document. Unfortunately, such a multi-model approach also introduces additional vari-abilities due to the model diversity. We propose to extract the useful information from the full model-set by using an i-vector based approach, previously developed in the context of speaker recognition. Experiments are conducted on the DECODA corpus , that contains records from the call center of the Paris Transportation Company. Results show the effectiveness of the proposed representation paradigm, our identification system reaching an accuracy of 84.7%, with a gain of 3.3 points compared to the baseline.

International audience

This paper deals with the automatic analysis of conversations between a customer and an agent in a call centre of a customer care service. The purpose of the analysis is to hypothesize themes about problems and complaints discussed in the conversation. Themes are defined by the application documentation topics. A conversation may contain mentions that are irrelevant for the application purpose and multiple themes whose mentions may be interleaved portions of a conversation that cannot be well defined. Two methods are proposed for multiple theme hypothesization. One of them is based on a cosine similarity measure using a bag of features extracted from the entire conversation. The other method introduces the concept of thematic density distributed around specific word positions in a conversation. In addition to automatically selected words, word bigrams with possible gaps between successive words are also considered and selected. Experimental results show that the results obtained with the proposed methods outperform the results obtained with support vector machines on the same data. Furthermore, using the theme skeleton of a conversation from which thematic densities are derived, it will be possible to extract components of an automatic conversation report to be used for improving the service performance.

International audience

The integration of a cache memory into a connectionist language model is proposed in this paper. The model captures long term dependencies of both words and concepts and is particularly useful for Spoken Language Understanding tasks. Experiments conducted on a human-machine telephone dialog corpus are reported, and an increase in performance is observed when features of previous turns are taken into account for predicting the concepts expressed in a user turn. In terms of Concept Error Rate we obtained a statistically significant improvement of 3.2 points over our baseline (10% relative improvement) on the French Media corpus.

A conditional model is introduced for triggering understanding actions that correct errors of frame hypothesization and composition. Experimental evidence is provided using the French MEDIA corpus that these models trained with automatic speech recognition hypotheses trigger effective corrections of more than half of the errors. The overall frame recall increases from 0.76 to 0.84 while precision increases from 0.78 to 0.85. The number of fully corrected dialog turns in creases of 8.8% making about half of the turns fully correct.

A research on the analysis of human/human conversations in a call centre is described. The purpose of the research is to provide short reports of each conversation with information useful for monitoring the call centre efficiency. Data from real users discussing over the telephone with agents are processed by an automatic speech recognition (ASR) system. Reports are grouped into classes by the agents based on predefined taxonomy. A train set of manually transcribed data is used for training the extraction of features relevant to the application and the classification of the conversations. The use of all the words of the application vocabulary, of automatically selected key_words, and of automatically learned sentence chunks containing semantic classes of words are compared and evaluated with a totally different test set. The results show a significant increase in performance when chunks are used even in comparison with the use of bags of words obtained with a boosting algorithm.

One of the first steps in building a spoken language understanding (SLU) module for dialogue systems is the extraction of flat concepts out of a given word sequence, usually provided by an automatic speech recognition (ASR) system. In this paper, six different modeling approaches are investigated to tackle the task of concept tagging. These methods include classical, well-known generative and discriminative methods like Finite State Transducers (FSTs), Statistical Machine Translation (SMT), Maximum Entropy Markov Models (MEMMs), or Support Vector Machines (SVMs) as well as techniques recently applied to natural language processing such as Conditional Random Fields (CRFs) or Dynamic Bayesian Networks (DBNs). Following a detailed description of the models, experimental and comparative results are presented on three corpora in different languages and with different complexity. The French MEDIA corpus has already been exploited during an evaluation campaign and so a direct comparison with existing benchmarks is possible. Recently collected Italian and Polish corpora are used to test the robustness and portability of the modeling approaches. For all tasks, manual transcriptions as well as ASR inputs are considered. Additionally to single systems, methods for system combination are investigated. The best performing model on all tasks is based on conditional random fields. On the MEDIA evaluation corpus, a concept error rate of 12.6% could be achieved. Here, additionally to attribute names, attribute values have been extracted using a combination of a rule-based and a statistical approach. Applying system combination using weighted ROVER with all six systems, the concept error rate (CER) drops to 12.0%.

Across language portability of a spoken language understanding system (SLU) deals with the possibility of reusing with moderate effort in a new language knowledge and data acquired for another language. The approach proposed in this paper is motivated by the availability of the fairly large MEDIA corpus carefully transcribed in French and semantically annotated in terms of constituents. A method is proposed for manually translating a portion of the training set for training an automatic machine translation (MT) system to be used for translating the remaining data. As the source language is annotated in terms of concept tags, a solution is presented for automatically transferring these tags to the translated corpus. Experimental results are presented on the accuracy of the translation expressed with the BLEU score as function of the size of the training corpus. It is shown that the process leads to comparable concept error rates in the two languages making the proposed approach suitable for SLU portability across languages.

Two approaches to Spoken Language Understanding based on frames describing chunked knowledge are described. They are applied to the MEDIA corpus annotated in terms of concepts expressing chunks of spoken sentences. General rules of knowledge composition and inference appear to be adequate to effectively applying the application ontology for obtaining frame based representations of dialogue turns. The main difficulty appears to be the characterization of the syntactic knowledge expressing semantic links between knowledge chunks. This knowledge can be hand-crafted or automatically learned from examples. It is shown that the latter approach outperforms the former if applied to ASR error prone transcriptions.

This paper describes a system for automatic opinion analysis from spoken messages collected in the context of a user satisfaction survey. Opinion analysis is performed from the perspective of opinion monitoring. A process is outlined for detecting segments expressing opinions in a speech signal. Methods are proposed for accepting or rejecting segments from messages that are not reliably analyzed due to the limitations of automatic speech recognition processes, for assigning opinion hypotheses to segments and for evaluating hypothesis opinion proportions. Specific language models are introduced for representing opinion concepts. These models are used for hypothesizing opinion carrying segments in a spoken message. Each segment is interpreted by a classifier based on the Adaboost algorithm which associates a pair of topic and polarity labels to each segment. The different processes are trained and evaluated on a telephone corpus collected in a deployed customer care service. The use of conditional random fields (CRFs) is also considered for detecting segments and results are compared for different types of data and approaches. By optimizing the choice of the strategy parameters, it is possible to estimate user opinion proportions with a Kullback-Leibler divergence of 0.047 bits with respect to the true proportions obtained with a manual annotation of the spoken messages. The proportions estimated with such a low divergence are accurate enough for monitoring user satisfaction over time.

In the context of spoken language interpretation, this paper introduces a stochastic approach to infer and compose semantic structures. Semantic frame structures are directly derived from word and basic concept sequences representing the users' utterances. A rule-based process provides a reference frame annotation of the speech training data. Then dynamic Bayesian networks are used to hypothesize frames from test data. The semantic frames used in this work are specialized on the task domain from the Berkeley FrameNet set. Experiments are reported on the French MEDIA dialog corpus. For all the data, the manual transcriptions and annotations at the word and concept levels are available. Tests are performed under 3 different conditions raising in difficulty wrt the errors in the word and concept sequence inputs. Three different stochastic models are compared and the results confirm the ability of the proposed probabilistic frameworks to carry out a reliable semantic frame annotation.

International audience

International audience

With the purpose of improving spoken language understanding (SLU) performance, a combination of different acoustic speech recognition (ASR) systems is proposed. State a posteriori probabilities obtained with systems using different acoustic feature sets are combined with log-linear interpolation. In order to perform a coherent combination of these probabilities, acoustic models must have the same topology (i.e. same set of states). For this purpose, a fast and efficient twin model training protocol is proposed. By a wise choice of acoustic feature sets and log-linear interpolation of their likelihood ratios, a substantial concept error rate (CER) reduction has been observed on the test part of the French MEDIA corpus.

A knowledge representation formalism for SLU is introduced. It is used for incremental and partially automated annotation of the Media corpus in terms of semantic structures. An automatic interpretation process is described for composing semantic structures from basic semantic constituents using patterns involving constituents and words. The process has procedures for obtaining semantic compositions and for generating frame hypotheses by inference. This process is evaluated on a dialogue corpus manually annotated at the word and semantic constituent levels.

This paper describes an alarm detection system dedicated to process automatically customer feedbacks in call-centers. Previous studies presented a strategy that consists in the robust detection of subjective opinions about a particular topic in a spoken message. In the present study, we focus on the alarm detection problem in a customer spoken feedback application. We want to characterize each customer's survey with a degree of emergency. All the messages considered as urgent need a quick answer from the call-center service in order to satisfy the customer. The strategy proposed is based on a classification scheme that takes into account all the features that can characterize a survey: answers to the closed questions, topics and opinions detected in the open question spoken message, confidence scores from the Automatic Speech Recognition (ASR) and Spoken Language Understanding (SLU) modules. A field trial realized among real customers has shown that despite the ASR robustness issues, our system efficiently ranks the most urgent messages and brings a finer analysis on the surveys than the one provided by processing the closed questions alone.

International audience

This paper describes an utterance classification method based on a multiple decoding scheme. We use the Spoken Language Understanding (SLU) strategy proposed within the European project LUNA. The goal of this classification process is to characterize each speaker's turn, in a dialog context, according to different categories relevant from an SLU point of view: out-of-domain messages, requests not covered by the interpretation module, frequent requests,.... These categories are used for two purposes in an off-line mode: system monitoring for detecting changes in users' behaviour and system adaptation by selecting dialogs likely to contain some phenomenon poorly covered by the models for an active learning scheme. All the models and the evaluations are performed on the France Telecom FT3000 corpus.

Following recent studies in stochastic dialog management, this paper introduces an unsupervised approach aiming at reducing the cost and complexity for the setup of a probabilistic POMDP-based dialog manager. The proposed method is based on a first decoding step deriving semantic basic constituents from user utterances. These isolated units and some relevant context features (as previous system actions, previous user utterances...) are combined to form vectors representing the on-going dialog states. After a clustering step, each partition of this space is intented to represent a particular dialog state. Then any new utterance can be classified according to these automatic states and the belief state can be updated before the POMDP-based dialog manager can take a decision on the best next action to perform. The proposed approach is applied to the French media task (tourist information and hotel booking). The media 10k-utterance training corpus is semantically rich (over 80 basic concepts) and is segmentally annotated in terms of basic concepts. Before user trials can be carried out, some insights on the method effectiveness are obtained by analysis of the convergence of the POMDP models.

A survey of research on spoken language understanding is presented. It covers aspects of knowledge representation, automatic interpretation strategies, semantic grammars, conceptual language models, semantic event detection, shallow semantic parsing, semantic classification, semantic confidence, active learning.

Telephone services are now deployed that allow users to react to telephone prompts in spoken natural language. These systems have limited domain semantics and dialogue strategies which are represented by finite state diagrams. Most of these systems adopt a sequential approach where the automatic speech recognition (ASR) process, the spoken language understanding (SLU) process and the dialogue management (DM) are separate processes. In the framework of the France Telecom 3000 voice service, we propose in this paper to study several strategies in order to integrate more closely these three processes: ASR, SLU, and DM. By means of a finite state machine paradigm encoding the different models used by these three levels we show how the search for the best sequence of dialogue states can be done simultaneously at the word, concept, interpretation and dialogue state levels.

Within the framework of automatic analysis of spoken telephone surveys we propose a robust Speech Mining strategy that selects, from a large database of spoken messages, the ones likely to be correctly processed by the Automatic Speech Recognition and Classification processes. The problem considered in this paper is the analysis of messages uttered by the users of a telephone service in response to a recorded message that asks if a problem they had was satisfactorily solved. Very often in these cases, subjective information is combined with factual information. The purpose of this type of analysis is the extraction of the distribution of users opinions. Therefore it is very important to check the representativeness of the subset of messages kept by the rejection strategies. Several measures, based on the Kullback-Leibler divergence, are proposed in order to evaluate the correctness of the information extracted as well as its representativeness .

Within the context of a deployed spoken dialog service, this study presents a new interpretation strategy based on the sequential use of different ASR output representations: 1-best strings, word lattices and confusion networks. The goal is to reject as early as possible in the decoding process the nonrelevant messages containing non-speech or out-of-domain content. This is done through the 1-pass of the ASR decoding process thanks to specific acoustic and language models. A confusion network (CN) is then calculated for the remaining messages and another rejection process is applied with the confidence measures obtained in the CN. The messages kept at this stage are considered relevant; therefore the search for the best interpretation is applied to a richer search space than just the 1-best word string: either the whole CN or the whole word lattice. An improved, SLU oriented, CN generation algorithm is also proposed that significantly reduces the size of the CN obtained while improving the recognition performance. This strategy is evaluated on a large corpus of real users’ messages obtained from a deployed service. 1

Recognition errors made by automatic speech recognition (ASR) systems may not prevent the development of useful dialogue applications if the interpretation strategy has an introspection capability for evaluating the reliability of the results. This paper proposes an interpretation strategy which is particularly effective when applications are developed with a training corpus of moderate size. From the lattice of word hypotheses generated by an ASR system, a short list of conceptual structures is obtained with a set of finite state machines (FSM). Interpretation or a rejection decision is then performed by a tree-based strategy. The nodes of the tree correspond to elaboration-decision units containing a redundant set of classifiers. A decision tree based and two large margin classifiers are trained with a development set to become interpretation knowledge sources. Discriminative training of the classifiers selects linguistic and confidence-based features for contributing to a cooperative assessment of the reliability of an interpretation. Such an assessment leads to the definition of a limited number of reliability states. The probability that a proposed interpretation is correct is provided by its reliability state and transmitted to the dialogue manager. Experimental results are presented for a telephone service application

A method is described for predicting acoustic feature variability by analyzing the consensus and relative entropy of phoneme posterior probability distributions obtained with different acoustic models having the same type of observations. Variability prediction is used for diagnosis of automatic speech recognition (ASR) systems. When errors are likely to occur, different feature sets are considered for correcting recognition results. Experimental results are provided on the CH1 Italian portion of AURORA3

This paper briefly reviews state of the art related to the topic of speech variability sources in automatic speech recognition systems. It focuses on some variations within the speech signal that make the ASR task difficult. The variations detailed in the paper are intrinsic to the speech and affect the different levels of the ASR processing chain. For different sources of speech variation, the paper summarizes the current knowledge and highlights specific feature extraction or modeling weaknesses and current trends

A technique is proposed for the adaptation of automatic speech recognition systems using hybrid models combining artificial neural networks with hidden Markov models. The application of linear transformations not only to the input features, but also to the outputs of the internal layers is investigated. The motivation is that the outputs of an internal layer represent a projection of the input pattern into a space where it should be easier to learn the classification or transformation expected at the output of the network. A new solution, called conservative training, is proposed that compensates for the lack of adaptation samples in certain classes. Supervised adaptation experiments with different corpora and for different adaptation types are described. The results show that the proposed approach always outperforms the use of transformations in the feature space and yields even better results when combined with linear input transformations

International audience

This article proposes a new confidence measure estimated for concept hypotheses provided by a semantic language model used in the context of a dialog application. This confidence measure is based upon the ontology and more precisely, upon the semantic relations between concepts. It aims at measuring how high a concept hypothesis is related to the other hypotheses of an utterance. The semantic relation confidence measure is evaluated alone, and in combination with a classical acoustic confidence measure. The two measures are also used as parameters of a decision tree. It is shown that the two confidence measures are complementary and yield good performance in terms of cross entropy relative reduction .

In a previous paper, extensions of the 2-level stochastic speech understanding system have been proposed. Firstly the 3-level system is obtained through the introduction of a stochastic concept value normalization module. Then the 2+1-level system is obtained as a degraded 3-level system where the conceptual decoding and value normalization steps are decoupled, thus allowing to greatly reduce the model complexity and improve its trainability. In this paper, a multi-level spoken language understanding system is presented. This stochastic module is for the first time based on dynamic Bayesian networks. Factored language models with a generalized parallel backoff procedure are used as edge implementation to provide efficiently smoothed conditional probability estimates. This framework allows a great flexibility in terms of probability representation facilitating the development of the stochastic levels of the system. The proposed approaches, 3-level and 2+1-level, are evaluated on the French MEDIA task (tourist information and hotel booking). The MEDIA 10k-utterance training corpus is segmentally annotated, allowing a direct training of the various levels of the conceptual models. The best DBN-based system obtains performance comparable to those of the MEDIA'05 evaluation campaign best system (H. Bonneau-Maynard et al., 2005).

A soft decision gain modification is introduced and applied to the Ephraim-Malah gain function based on maximum mean-square-error estimation after amplitude compression. Nonlinear evaluations of the noise overestimation factor and spectral floor are used in the same way for the proposed gain modification and for nonlinear spectral subtraction (NSS). Consistent and statistically significant automatic speech recognition improvements of the proposed approach with respect to NSS are observed for different noise conditions considered in the Aurora-2 and Aurora-3 corpora. As the nonlinearity affects the two approaches in the same way, the result of comparison is particularly interesting.

The paper presents a semantic interpretation strategy, for spoken dialogue systems, including an error correction process. Semantic interpretations output by the spoken understanding module may be incorrect, but some semantic components may be correct. A set of situations are introduced, describing semantic confidence based on the agreement of semantic interpretations proposed by different classification methods. The interpretation strategy considers, with the highest priority, the validation of the interpretation arising from the most likely sequence of words. If our confidence score model gives a high probability that this interpretation is not correct, then possible corrections of it are considered using the other sequences in the N-best lists of possible interpretations. This strategy is evaluated on a dialogue corpus provided by France Telecom R&D and collected for a tourism telephone service. Significant reduction in understanding error rate are obtained as well as powerful new confidence measures.

International audience

This paper describes the construction and use of language models specific to semantic concepts. A tolerance mechanism is introduced in order to design conceptual constituents that cover longer word patterns while remaining robust to spontaneous speech disfluencies and recognition errors. These models are obtained from sentence patterns which grow from essential information to longer sequences of words. It is shown that these augmented conceptual constituents provide reductions in concept error rates on a spoken dialog application task. The composition of semantic constituents into semantic structures is discussed together with the introduction of language models for semantic composition. Experiments are described which show the advantage of using these language models to take into account structured semantic information.

A soft decision gain modification is introduced and applied to the Ephraim-Malah gain function based on maximum mean square error estimation (MMSE) (Ephraim, Y. and Malah, D., IEEE Trans. Acoust. Speech Sig. Process., vol.ASSP-32, no.6, p.1109-21, 1984; vol.ASSP-33, no.2, p.443-5, 1985) after amplitude compression. Non-linear evaluations of the noise overestimation factor and spectral floor are used in the same way for the proposed gain modification and for non-linear spectral subtraction (NSS). Consistent and statistically significant ASR improvements of the proposed approach with respect to NSS are observed for different noise conditions considered in the AURORA2 and AURORA3 corpora. As the non-linearity affects the two approaches in the same way, the comparison result is particularly interesting.

The paper proposes a new application of automatically trained decision trees to derive the interpretation of a spoken sentence. A new strategy for building structured cohorts of candidates is also described. By evaluating predicates related to the acoustic confidence of the words expressing a concept, the linguistic and semantic consistency of candidates in the cohort and the rank of a candidate within a cohort, the decision tree automatically learns a decision strategy for rescoring or rejecting an n-best list of candidates representing a user's utterance. A relative reduction of 18.6% in the understanding error rate is obtained by our rescoring strategy with no utterance rejection and a relative reduction of 43.1% of the same error rate is achieve with a rejection rate of only 8% of the utterances.

International audience

This paper deals with the difficult task of recognition of a large vocabulary of proper names in a directory assistance application. Research on the European project SMADA has shown that there is a need of an elaborate and effective decision strategy that limits the risk of false automation. This paper proposes a new strategy which integrates, as well as a general decoder, a set of decoders specialized in some specific situations. Specialized recognition processes do not need to be applicable for every input, but they have to be scheduled and performed only under certain conditions. A first implementation of such a model is proposed here, through a rejection strategy of the hypotheses output by a general decoder. This strategy leads to a very significant improvement over the results obtained by a standard rejection method based on acoustic confidence scores only.

International audience

Abstract Design recovery, which is a part of the reverse engineering process of source code, must supply programmers with all the information they need to fully understand a program or a system. In this paper, a connectionist method that can be used for design recovery in conjunction with more traditional approaches is proposed for analyzing the informal information (comments and mnemonics) in programs. An approach based on artificial neural networks (ANNs) was chosen because of its property of being robust (capable of tolerating noisy inputs), because of its associative memory ability (capable of retrieving a concept given only the context of the input word that originally fired the concept), and because of its generalization power (ability to learn conceptually relevant micro‐features of the domain). The proposed approach uses a combination of top down domain analysis (i.e., the creation of a concept hierarchy by a domain expert, to be used in the construction of the training set) and a bottom up approach (i.e., the analysis of the informal information using ANNs). A preprocessing system that extracts the relevant comments and identifier names and transforms them into an input for the ANNs has been developed. Feed‐forward neural networks (FNNs) and recurrent neural networks (RNNs) were tried. RNN architectures are capable of learning sequences and are able to make use of the word ordering of the sentence. The networks were trained on part of the source code of an existing system and tested on a different portion of the system code. Test results, consisting of coverage and evaluation figures, are presented. They show a remarkably higher accuracy when ANNs, in general, are used as opposed to simple lexical methods. RNNs, in particular, also show higher coverage and accuracy than FNNs. Copyright © 2003 John Wiley & Sons, Ltd.

This paper introduces new recognition strategies based on reasoning about results obtained with different Language Models (LMs). Strategies are built following the conjecture that the consensus among the results obtained with different models gives rise to different situations in which hypothesized sentences have different word error rates (WER) and may be further processed with other LMs. New LMs are built by data augmentation using ideas from latent semantic analysis and trigram analogy. Situations are defined by expressing the consensus among the recognition results produced with different LMs and by the amount of unobserved trigrams in the hypothesized sentence. The diagnostic power of the use of observed trigrams or their corresponding class trigrams is compared with that of situations based on values of sentence posterior probabilities. In order to avoid or correct errors due to syntactic inconsistence of the recognized sentence, automata, obtained by explanation-based learning, are introduced and used in certain conditions. Semantic Classification Trees are introduced to provide sentence patterns expressing constraints of long distance syntactic coherence. Results on a dialogue corpus provided by France Telecom R&D have shown that starting with a WER of 21.87% on a test set of 1422 sentences, it is possible to subdivide the sentences into three sets characterized by automatically recognized situations. The first one has a coverage of 68% with a WER of 7.44%. The second one has various types of sentences with a WER around 20%. The third one contains 13% of the sentences that should be rejected with a WER around 49%. The second set characterizes sentences that should be processed with particular care by the dialogue interpreter with the possibility of asking a confirmation from the user.

This paper deals with the difficult task of recognition of a large vocabulary of proper names in a directory assistance application. Rather than augmenting the lexicon with alternate pronunciations. which is unsuitable for very large vocabularies of proper names, a class of distortions of the canonical form is used as knowledge source (KS) for a new evaluation of the N best hypotheses generated in a first recognition phase, in which new probability distributions are used. The KS is the result of applying constraints inspired by speech science to distortions obtained by automatic learning. Experiments on a very large French directory document the validity of the approach.

International audience

Compares the performance of an operational automatic speech recognition system when Mel frequency-scaled cepstral coefficients (MFCCs), J-Rasta perceptual linear prediction coefficients (J-Rasta PLP) and energies from a multi resolution analysis (MRA) tree of filters are used as input features to a hybrid system consisting of a neural network (NN) which provides observation probabilities for a network of hidden Markov models (HMM). Furthermore, the paper compares the performance of the system when various combinations of these features are used showing a WER reduction of 16% w.r.t. the use of J-Rasta PLP coefficients, when J-Rasta PLP coefficients are combined with the energies computed at the output of the leaves of an MRA filter tree. Such a combination is practically feasible thanks to the NN architecture used in the system. Recognition is performed without any language model on a very large test set including many speakers uttering proper names from different locations of the Italian public telephone network.

A method is presented for augmenting word n-gram counts in a matrix which represents a 2-gram language model (LM) This method is based on numerical distances in a reduced space obtained by singular value decomposition. Rescoring word lattices in a spoken dialogue application using an LM containing augmented counts has lead to a word error rate (WER) reduction of 6.5%. By further interpolating augmented counts with the counts extracted from a very large newspaper corpus, but only for selected histories, a total WER reduction of 11.7% was obtained. We show that this approach gives better results than a global count interpolation for all histories of the LM.

Within the framework of Natural Spoken Dialogue systems, this paper describes a method for dynamically adapting a Language Model (LM) to the dialogue states detected. This LM combines a standard n-gram model with Stochastic Finite State Automata (SFSAs). During the training process, the sentence corpus used to train the LM is split into several hierarchical clusters in a 2-step process which involves both explicit knowledge and statistical criteria. All the clusters are stored in a binary tree where the whole corpus is attached to the root node. Each level of the tree corresponds to a higher specialization of the sub-corpora attached to the nodes and each node corresponds to a different dialogue state. From the same sentence corpus, SFSAs are extracted in order to model longer contexts than the ones used in the standard n-gram model. A set of SFSAs is attached to each node of the tree as well as a sub-LM which combines a bigram trained on the sub-corpus of the node and the SFSAs selected. A first decoding process calculates a word-graph as well as a first sentence hypothesis. This first hypothesis will be used to find the optimal node in the LM tree. Then, a rescoring process of the word graph using the LM attached to the node selected is performed. By adapting the LM to the dialogue state detected, we show a statistically significant gain in WER on a dialogue corpus collected by France Telecom R&D .

Techniques for automatic query expansion from top retrieved documents have shown promise for improving retrieval effectiveness on large collections; however, they often rely on an empirical ground, and there is a shortage of cross-system comparisons. Using ideas from Information Theory, we present a computationally simple and theoretically justified method for assigning scores to candidate expansion terms. Such scores are used to select and weight expansion terms within Rocchio's framework for query reweigthing. We compare ranking with information-theoretic query expansion versus ranking with other query expansion techniques, showing that the former achieves better retrieval effectiveness on several performance measures. We also discuss the effect on retrieval effectiveness of the main parameters involved in automatic query expansion, such as data sparseness, query difficulty, number of selected documents, and number of selected terms, pointing out interesting relationships.

The paper provides a theoretical justification that gravity centers (GC) in frequency bands computed from zero-crossing information are far more robust to additive telephone noise than GCs computed from FFT spectra. Experiments on two different corpora confirm the theoretical results when GCs are added to standard mel frequency-scaled cepstral coefficients (MFCC) and their time derivatives. A 20.1% word error reduction is observed on a large telephone corpus of Italian cities, with an average signal-to-noise ratio (SNR) of 15 dB, if GCs are computed from zero-crossings, while performance deteriorates when GCs are computed from FFT spectra.

This paper describes a method for building statistical Language Models (LMs) dedicated to specific dialogue situations. The architecture of the speech recognition system proposed uses several LMs. The first stage of this system, consists of producing a word-lattice from a given sentence uttered by a speaker. A general LM calculates a sentence-hypothesis. Then, in a second stage, the system chooses a specialized LM according to the word-lattice and the previous hypothesis. Another decoding process is performed using this specialized LM in order to produce a new sentence- hypothesis. Finally, a decision-module processes these two hypotheses in order to assign three confidence levels to the sentence-hypothesis produced. These confidence levels can be used by the dialogue manager in order to improve the dialogue, by asking a confirmation to the speaker when a sentence is labeled ambiguous. This research is supported by France Telecom's R&D under the contract 971B427.

The paper describes a system, in JAVA for written and visual scenario generation used to collect speech corpora in the framework of a tourism information system. Experimental evidence shows that the corpus generated with visual scenarios has a higher perplexity and a richer vocabulary than the corpus generated using the same conceptual derivations to produce textual scenarios. Furthermore, there is evidence that textual scenarios influence speakers in the choice of the lexicon used to express the concepts more than visual scenarios.

This paper presents a new database collected from a bilingual speakers set (49), in two different languages: Spanish and Catalan.Phonetically there are significative differences between both languages.These differences have let us to establish several conclusions on the relevance of language in speaker recognition, using two methods: vector quantization and covariance matrices.

Maximum a posteriori adaptation method combines the prior knowledge with adaptation data from a new speaker, which has a nice asymptotical property, but has a slow adaptation rate for not modifying unseen models. In a strictly Bayesian approach, prior parameters are assumed known, based on common or subjective knowledge. But a practical solution is to adopt an empirical Bayesian approach, where the prior parameters are estimated directly from training speech data itself. So there is a problem of mismatches between training and testing conditions. In this paper we propose a prior parameter transformation (PPT) adaptation approach that transforms the prior parameters to be more representative of the new speaker. It can influence unseen models by tying prior parameter transformations across different models according to amount of adaptation data available. Based on the improved prior information better model parameters can be obtained even with small amount of adaptation data.

International audience

Provides an abstract for each of the exprt presentations and a brief professional biography of each presenter. The complete presentations were not made available for publication as part of the conference proceedings.

A search technique incorporating the automatic modeling of lexical variability is introduced for medium or large-vocabulary speaker-independent speech recognition. Current state-of-art systems depend on being able to model the entire language based on acoustic features and the constraints of syntax or inter-word probabilities. These methods often fail in the presence of multiple speakers, new vocabulary, noise, and spontaneous speech phenomena. A new approach for word hypothesization is proposed, based on an acoustic-phonetic unit called the pseudo-syllable segment. An algorithm is described for transforming a sequence of syllables into words. Techniques are suggested for controlling the accuracy of the syllabic hypothesis set, and learning the phonotactics of syllables automatically in a statistical framework.

We propose multilevel semantic classification trees to combine different information sources for predicting speech events (e.g. word chains, phrases, etc.). Traditionally in speech recognition systems these information sources (acoustic evidence, language model) are calculated independently and combined via Bayes rule. The proposed approach allows one to combine sources of different types it is no longer necessary for each source to yield a probability. Moreover the tree can look at several information sources simultaneously. The approach is demonstrated for the prediction of prosodically marked phrase boundaries, combining information about the spoken word chain, word category information, prosodic parameters, and the result of a neural network predicting the boundary on the basis of acoustic-prosodic features. The recognition rates of up to 90% for the two class problem boundary vs. no boundary are already comparable to results achieved with the above mentioned Bayes rule approach that combines the acoustic classifier with a 5-gram categorical language model. This is remarkable, since so far only a small set of questions combining information from different sources have been implemented.

New techniques are introduced to manage speaker variability in large-vocabulary speaker-independent automatic speech recognition systems. Stochastic inference networks are used to model speaker intentions and automatically derive multiple phonemic transcriptions of words. The statistical inferences are generalized and managed by a justification maintenance system module integrated with the speech module, to dynamically adapt the system's performance to the variability of multi-speaker use. Additionally, lexical access to the database of phonemic transcriptions is mediated in multi-pass search with a new method of rejection based on beam-search derived lattices of pseudo-syllabic segments.

In the domain of mobile robotic task execution under dialogue control, a primary goal is to identify the task target which is specified by a natural language description. A number of concepts are expressed in the user spoken language by vague terms like "the big box" and "very close to the door". We use fuzzy logic to map these vague terms onto the quantitative data collected by system sensors. Fuzziness may cause uncertainty in interpretation and, in particular, in understanding references. This uncertainty is abated by collecting additional information through queries to the user and autonomous sensing. Entropy is used to select the queries having the greatest discriminatory power among referent candidates. In addition, we examine the trade-off between querying, sensing and uncertainty. A framework to deal with each of these issues has been developed and is presented.

The definition and use of multi-valued constant propagation analysis (MVCP), which is an extension of simple constant propagation analysis, is presented in this paper in the context of a user interface reengineering process. A brief description of the adopted COBOL/CICS user interface reengineering model, which makes use of an Abstract User Interface Description Language (AUIDL) to represent user interface structures and behavior, is also given. The experimental context is described and results are shown and discussed. Suggestions for further directions of research and investigation are also presented.

This article describes a new method for building a natural language understanding (NLU) system, in which the system's rules are learnt automatically from training data. The method has been applied to design of a speech understanding (SU) system. Designers of such systems rely increasingly on robust matchers to perform the task of extracting meaning from one or several word sequence hypotheses generated by a speech recognizer. We describe a new data structure, the semantic classification tree (SCT), that learns semantic rules from training data and can be a building block for robust matchers for NLU tasks. By reducing the need for handcoding and debugging a large number of rules, this approach facilitates rapid construction of an NLU system. In the case of an SU system, the rules learned by an SCT are highly resistant to errors by the speaker or by the speech recognizer because they depend on a small number of words in each utterance. Our work shows that semantic rules can be learned automatically from training data, yielding successful NLU for a realistic application.>

Previously (see ICASSP-93, vol.2, p.55, 1993 and Eurospeech 93, vol.2, p.1331, 1993) we described a speech understanding system called "CHANEL" with two components: a chart-based parser that analyzes semantically important word islands within an utterance; and a component based on "semantic classification trees" (SCTs) that builds the representation for the complete utterance. The construction of a natural-language understanding (NLU) system is a task that has traditionally required lavish expenditure of programmer-hours. By dividing the task in this way, we enabled many of the system's rules (those contained in the SCT component) to be learned automatically from training data, freeing human expertise to be applied where it is most effective. This paper describes recent improvements to both components of CHANEL, along with a new module that handles context-dependent utterances. The new version of CHANEL has a new use for SCTs: a special SCT decides whether a sentence is context-dependent or not.>

Complete automation of design recovery of large systems is a desirable but impractical goal due to complexity and size issues, so current research efforts focus on redocumentation and partial design recovery. Pattern matching lies at the center of any design recovery system. In the context of a larger project to develop an integrated reverse engineering environment, we are developing a framework for performing clone detection, code localization, and plan recognition. This paper discusses a plan localization and selection strategy based on a dynamic programming function that records the matching process and identifies parts of the plan and code fragment that are most "similar". Program features used for matching are currently based on data flow, control flow, and structural properties. The matching model uses a transition network and allows for the detection of insertions and deletions, and it is targeted for legacy C-based systems.>

An architecture for speech recognition is proposed, based on four stages: (1) recognition of the most likely phone sequence using centisecond Hidden Markov Models (HMMs); (2) phone-based lexical and syntactical forward decoding; (3) A phone-based backward pass, producing a Word Hypothesis Structure (WHS); (4) accurate rescoring of the search sub-space represented by the WHS using centisecond HMMs. Experiments carried out on two different tasks show that a recognizer based on the proposed four-stage architecture is able to achieve comparable performance respect to a classic one-stage recognizer. Experimental results show also that the same recognition performance can be obtained with WHSs built with this approach and WHSs built using centisecond HMMs with a potential speed-up, in WHS generation, proportional to the average phoneme duration in centiseconds.

The possibility of using stochastic context-free grammars (SCFG's) in language modeling (LM) has been considered previously. When these grammars are used, search can be directed by evaluation functions based on the probabilities that a SCFG generates a sentence, given only some words in it. Expressions for computing the evaluation function have been proposed by Jelinek and Lafferty (1991) for the recognition of word sequences in the case in which only the prefix of a sequence is known. Corazza et al. (1991) have proposed methods for probability computation in the more general case in which partial word sequences interleaved by gaps are known. This computation is too complex in practice unless the lengths of the gaps are known. This paper proposes a method for computing the probability of the best parse tree that can generate a sentence only part of which (consisting of islands and gaps) is known. This probability is the minimum possible, and thus the most informative, upper-bound that can be used in the evaluation function. The computation of the proposed upper-bound has cubic time complexity even if the lengths of the gaps are unknown. This makes possible the practical use of SCFG for driving interpretations of sentences in natural language processing.>

Hidden markov models (HMM's) are one of the most powerful speech recognition tools available today. Even so, the inadequacies of HMM's as a "correct" modeling framework for speech are well known. In this context, it is argued in this paper that the maximum mutual information estimation (MMIE) formulation for training is more appropriate than maximum likelihood estimation (MLE) for reducing the error rate. Corrective MMIE training is introduced. It is a very efficient new training algorithm which uses a modified version of a discrete reestimation formula recently proposed by Gopalakrishnan et al.( see IEEE Trans. Inform. Theory, Jan. 1991). Reestimation formulas are proposed for the case of diagonal Gaussian densities and their convergence properties are experimentally demonstrated. A description of how these formulas are integrated into our training algorithm is given. Using the MMIE framework for training, it is shown how weighting the contribution of different parameter sets in the computation of output probabilities introduces substantial recognition improvements. Using the TIDIGITS connected digit corpus, a large number of experiments are performed with the ideas, techniques, and algorithms presented in this paper. These experiments show that MMIE systematically provides substantial error rate reductions with respect to MLE alone and that, thanks to the new training techniques, these results can be obtained at an acceptable computational cost. The best results obtained in the experiments were 0.29% word error rate and 0.89% string error rate on the adult portion of the corpus.>

An extension of simple constant propagation analysis is presented in the context of the ongoing Macroscope project on the reengineering of user interfaces. Multi-valued constant propagation analysis (MVCP) is needed to extract user interface behavioral specifications from source code. Structural and behavioral specifications are used to generate new user interface code that will be integrated into the original system. The flow analysis aspects involved in the MVCP approach are described together with an overall view of the user interface reengineering project. The motivations and advantages of such an analysis method are presented.>

Stochastic signal models represent a powerful tool for automatic speech recognition. A particular type of stochastic modeling based on first-order hidden Markov models (HMMs), has been increasingly popular, because it has a solid theoretical basis and offers practical advantages. The authors extend the standard HMM theory to parallel hidden Markov models (PHMMs). The parallel model consists of two statistically related HMMs. This configuration has mixture densities of HMM observations whose weights can be made variable depending on the probability of other HMMs being in certain states. This allows one to dynamically adapt observation statistics to acoustic contexts. Some preliminary experiments have been carried out in order to compare the PHMMs with standard HMMs and the results are presented.>

The authors describe the latest developments by the speech research group at CRIM in speaker-independent connected digit recognition, using hidden Markov models (HMMs) trained with maximum mutual information estimation (MMIE). The work presented is a continuation of work previously described by the authors (see Proc. 1991 IEEE Inf. Conf. on Acoust. Speech and Sign. Process., pp.533-536). The main differences are: (1) use of the 20-kHz TI/NIST corpus available on CD-ROM (instead of the 10-kHz distribution tape), (2) use of word models (instead of sub-word units), (3) addition of second derivative parameters, and (4) a more elaborate training procedure for codebook exponents. The experiments described were all performed on the complete adult portion of the corpus. The baseline system, using discrete HMMs and MMIE, has a 0.67% word error rate and a 2.03% string error rate. The authors describe techniques that allowed them to improve greatly the recognition rate.>

The integration of multilayered and recurrent artificial neural networks (ANNs) with hidden Markov models (HMMs) is addressed. ANNs are suitable for approximating functions that compute new acoustic parameters, whereas HMMs have been proven successful at modeling the temporal structure of the speech signal. In the approach described, the ANN outputs constitute the sequence of observation vectors for the HMM. An algorithm is proposed for global optimization of all the parameters. Results on speaker-independent recognition experiments using this integrated ANN-HMM system on the TIMIT continuous speech database are reported.

The authors describe the latest development by the speech research group at CRIM (Centre de Recherche Informatique de Montreal) in speaker-independent connected digit recognition, using hidden Markov Models (HMMs) trained with maximum mutual information estimation, in conjunction with connectionist models. The experiments described were all done on the complete adult portion of the 10 kHz speaker-independent TI/NIST connected digit database. The baseline system, using discrete HMMs and maximum likelihood estimation, has a 98.6% word recognition rate and a 96.1% string recognition rate. The authors describe techniques that made it possible to improve greatly the baseline system recognition rate. The 99.3% recognition rate and 98.0% string recognition rate were obtained with a single model per unit using discrete HMMs and recurrent neural networks. Using semi-continuous HMMs with two models per digit (one for male and one for female speakers), a 99.5% word recognition rate and a 98.4% string recognition rate were achieved.>

An approach to designing dialogue systems, as applied to a robot with speech recognition capabilities, is described. One of the goals of this research is to develop probabilistic methods for dialogue systems analogous to those used in language modeling for dictation systems. These methods will use knowledge about dialogue structure, performance, syntax, and semantics to constrain the speech recognition task.>

Dans le cadre d'un decodeur acoustique-phonetique hybride (ANN/HMM), trois reseaux de neurones (ANNs) specialises ont ete developpes et evalues. Un de ces ANNs detecte le mode d'articulation. Les deux autres ANNs decrivent le signal en termes du lieu d'articulation. Un reseau est utilise pour classifier les consonnes nasales et plosives. Un autre est utilise pour la classification des fricatives Le design de ces reseaux est inspire par des connaissances acoustiques et phonetiques. Les entrees, la topologie et le codage des sorties ont ete optimises pour chacun des reseaux

In this paper we compare two hybrid acoustic-phonetic decoders based on Artificial Neural Networks (ANN). We evaluate them on the task of recognizing stop phones in continuous speech independently from the speaker. ANNs are well suited to perform detailed phonetic distinctions. In general, techniques based on Dynamic Programming (DP), in particular Hidden Markov Models (HMMs), have proven to be successful at modeling the temporal structure of the speech signal. In the approach described here, the ANN outputs constitute the sequence of observation vectors for the HMM. An algorithm is proposed for global optimization of all the parameters of the ANN/HMM decoder. Comparative experiments using this ANN/HMM hybrid decoder and another ANN-DP hybrid are reported for the TIMIT database. 1 Introduction Artificial Neural Networks (ANNs) effectively perform phonetic classification, but have not proven yet to model the temporal structure of the speech signal reasonably well [Lip89, Rob90, Ben90b...

Automatic speech understanding is the process of deriving a complete sentence interpretation of an acoustic signal. Stochastic language models can be of considerable help for the solution of this problem. In this paper we present a new method to apply stochastic context-free grammar models to the search of the most likely syntactic interpretation of the signal. The problem is discussed both theoretically and computationally. The analysis is also extended to cases in which the underlying parsing process is carried out in a bidirectional way. Introduction Automatic Speech Understanding (ASU) differs from Automatic Speech Recognition (ASR) because it has to produce a conceptual representation of a spoken message rather than a sequence of recognized words. The structure of a conceptual representation depends on the use it has to be made of it. Examples of actions based on conceptual representations are data-base query, inference, robot planning or replanning. In all these cases...

The authors describe an effort to adapt island-driven parsers to handle stochastic context-free grammars. These grammars could be used as language models (LMs) by a language processor (LP) to computer the probability of a linguistic interpretation. As different islands may compete for growth, it is important to compute the probability that an LM generates a sentence containing islands and gaps between them. Algorithms for computing these probabilities are introduced. The complexity of these algorithms is analyzed both from theoretical and practical points of view. It is shown that the computation of probabilities in the presence of gaps of unknown length requires the impractical solution of a nonlinear system of equations, whereas the computation of probabilities for cases with gaps containing a known number of unknown words has polynomial time complexity and is practically feasible. The use of the results obtained in automatic speech understanding systems is discussed.>

A hybrid coder is introduced for obtaining descriptions of speech patterns. This coder uses vector quantization (VQ) techniques on mel-scale cepstral coefficients and their derivatives together with a recurrent network (RN) for describing suprasegmental features of speech. The purpose of these features is to focus the search when hidden Markov models (HMMs) are used for speech unit or word models. Preliminary experiments of speaker-independent connected digit recognition show that using a hybrid coder based on a RN improves recognition performance.>

A model for spoken dialogue between an operator and a mobile robot in the procedural network formalism is described. The model is used to improve speech recognition by the robot and aid in the design of the robot's language generation component. Some probabilistic language models for understanding dialogue, which may be generally applicable to a variety of systems, are described.>

Speech-recognition systems must often decide between competing ways of breaking up the acoustic input into strings of words. Since the possible strings may be acoustically similar, a language model is required; given a word string, the model returns its linguistic probability. Several Markov language models are discussed. A novel kind of language model which reflects short-term patterns of word use by means of a cache component (analogous to cache memory in hardware terminology) is presented. The model also contains a 3g-gram component of the traditional type. The combined model and a pure 3g-gram model were tested on samples drawn from the Lancaster-Oslo/Bergen (LOB) corpus of English text. The relative performance of the two models is examined, and suggestions for the future improvements are made.>

Combining a structural or knowledge-based approach for describing speech units with neural networks capable of automatically learning relations between acoustic properties and speech units is investigated. The authors show how speech coding can be performed by sets of multilayer neural networks whose execution is decided by a data-driven strategy. Coding is based on phonetic properties characterizing a large population of speakers. Results on speaker-independent recognition of vowels using an ear model for preprocessing are reported.>

The Boltzmann machine algorithm and the error back propagation algorithm were used to learn to recognize the place of articulation of vowels (front, center or back), represented by a static description of spectral lines. The error rate is shown to depend on the coding. Results are comparable or better than those obtained by us on the same data using hidden Markov models. The authors also show a fault tolerant property of the neural nets, i.e. that the error on the test set increases slowly and gradually when an increasing number of nodes fail.>

The possibility of programming the execution of multilayered networks is considered. This planned execution produces a coded description of speech segments in terms of degrees of evidence of phonetic features. Continuous-parameter-density hidden Markov models are suggested for modeling the time evolution of feature degrees. Experimental results on plosive sounds, other consonants, and vowels are presented.>

This paper shows how a semiautomatic design of a speech recognition system can be done as a planning activity. Recognition performances are used for deciding plan refinement. Inductive learning is performed for setting action preconditions. Experimental results in the recognition of connected letters spoken by 100 speakers are presented.

In most of the existing Automatic Speech Recognition Systems which make use of Markov Models, the output of the Markov Chain are strings whose symbols belong to a finite alphabet and are generated sequentially over the time domain. We propose a Markov Model System in which symbols are substituted by Spectral Lines which are sequentially generated over the frequency domain. Each spectral line is represented by Continuous Distribution of Parameters. Switching from time-domain to frequency domain drastically reduces the number of states on the Markov chain and the use of continuous parameters eliminates quantization error completely. An application will be presented with experimental results in a multi-speaker environment.

This paper shows how a semi-automatic design of a speech recognition system can be done as a planning activity. Recognition performances are used for deciding plan refinement. Inductive learning is performed for setting action preconditions. Experimental results in the recognition of connectedly spoken letters pronounced by 70 new speakers are presented.

A network for performing multiplications of two two's complement numbers is proposed. The network can be implemented in a synchronous or an asynchronous way. If the factors to be multiplied have N bits, the area complexity of the network is O(N2) for practical values of N as in the case of cellular multipliers. Due to the design approach based on a recursive algorithm, a time complexity O(log N) is achieved.

A planning system for recognizing connected letters belonging to the set [E1 = (P,T,K,B,D,V,E,G,C,3) ] is described. Experimental results related to ten (five male and five female) speakers are reported.

A distributed rule-based system for automatic speech recognition is described. Acoustic property extraction and feature hypothesization are performed by the application of sequences of operators. These sequences, called plans, are executed by cooperative expert programs. Experimental results on the automatic segmentation and recognition of phrases, made of connected letters and digits, are described and discussed.

A new recursive algorithm for deriving the layout of parallel multipliers is presented. Based on this algorithm, a network for performing multiplications of two's complement numbers is proposed. The network can be implemented in a synchronous or an asynchronous way. If the factors to be multiplied have N bits, the area complexity of the network is O ( N 2 ) for practical values of N as in the case of cellular multipliers. Due to the design approach based on a recursive algorithm, a time complexity O (log N ) is achieved. It is shown how the structure can he pipelined with period complexity O (1) and used for single and double precision multiplication.

A rule based Expert System generating for phonetic hypotheses on continuously spoken French sentences is presented.

An Expert system is introduced for extracting acoustic cues from continuous speech. Part of the knowledge of such a system is a semantic Syntax-Directed Translation algorithm that segments continuous speech into Pseudo-Syllabic segments and generates hypotheses about phonetic features in each segment. Experimental results are provided about the performances of this system.

A parser for segmenting continuous speech into pseudo-syllabic nuclei is presented. It uses an Augmented Transition Network Grammar whose rules relate a minimum set of phonetic features useful for segmentation with acoustic features extracted from the speech spectra. Possible ambiguities are dealt with by assigning degrees of worthiness to the rules as well as to the acoustic features extracted from the speech spectra. The parser gives single syllabic hypotheses with non-fuzzy bounds when the input data are enough evident and lattices of syllabic hypotheses with well defined bounds when the input data are not enough precise.

A model for assigning phonetic and phonemic labels to speech segments is presented. The system executes fuzzy algorithms that assign degrees of worthiness to structured interpretations of syllabic segments extracted from the signal of a spoken sentence. The knowledge source is a series of syntactic rules whose syntactic categories are phonetic and phonemic features detected by a precategorical and a categorical classification of speech sounds. Rules inferred from experiments and results for male and female voices are presented.

The paper describes a computer program for the automatic recognition of stop consonants in continuous speech. The recognition is performed by a fuzzy algorithm that accounts for the imprecision of the features extracted and of the rules. The rules belong to a fuzzy grammar and account for coarticulation and contextual effects.

The paper deals with the representation of lexical knowledge in terms of phonetic and phonemic features. The representation is based on a set of frames containing preconditions and actions for each word. Preconditions are compounded in an automaton allowing a fast detection of hypothesizable words.

Some methodologles based on fuzzy set theory are proposed for emitting and verifying lexical hypotheses in a Speech Understanding System. Context-independent and context-dependent features are extracted from spectrogram patterns with a procedure that establishes a correspondence relation between a speech segment and its linguistic interpretation. The syllabic hypotheses are used for emitting lexical hypotheses by an algorithm that compensates for partial corruption of the syllabic information exploiting the lexical redundancy.

This paper presents a new method to study arithmetic combinatorial circuits. Using polynomial associated to the input-output sequences and to the systems it is possible to solve the problem of minimization of the number of the component blocks. Particularly, the important case of the multiple outputs elementary units can be treated. Applications of the introduced procedures to multiplier and to fast networks for performing convolution are presented.

This paper describes a special-purpose processor for use in performing various operations on sampled signals. The system is fast, flexible, and programmable for performing, in real time, operations such as fast Fourier transformation (FFT) and digital filtering of acoustically derived signals.

An algorithm for performing signed-number multiplications using positive-number full multipliers is presented. It leads to very simple input-and output-correcting networks which are cellular iterative and introduce negligible additional delay.