We investigate the role of prompt-based demonstrators in improving natural language generation for coaching-oriented dialogue systems in different languages. These systems present significant challenges due to their need for semantically accurate, goal-driven responses across diverse dialogue act taxonomies and languages. We define three types of prompt demonstrators, i.e., pairs of meaning representation-utterance, that include different degrees of specification in such meaning representation. We then fine-tune pretrained language models separately for four very different languages and evaluate how the specificity of these demonstrators affects the quality of the generated sentences. Our experiments show that more specific prompts lead to more coherent and accurate outputs, particularly for low-resource languages and small models. Additionally, we observe promising zero-shot performance with larger models, showing a complementary value of prompts. These results demonstrate that simple prompting strategies, combined with fine-tuning, can significantly improve output quality in complex dialogue generation tasks across languages.

The EMPATHIC project aimed to design an emotionally expressive virtual coach capable of engaging healthy seniors to improve well-being and promote independent aging. In particular, the system's human sensing capabilities allow for the perception of emotional states to provide a personalized experience. This paper outlines the development of the emotion expression recognition module of the virtual coach, encompassing data collection, annotation design, and a first methodological approach, all tailored to the project requirements. With the latter, we investigate the role of various modalities, individually and combined, for discrete emotion expression recognition in this context: speech from audio, and facial expressions, gaze, and head dynamics from video. The collected corpus includes users from Spain, France, and Norway, and was annotated separately for the audio and video channels with distinct emotional labels, allowing for a performance comparison across cultures and label types. Results confirm the informative power of the modalities studied for the emotional categories considered, with multimodal methods generally outperforming others (around 68% accuracy with audio labels and 72-74% with video labels). The findings are expected to contribute to the limited literature on emotion recognition applied to older adults in conversational human-machine interaction, and guide the development of future systems.

Dates are often treated as exact measures, yet their precision is limited. This applies not only to physical clocks but also to human memory. In conversations, we typically share timeframes relevant to the topic, not pinpoint accuracy. Precise birth times, for example, hold little conversational value compared to their biographical significance. Similarly, couples cherish their wedding day but might recall only the year they bought their house. This paper explores the advantages of incorporating fuzzy dates into personal knowledge graphs constructed through dialogues within the LifeLine feature of the “my eViTA” smartphone app. Fuzzy dates acknowledge the inherent imprecision of human memory and prioritize contextually relevant timeframes for a more natural and insightful knowledge representation.

Izaskun Fernandez, Cristina Aceta, Cristina Fernandez, Maria Ines Torres, Aitor Etxalar, Ariane Mendez, Maia Agirre, Manuel Torralbo, Arantza Del Pozo, Joseba Agirre, Egoitz Artetxe, Iker Altuna. Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2024.

Alain Vazquez Risco, Angela Maria Ramirez, Neha Pullabhotla, Nan Qiang, Haoran Zhang, Marilyn Walker, Maria Ines Torres. Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2024.

Following the success of Natural Language Processing (NLP) transformers pretrained via self-supervised learning, similar models have been proposed recently for speech processing such as Wav2Vec2, HuBERT and UniSpeech-SAT. An interesting yet unexplored area of application of these models is Spoken Dialogue Systems, where the users' audio signals are typically just mapped to word-level features derived from an Automatic Speech Recogniser (ASR), and then processed using NLP techniques to generate system responses. This paper reports a comprehensive comparison of dialogue policies trained using ASR-based transcriptions and extended with the aforementioned audio processing transformers in the DSTC2 task. Whilst our dialogue policies are trained with supervised and policy-based deep reinforcement learning, they are assessed using both automatic task completion metrics and a human evaluation. Our results reveal that using audio embeddings is more beneficial than detrimental in most of our trained dialogue policies, and that the benefits are stronger for supervised learning than reinforcement learning.

In this work we develop a fully data driven conversational agent capable of carrying out motivational coaching sessions in Spanish, French, Norwegian, and English. Unlike the majority of coaching, and in general well-being related conversational agents that can be found in the literature, ours is not designed by hand-crafted rules. Instead, we directly model the coaching strategy of professionals with end users. To this end, we gather a set of virtual coaching sessions through a Wizard of Oz platform, and apply state of the art Natural Language Processing techniques. We employ a transfer learning approach, pretraining GPT2 neural language models and fine-tuning them on our corpus. However, since these only take as input a local dialogue history, a simple fine-tuning procedure is not capable of modeling the long-term dialogue strategies that appear in coaching sessions. To alleviate this issue, we first propose to learn dialogue phase and scenario embeddings in the fine-tuning stage. These indicate to the model at which part of the dialogue it is and which kind of coaching session it is carrying out. Second, we develop a global deep learning system which controls the long-term structure of the dialogue. We also show that this global module can be used to visualize and interpret the decisions taken by the the conversational agent, and that the learnt representations are comparable to dialogue acts. Automatic and human evaluation show that our proposals serve to improve the baseline models. Finally, interaction experiments with coaching experts indicate that the system is usable and gives rise to positive emotions in Spanish, French and English, while the results in Norwegian point out that there is still work to be done in fully data driven approaches with very low resource languages.

Neural transformer architectures have gained a lot of interest for text-based dialogue management in the last few years. They have shown high learning capabilities for open domain dialogue with huge amounts of data and also for domain adaptation in task-oriented setups. But the potential benefits of exploiting the users' audio signal have rarely been explored in such frameworks. In this work, we combine text dialogue history representations generated by a GPT-2 model with audio embeddings obtained by the recently released Wav2Vec2 transformer model. We jointly fine-tune these models to learn dialogue policies via supervised learning and two policy gradient-based reinforcement learning algorithms. Our experimental results, using the DSTC2 dataset and a simulated user model capable of sampling audio turns, reveal that audio embeddings lead to overall higher task success (than without using audio embeddings) with statistically significant results across evaluation metrics and training algorithms.

The main objective of the EMPATHIC project has been the design and development of a virtual coach to engage the healthy-senior user and to enhance well-being through awareness of personal status. The EMPATHIC approach addresses this objective through multimodal interactions supported by the GROW coaching model. The paper summarizes the main components of the EMPATHIC Virtual Coach (EMPATHIC-VC) and introduces a demonstration of the coaching sessions in selected scenarios.

The present investigation aims at assessing elders' ability to decode facial emotional expressions conveyed by differently aged people in order to confirm (or disconfirm) the appropriateness of the “own age bias” theory, as well as investigate effects of different ages and different emotional categories. The study, involves 44 healthy elders (23 females), aged 65+ (mean age=75.09; SD=±7.9) which were requested to label 76 pictures depicting elders, middle-aged and young women and men displaying the six facial emotional expressions of disgust, anger, fear, sadness, happiness and neutrality. Results show a complex pattern of influences that calls for more deep investigations on the features to be accounted by providing socially and emotionally believable interfaces of effective and efficient algorithms to detect and decode their users' emotional facial expressions.

T e-Coaches are promising intelligent systems that aims at supporting human everyday life, dispatching advices through different interfaces, such as apps, conversational interfaces and augmented reality interfaces. This workshop aims at exploring how e-coaches might benefit from spatially and time-multiplexed interfaces and from different communication modalities (e.g., text, visual, audio, etc.) according to the context of the interaction.

The presented study investigated the preferences of seniors towards artificial avatars showing personality both from a pragmatic and a hedonic point of view. Also, preferences for technological devices were considered. The involved participants were 45 adults (20 female) aged 65+ years in good health. They were asked to watch video clips of 4 agents (two males and two females) showing different personality traits (i.e. angry, depressed, joyful, and practical), and subsequently had to complete a questionnaire. Subjects were not informed about an avatar’s personality and not openly interviewed regarding this subject. Rather, the administered questionnaire was devoted to test their perception of agents and whether such complies with the intended characteristics. Results show that subjects prefer female agents with a positive personality (joyful and practical) on both pragmatic and hedonic dimensions of the interactive system.

Incremental Dialog Processing (IDP) enables Spoken Dialog Systems to gradually process minimal units of user speech in order to give the user an early system response. In this paper, we present an application of IDP that shows its effectiveness in a task-oriented dialog system. We have implemented an IDP strategy and deployed it for one month on a real-user system. We compared the resulting dialogs with dialogs produced over the previous month without IDP. Results show that the incremental strategy significantly improved system performance by eliminating long and often off-task utterances that generally produce poor speech recognition results. User behavior is also affected; the user tends to shorten utterances after being interrupted by the system.

[EN] The new Interactive Pattern Recognition (IPR) framework has \nbeen proposed to deal with human-machine interaction. In this \ncontext a new formulation has been recently defined to represent \na Spoken Dialogue System as an IPR problem. In this work this \nformulation is applied to define graphical models that deal with \nSpoken Dialogue Systems. The definition of both a Dialogue \nManager and a User Model are shown and the estimation of \nthe parameters and smoothing techniques are presented in the \npaper. These models were evaluated in a dialogue generation \ntask on two very different corpora: Dihana corpus consisting \nof Spanish spoken dialogues acquired with the Wizard of Oz \ntechnique and Let’s Go corpus consisting of spoken dialogues \nin English between real users and the Ravenclaw dialogue manager \ndeveloped by CMU. The results obtained show that original \nand simulated dialogues exhibited very similar behaviours, \nthus demonstrating the learning capacity of the proposed models \nin both a controlled Wizard of Oz task and a spoken dialogue \nsystem that interacts with real users. This formulation can then \nbe considered as a promising framework to deal with Spoken \nDialogue Systems.

Stochastic finite-state transducers constitute a type of word-based models that allow an easy integration with acoustic model for speech translation. The aim of this work is to develop a novel approach to phrase-based statistical finite-state transducers. In this work, we explore the use of linguistically motivated phrases to build phrase-based models. The proposed phrase-based transducer has been tested and compared to a word-based equivalent machine, yielding promising results in the reported preliminary text and speech translation experiments.

This paper evaluates different approaches on speech to sign language machine translation. The framework of the application focuses on assisting deaf people to apply for the passport or related information. In this context, the main aim is to automatically translate the spontaneous speech, uttered by an officer, into Spanish Sign Language (SSL). In order to get the best translation quality, three alternative techniques have been evaluated: a rule-based approach, a phrase-based statistical approach, and a approach that makes use of stochastic finite state transducers. The best speech translation experiments have reported a 32.0 % SER (Sign Error Rate) and a 7.1 BLEU (BiLingual Evaluation Understudy) including speech recognition errors.

Spontaneous speech is full of acoustic disfluencies that rarely appear in read or laboratory speech. A very simple and straightforward approach is presented, in which acoustic disfluences are modelled by augmenting the inventory of sublexical units, which originally consisted of 23 context independent phones plus a special unit for silent pauses. This set was augmented with 12 additional units accounting for lengthenings of sounds, filled pauses and noises. Two speech databases, both in Spanish, were used in the experiments. A phonetically balanced database was used for initializing the acoustic models. A spontaneous speech database consisting of 227 dialogues was used both for training and testing purposes. Recognition rates, in terms of acoustic-phonetic accuracy and word accuracy, with and without filtering acoustic disfluencies prior to alignments, were obtained to evaluate the contribution of these models to speech recognition. Also, some specific but significant examples were explored and discussed. Experimental results showed that using explicit models of acoustic disfluencies clearly improved the performance of a spontaneous speech recognition system.

Continuous Speech Recognition systems require a Language Model (LM) to represent the syntactic constraints of the language. In LMs development a smoothing technique needs to be applied to also consider events not represented in the training corpus. In this work, several back-off smoothing approaches have been compared: classical discounting-distribution schema including Witten-Bell, Absolute and Linear discounting and a new proposal, the Delimited discounting. Delimited discounting deals with the Turing discounting problems

This paper describes the work carried out to select the most suitable set of Sublexical Units for Continuous Speech Recognition of Basque. Even if there are several dialects in Basque, only one of them has been used to choose the preliminary set of sounds. Bearing in mind this aim, a wide experimentation has been carried out to select Context Independent Phone-Like Units. Then, in order to obtain robust acoustic models for the language, the units have been evaluated with most of the dialectal variants of Basque. Finally, Decision-Trees based Context Dependent Sublexical Units are selected. For building the trees the classical methodology of Bahl and the efficient Growing and Pruning algorithm have been used. 1.

ABSTRACT 1 The selection of appropriated Lexical Units is an important issue in the Language Model (LM) generation. Word has been used classically as unit in most of the Continuous Speech Recognition systems. However, during the last years proposals of non-word units have begun to appear. Since Basque is an agglutinative language with a certain structure inside the word, the nonword units could be an adequate option. In this work, a statistical analysis of the morphological structure of Basque has been carried out. This analysis shows a slight increment of the rates of confusion in Continuous Speech Recognition Systems due to the great increment of acoustically similar and short units. Finally several proposals of Lexical Units are analyzed to deal with the problem. 1.

Continuous Speech Recognition (CSR) systems require acoustic models to represent the characteristics of the acoustic signal and Language Models (LM) to represent the syntactic constraints of the language. Both acoustic and LM probability distributions are usually independently obtained and evaluated. Then, the respective “best ” models are selected to be integrated in the CSR systems. But, in this paper it was proved that the use of more accurate acoustic models (for example the use of semicontinuous models instead of discrete ones or the use of a bigger number of then representing a more complete set of sub-lexical units) didn’t always mean a better performance of the integrated system because the acoustic improvements were softened when the LM probabilities were applied. This experimental evaluation was carried out over a Spanish speech application task. 1

A syntactic approach of the well-known N-grams models, the K-testable language in the strict sense (K-TSS), is used in this work to be integrated in a continuous speech recognition (CSR) system. The use of smoothed K-TSS regular grammars allowed to obtain a deterministic stochastic finite state automaton (SFSA) integrating K k-TSS models into a self-contained model. An efficient representation of the whole model in a simple array of adequate size is proposed. This structure can be easily handled at decoding time by a simple search function through the array. This formulation strongly reduced the number of parameters to be managed and thus the computing complexity of the model. An experimental evaluation of the proposed SFSA representation was carried out over an Spanish recognition task. These experiments showed important memory saving to allocate K-TSS language models, more important for higher values of K. They also showed that the decoding time did not meaningfully increased when K did. The lower word error rates for the Spanish task tested were achieved for K=4 and 5. As a consequence the ability of this syntactic approach of the N-grams to be well integrated in a CSR system, even for high values of K, has been established.

This paper presents a new system for the continuous speech recognition of Spanish, integrating previous works in the fields of acoustic-phonetic decoding and language modelling. Acoustic and language models separately trained with speech and text samples, respectivelyare integrated into one single automaton, and their probabilities combined according to a standard beam search procedure. Two key issues were to adequately adjust the beam parameter and the weight affecting the language model probabilities. For the implementation, a client-server arquitecture was selected, due to the desirable working scene where one or more simple machines in the client side make the speech analysis task, and a more powerful workstation in the server side looks for the best sentence hypotheses. Preliminary experimentation gives promising results with around 90% word recognition rates in a medium size word speech recognition task.

Speaker Verification and Utterance Verification are examples of techniques that can be used for Speaker Authentication purposes. Speaker Verification consists of accepting or rejecting the claimed identity of a speaker by processing samples of his/her voice. Utterance Verification systems make use of a set of speaker-independent speech models to recognize a certain utterance and decide whether a speaker has uttered it or not. If the utterances consist of passwords, this technique can be used for identity verification purposes. Up to now, both techniques have been used separately. We propose an architecture consisting of both systems working in parallel with a novel output combination technique. Thus, a Neural Network is designed to learn from the data how to balance the influence of both outputs in order to jointly minimize the False Acceptance and False Rejection rates. The better performance of this architecture is compared with those of the individual systems in an over the phone speaker recognition task.

The class of K-testable languages in the strict sense (K-TLSS) is a subclass of the regular languages. Stochastic K-TLSS language models describe the same probability distribution as N-gram models, and smoothing techniques (backoff-like methods) can be applied efficiently. Once we have a set of k-TLSS models (k=1...K) and a smoothing technique that specifically fits them, we propose an integration into a unique self-contained [K-TLSS(S)] model which embeds the smoothing within the topology, allowing extremely simple parsing procedures. To build this model, we designed a more general syntactic mechanism that we call a "stochastic deterministic finite state automaton with recursive transitions". The topology of the new K-TLSS(S) model allows an easy pruning procedure. Pruned K-TLSS(S) models give probability distributions that are equivalent to variable-length N-gram models. Experimental results give as a conclusion that the effect of a small pruning is always positive.

Continuous Speech Recognition Systems (CSR) usually include large sets of context dependent units to model contextual variations in the pronunciation of phones. The goal of this work was to obtain adequate sets of sub-lexical models by using acoustic information but excluding any previous phonological knowledge. At each iteration of a classical Viterbi training scheme each acoustic model was split into a set of more accurate models. This approach was evaluated over a Spanish acoustic phonetic decoding task. The experimental results showed that this approach produces similar recognition rates than classical triphones.

N-grams have been extensively and successfully used for language modelling in continuous speech recognition tasks. On the other hand, it has been shown that k-testable stochastic languages (k-TS) are strictly equivalent to N-grams. A major problem to be solved when using a language model is the estimation of the probabilities of events not represented in the training corpus, i.e. unseen events. The aim of this work is to improve other well established smoothing procedures by interpolating models with different levels of complexity (quality weighted interpolation-QWI). The effect of QWI was experimentally evaluated over a set of back-off smoothed k-TS language models. These experiments were carried out over several corpora using the test-set perplexity as an evaluation criterion. In all the cases the introduction of QWI resulted in a reduction of the test-set perplexity.

The design of current acoustic-phonetic decoders for a specific language involves the selection of an adequate set of sublexical units, and a choice of the mathematical framework for modelling the corresponding units. In this work, the baseline chosen for continuous Spanish speech consists of 23 sublexical units that roughly correspond to the 24 Spanish phonemes. The process of selection of such a baseline was based on language phonetic criteria and some experiments with an available speech corpora. On the other hand, two types of models were chosen for this work, conventional Hidden Markov Models and Inferred Stochastic Regular Grammars. With these two choices we could compare classical Hidden Markov modelling where the structure of a unit-model is deductively supplied, with Grammatical Inference modelling where the baseforms of model-units are automatically generated from training samples. The best speaker-independent phone recognition rate was 64% for the first type of modelling, and 66% for the second type.