Understanding how language is processed and represented cognitively increasingly involves the use of specialized language models. Yet, because most models are predominantly trained on written text, they struggle to reflect the characteristics of language as it naturally unfolds in spoken interaction. This gap limits their capabilities in capturing features typical of spontaneous speech, such as repetitions, feedback cues, and hesitations. In this work, we introduce linguistically motivated evaluation metrics designed to target these specific spoken-language phenomena. We apply them to analyse outputs from language models fine-tuned on spoken English and French, comparing their behaviour statistically with human dialogue corpora. Our findings highlight the value of these metrics for assessing the degree to which model-generated utterances resemble authentic human conversation.

In the realm of human communication, feedback plays a pivotal role in shaping the dynamics of conversations. This study delves into the multifaceted relationship between listener feedback, narration quality and distraction effects. We present an analysis conducted on the SMYLE corpus, specifically enriched for this study, where 30 dyads of participants engaged in 1) face-to-face storytelling (8.2 hours) followed by 2) a free conversation (7.8 hours). The storytelling task unfolds in two conditions, where a storyteller engages with either a “normal” or a “distracted” listener. Examining the feedback impact on storytellers, we discover a positive correlation between the frequency of specific feedback and the narration quality in normal conditions, providing an encouraging conclusion regarding the enhancement of interaction through specific feedback in distraction-free settings. In contrast, in distracted settings, a negative correlation emerges, suggesting that increased specific feedback may disrupt narration quality, underscoring the complexity of feedback dynamics in human communication. The contribution of this paper is twofold: first presenting a new and highly enriched resource for the analysis of discourse phenomena in controlled and normal conditions; second providing new results on feedback production, its form and its consequence on the discourse quality (with direct applications in human-machine interaction).

Interaction theories suggest that the emergence of mutual understanding between speakers in natural conversations depends on the construction of a shared knowledge base (common ground), but the details of which information and the circumstances under which it is memorized are not explained by any model. Previous works have looked at metrics derived from Information Theory to quantify the dynamics of information exchanged between participants, but do not provide an efficient way to locate information that will enter the common ground. We propose a new method based on the segmentation of a conversation into themes followed by their summarization. We then obtain the location of information transfers by computing the distance between the theme summary and the different utterances produced by a speaker. We evaluate two Large Language Models (LLMs) on this pipeline, on the French conversational corpus Paco-Cheese. More generally, we explore how the recent developments in the field of LLMs provide us with the means to implement these new methods and more generally support research into questions that usually heavily relies on human annotators.

In this article, we present a new open-source tool, called OUTCOME Virtual, to automatically replicate humans’ behaviors on virtual characters. The tool takes as input a human video and extracts action units, head movements, and gaze direction to animate a virtual character’s face and head with the synchronized speech in Unity. The tool is designed to be easily configured, in particular concerning the association between actions units and blend shapes, and with a user-friendly interface to select the appearance of virtual characters, the lip synchronization and the noise-correction option. The tool is illustrated in this paper with the generation of listener feedback.

This study investigates conversational feedback, that is, a listener's reaction in response to a speaker, a phenomenon which occurs in all natural interactions. Feedback depends on the main speaker's productions and in return supports the elaboration of the interaction. As a consequence, feedback production has a direct impact on the quality of the interaction. This paper examines all types of feedback, from generic to specific feedback, the latter of which has received less attention in the literature. We also present a fine-grained labeling system introducing two sub-types of specific feedback: positive/negative and given/new. Following a literature review on linguistic and machine learning perspectives highlighting the main issues in feedback prediction, we present a model based on a set of multimodal features which predicts the possible position of feedback and its type. This computational model makes it possible to precisely identify the different features in the speaker's production (morpho-syntactic, prosodic and mimo-gestural) which play a role in triggering feedback from the listener; the model also evaluates their relative importance. The main contribution of this study is twofold: we sought to improve 1/ the model's performance in comparison with other approaches relying on a small set of features, and 2/ the model's interpretability, in particular by investigating feature importance. By integrating all the different modalities as well as high-level features, our model is uniquely positioned to be applied to French corpora.

The specificities of Arabic parsing, such as agglutination, vocalization, and the relatively order-free words in Arabic sentences, remain major issues to consider. To promote its robustness, such parseing should define different types of constraints. Property Grammar (PG) formalism verifies the satisfiability of the constraints directly on the units of the structure, thanks to its properties (or relations). In this context, we propose to build a probabilistic parser with syntactic properties, using a PG, and we measure the production rules in terms of different implicit information and in particular the syntactic properties. We experimented with our parser on the treebank ATB, using the parsing algorithm CYK, and we obtained encouraging results. Our method is also automatic for implementation of most property types. Its generalization for other languages or corpus domains (using treebanks) could be a good perspective. Its combination with pre-trained models of BERT may also make our parser faster.

The mechanisms underlying human communication have been under investigation for decades, but the answer to how understanding between locutors emerges remains incomplete.Interaction theories suggest the development of a structural alignment between the speakers, allowing for the construction of a shared knowledge base (common ground).In this paper, we propose to apply metrics derived from information theory to quantify the amount of information exchanged between participants, the dynamics of information exchanges, to provide an objective way to measure the common ground instantiation.We focus on a corpus of free conversations augmented with prosodic segmentation and an expert annotation of thematic episodes.We show that during free conversations, the amount of information remains globally constant at the scale of the conversation, but varies depending on the thematic structuring, underlining the role of the speaker introducing the theme.We propose an original methodology applied to uncontrolled material.

We present in this paper the first natural conversation corpus recorded with all modalities and neuro-physiological signals. 5 dyads (10 participants) have been recorded three times, during three sessions (30mns each) with 4 days interval. During each session, audio and video are captured as well as the neural signal (EEG with Emotiv-EPOC) and the electro-physiological one (with Empatica-E4). This resource original in several respects. Technically, it is the first one gathering all these types of data in a natural conversation situation. Moreover, the recording of the same dyads at different periods opens the door to new longitudinal investigations such as the evolution of interlocutors’ alignment during the time. The paper situates this new type of resources with in the literature, presents the experimental setup and describes different annotations enriching the corpus.

Dialogue Act classification becomes a complex task when dealing with fine-grain labels.Many applications require such level of labelling, typically automatic dialogue systems.We present in this paper a 2-level classification technique, distinguishing between generic and specific dialogue acts (DA).This approach makes it possible to benefit from the very good accuracy of generic DA classification at the first level and proposes an efficient approach for specific DA, based on high-level linguistic features.Our results show the interest of involving such features into the classifiers, outperforming all other feature sets, in particular those classically used in DA classification.

We present in this paper a method for generating in real time a great variability of multimodal backchannel feedbacks, increasing the naturalness of IVAs. The originality of the approach lies in its capacity to generate all types of features into a unique loop.

This paper presents an original dataset of controlled interactions, focusing on the study of feedback items. It consists on recordings of different conversations between a doctor and a patient, played by actors. In this corpus, the patient is mainly a listener and produces different feedbacks, some of them being (voluntary) incongruent. Moreover, these conversations have been re-synthesized in a virtual reality context, in which the patient is played by an artificial agent. The final corpus is made of different movies of human-human conversations plus the same conversations replayed in a human-machine context, resulting in the first human-human/human-machine parallel corpus. The corpus is then enriched with different multimodal annotations at the verbal and non-verbal levels. Moreover, and this is the first dataset of this type, we have designed an experiment during which different participants had to watch the movies and give an evaluation of the interaction. During this task, we recorded participant’s brain signal. The Brain-IHM dataset is then conceived with a triple purpose: 1/ studying feedbacks by comparing congruent vs. incongruent feedbacks 2/ comparing human-human and human-machine production of feedbacks 3/ studying the brain basis of feedback perception.

A key challenge when studying human-agent interaction is the evaluation of user's experience. In virtual reality, this question is addressed by studying the sense of "presence'' and"co-presence'', generally assessed thanks to well-grounded subjective post-experience questionnaires. In this article, we aim at exploring behavioral measures of presence and co-presence by analyzing multimodal cues produced during an interaction both by the user and the virtual agent. In our study, we started from a corpus of human-agent interaction collected in a task-oriented context: a virtual environment aiming at training doctors to break bad news to a patient (played by a virtual agent). Based on this corpus, we have used machine learning algorithms to explore the possibility of predicting user's sense of presence and co-presence. In particular, we have applied and compared two techniques, Random forest and SVM, both showing very good results in predicting the level of presence and co-presence.

In the intelligent virtual agent domain, several machine learning models have been proposed to automatically determine the feedbacks of virtual agents during an interaction, using human-human interaction datasets as training corpora and most commonly based on verbal and prosodic features \citeMorency2010, Truong2010a. These approaches suppose an accurate system to automatically recognize speech and prosody. That makes the overall model's performance dependent on the individual performances of speech and prosody recognizers. As a consequence, one challenge remains to identify features that could be easily and accurately recognized during a human-machine interaction for predicting virtual agents' feedbacks in real time.

Abstract Most compositional distributional semantic models represent sentence meaning with a single vector. In this paper, we propose a structured distributional model (SDM) that combines word embeddings with formal semantics and is based on the assumption that sentences represent events and situations. The semantic representation of a sentence is a formal structure derived from discourse representation theory and containing distributional vectors. This structure is dynamically and incrementally built by integrating knowledge about events and their typical participants, as they are activated by lexical items. Event knowledge is modelled as a graph extracted from parsed corpora and encoding roles and relationships between participants that are represented as distributional vectors. SDM is grounded on extensive psycholinguistic research showing that generalized knowledge about events stored in semantic memory plays a key role in sentence comprehension.We evaluate SDMon two recently introduced compositionality data sets, and our results show that combining a simple compositionalmodel with event knowledge constantly improves performances, even with dif ferent types of word embeddings.

In human-agent interaction, one key challenge is the evaluation of the user's experience. In the virtual reality domain, the sense of presence and co-presence, reflecting the psychological immersion of the user, is generally assessed through well-grounded subjective post-experience questionnaires. In this article, we aim at presenting a new way to automatically predict the sense of presence and co-presence of a user at the end of an interaction based on specific verbal and non-verbal behavioral cues automatically computed. A random forest algorithm has been applied on a human-agent interaction corpus collected in the specific context of a virtual environment developed to train doctors to break bad news to a virtual patient. The performance of the models demonstrate the capacity to automatically and accurately predict the level of presence and co-presence, but also show the relevancy of the verbal and non-verbal behavioral cues as objective measures of presence.

In this paper, we introduce a new distributional method for modeling predicateargument thematic fit judgments.We use a syntax-based DSM to build a prototypical representation of verb-specific roles: for every verb, we extract the most salient second order contexts for each of its roles (i.e. the most salient dimensions of typical role fillers), and then we compute thematic fit as a weighted overlap between the top features of candidate fillers and role prototypes.Our experiments show that our method consistently outperforms a baseline re-implementing a state-of-theart system, and achieves better or comparable results to those reported in the literature for the other unsupervised systems.Moreover, it provides an explicit representation of the features characterizing verbspecific semantic roles.

This paper aims to establish a link between linguistics and fuzzy phenomena. Throughout the history, linguistics has been relying on discrete descriptions to explain Natural Language Processing. This fact has a negative impact in various areas of language and technology since linguistics rejects all the inputs which are out from a discrete rule. Although this paper might approach the subject more from a linguistic than from a mathematical point of view, we present the theoretical considerations and reasoning used in elaborating a formal characterization of fuzziness in natural language grammars. Property Grammars will be used as the formal theory in order to explain Natural Language fuzziness and variability.

Several studies on sentence processing suggest that the mental lexicon keeps track of the mutual expectations between words.Current DSMs, however, represent context words as separate features, thereby loosing important information for word expectations, such as word interrelations.In this paper, we present a DSM that addresses this issue by defining verb contexts as joint syntactic dependencies.We test our representation in a verb similarity task on two datasets, showing that joint contexts achieve performances comparable to single dependencies or even better.Moreover, they are able to overcome the data sparsity problem of joint feature spaces, in spite of the limited size of our training corpus.

International audience

International audience

This paper introduces a method combining spell checking and phonetic interpretation in order to automatically rewrite questions typed by dyslexic spellers. The method uses a finite state automata framework. Dysorthographics refers to incorrect word segmentation which usually causes classical spelling correc-tors fail. The specificities of the information retrieval context are that flexion errors have no impact since the sentences are lemmatised and filtered and that several hypothesis can be processed for one query. Our system is evaluated on questions collected with the help of an orthophonist. The word error rate on lemmatised sentences falls from 60% to 22% (falls to 0% on 43% of sentences).

We propose in this paper a method for quantifying sentence grammaticality. The approach based on Property Grammars, a constraint-based syntactic formalism, makes it possible to evaluate a grammaticality index for any kind of sentence, including ill-formed ones. We compare on a sample of sentences the grammaticality indices obtained from PG formalism and the acceptability judgements measured by means of a psycholinguistic analysis. The results show that the derived grammaticality index is a fairly good tracer of acceptability scores.

We propose in this paper a new approach for representing the prosody/syntax interface. We use for this a particular formalism, called Property Grammars, in which all information is represented by means of constraints. We show how alignment constraints can implement such an interface. One of the interests of these constraints, in comparison with other approaches such as optimality theory, is the possibility of representing different information at the same level (allowing then a parallel treatment of prosody and syntax). This discussion is illustrated with the example of dislocated constructions. 1.

The constitution of multi-level databases integrating, for example, both prosodic and morphosyntactic levels of representation presents a number of problems, some specific to the individual domains, and others concerning the integration of the two domains. It is argued that the formalism of annotation graphs provides an adequate solution to these problems, which can be implemented in an XML representation. It is further argued that a generic query language, DQL, currently being developed, will provide a satisfactory framework both for querying and for manipulating documents of this type.

We propose the use of two kinds of constraints in order to control the evaluation of ambiguous structures. The first ones concern the immediate context of the words. In case of ambiguity, these constraints form a network controlling an ambiguous area. The second kind of constraint relies on descriptions of the elementary trees that can be attached to the words. Such descriptions, called unary quasitrees, allow the representation of the dependency relations between the words. They form a proof net on which a filtering method can be applied in order to reduce the number of constraints. The elaboration of the constraint networks and its simplification rely on information available at the lexical level. So, the method presented here simplifies the control of the disambiguation without doing an actual parse and can be reused whatever the linguistic formalism.

Ambiguity is one of the main sources of complexity in natural language processing. We propose an original solution relying on the use of named disjunctions which are set of ordered formulae linked by a name. This representation is implemented within a constraint logic programming paradigm (allowing the use of underspecified structures) by means of lazy evaluation techniques. This approach avoids the expansion of the disjunctions into a normal form and allows one, in some cases, to compute a partial syntactic structure without disambiguating.