Language-based interactive AI systems are required to provide adaptive explanations in order to create transparency and interpretability. A cornerstone ability for this is to respond to user questions. We target the problem of Natural Language Question Understanding (NLQU), which refers to interpreting a user question and mapping it to semantic representations (items in a structured knowledge base) that are relevant to address the underlying user’s knowledge gap and hence to answer the question. In contrast to pure Q&A systems that aim to directly map questions to answers, NLQU enables a dialog agent to employ different explanation strategies, including explaining prerequisite knowledge, adapting explanation speed, or identifying and repairing misunderstandings. This paper explores the use of modern Large Language Models (LLMs) along with Retrieval Augmented Generation (RAG) for NLQU. We present a RAG-based NLQU component, evaluate different approaches against a synthetic dataset, and test the final component on a natural language question dataset from a human-human explanation study. Different LLM models, prompts, and hyperparameters are tested and compared to a baseline method.

Generating accurate and realistic virtual human movements in real-time is of high importance for a variety of applications in computer graphics, interactive virtual environments, robotics, and biomechanics.This paper introduces a novel real-time inverse kinematics (IK) solver specifically designed for realistic human-like movement generation.Leveraging the automatic differentiation and just-in-time compilation of TensorFlow, the proposed solver efficiently handles complex articulated human skeletons with high degrees of freedom.By treating forward and inverse kinematics as differentiable operations, our method effectively addresses common challenges such as error accumulation and complicated joint limits in multi-constrained problems, which are critical for realistic human motion modeling.We demonstrate the solver's effectiveness on the SMPLX human skeleton model, evaluating its performance against widely used iterative-based IK algorithms, like Cyclic Coordinate Descent (CCD), FABRIK, and the nonlinear optimization algorithm IPOPT.Our experiments cover both simple end-effector tasks and sophisticated, multi-constrained problems with realistic joint limits.Results indicate that our IK solver achieves real-time performance, exhibiting rapid convergence, minimal computational overhead per iteration, and improved success rates compared to existing methods.The project code is available at https://github.com/hvoss-techfak/TF-JAX-IK

The ability to generate explanations that are understood by explainees is the quintessence of explainable artificial intelligence. Since understanding depends on the explainee’s background and needs, recent research focused on co-constructive explanation dialogues, where an explainer continuously monitors the explainee’s understanding and adapts their explanations dynamically. We investigate the ability of large language models (LLMs) to engage as explainers in co-constructive explanation dialogues. In particular, we present a user study in which explainees interact with an LLM in two settings, one of which involves the LLM being instructed to explain a topic co-constructively. We evaluate the explainees’ understanding before and after the dialogue, as well as their perception of the LLMs’ co-constructive behavior. Our results suggest that LLMs show some co-constructive behaviors, such as asking verification questions, that foster the explainees’ engagement and can improve understanding of a topic. However, their ability to effectively monitor the current understanding and scaffold the explanations accordingly remains limited.

This study explores how people interact in dyadic social eXtended Reality (XR), focusing on two main factors: the animation type of a conversation partner's avatar and how immersed the user feels in the virtual environment. Specifically, we investigate how 1) idle behavior, 2) AI-generated gestures, and 3) motion-captured movements from a confederate (a controlled partner in the study) influence the quality of conversation and how that partner is perceived. We examined these effects in both symmetric interactions (where both participants use VR headsets and controllers) and asymmetric interactions (where one participant uses a desktop setup). We developed a social XR platform that supports asymmetric device configurations to provide varying levels of immersion. The platform also supports a modular avatar animation system providing idle behavior, real-time AI-generated co-speech gestures, and full-body motion capture. Using a 2×3 mixed design with 39 participants, we measured users' sense of spatial presence, their perception of the confederate, and the overall conversation quality. Our results show that users who were more immersed felt a stronger sense of presence and viewed their partner as more human-like and believable. Surprisingly, however, the type of avatar animation did not significantly affect conversation quality or how the partner was perceived. Participants often reported focusing more on what was said rather than how the avatar moved.

The ICMI 2024 workshop on “Multimodal Co-Construction of Explanations with XAI” bridges the fields of Explainable Artificial Intelligence (XAI) and Multimodal Interaction, focusing on the recent perspective that effective AI explanations should be dynamically co-constructed through interactive, social processes involving both the explainer and the explainee. By framing XAI explanations as a multimodal, interactive co-construction challenge, the workshop seeks to explore how these two fields can collaboratively address the complexities of creating understandable and context-sensitive XAI systems.

Robrecht A, Voß H, Gottschalk L, Kopp S. A Study on Integrating Representational Gestures into Automatically Generated Embodied Explanations. In: Jack R, Chollet M, Aylett R, Bickmore T, Marsella S, Lucas G, eds. Proceedings of the ACM International Conference on Intelligent Virtual Agents. New York, NY, USA: ACM; 2024: 1-5.

Robrecht A, Kopp S. SNAPE: A Sequential Non-Stationary Decision Process Model for Adaptive Explanation Generation. Presented at the 15th International Conference on Agents and Artificial Intelligence , Lisbon.

Extended Reality (XR) has a potential to allow social interaction for people that are distant from one another, in educational, clinical or co-working applications, as well as for scientific studies. However, a full-blown embodied social presence and interaction via avatars in XR requires motion tracking hardware that many users do not have. At the same time, modern machine learning approaches enable the synthesis of natural and life-like nonverbal behavior, but only in offline settings and with considerable lag. We evaluate the applicability of current gesture generation systems for online interaction in social XR. We define a set of requirements for real-time-capable gesture generation and propose an approach to employ a state-of-the-art model in a real-time XR interaction pipeline. To test the model under conditions of online interaction, we divide an input audio stream into chunks of different lengths and stitch the resulting gesture animations together to form continuous motion. We evaluate the quality of the resulting multimodal avatar behavior in a user study. Our results show a significant trade-off between real-time generation capabilities and gesture quality. Suggestions for future improvement to retain model performance during online interaction in Social XR are made. A project page with videos of the generated gestures is available at https://nkrome.github.io/CAGE.html.

It is commonly assumed that explanations should be tailored to the addressee in order to yield higher understanding. Consequently, much work on explainable intelligent agents has been directed to user-adapted explanations. However, recent studies show ambiguous results with regard to the efficiency of adaptive and non-adaptive explanations. This raises the question whether an explanation, generated by a socially interactive agent, should be adapted. In this paper, we present a general approach to adaptive explanation generation as a non-stationary decision process, and we study the benefits and pitfalls of adapting explanations in an ongoing interaction with a user. Specifically, we report results from a between-subject online evaluation in a game explanation domain with three conditions (non-interactive, interactive but non-adaptive, adaptive). Results show that the decision for or against adaptivity depends on the goal of the explanation, the complexity of the domain and external constraints. Based on the collected data we discuss challenges that arise from the individuality of adaptive dialogues, such as comparability and the tendency to produce results with a large variance.

Due to their significance in human communication, the automatic generation of co-speech gestures in artificial embodied agents has received a lot of attention. Although modern deep learning approaches can generate realistic-looking conversational gestures from spoken language, they often lack the ability to convey meaningful information and generate contextually appropriate gestures. This paper presents an augmented approach to the generation of co-speech gestures that additionally takes into account given form and meaning features for the gestures. Our framework effectively acquires this information from a small corpus with rich semantic annotations and a larger corpus without such information. We provide an analysis of the effects of distinctive feature targets and we report on a human rater evaluation study demonstrating that our framework achieves semantic coherence and person perception on the same level as human ground truth behavior. We make our data pipeline and the generation framework publicly available.

We present two approaches to animate an android robot head according to audio speech input, both are adopted from recent machine learning based works in computer graphics animation. More concrete we implemented a viseme-based and a mesh-based approach on our robot. After a subjective comparison we conduct a speech-reading study to evaluate our preferred, the mesh-based, approach. The results show that on average the intelligibility is not increased by the visual cues provided through the robot head in comparison to noisy audio alone. This underlines the importance of carefully designing and controlling the facial co-speech movements of talking android heads.

Theory of Mind is the process of ascribing mental states to other individuals we interact with. It is used for sense-making of the observed actions and prediction of future actions. Previous studies revealed that humans mentalize about artificial agents, but it is not entirely clear how and to what extent. At the same time mentalizing about humans is often influenced by biases such as an egocentric bias. We present a study investigating differences in participants’ ToM and their susceptibility to an egocentric bias when observing humans vs robots. The participants observed an autonomous robot, a controlled robot, and a human in the same scenarios. The agents had to find an object in a laboratory. While watching the agents, participants had to make several action predictions as an implicit measure of ToM, potentially revealing an egocentric bias. At the end, questions about the agent’s responsibility, awareness and strategy were asked. The results indicate that while participants generally performed ToM for all types of agents, both the scenario as well as the agent type appear to influence participants’ likelihood of exhibiting an egocentric bias.

Social robots interacting with users in real-life environments will often show surprising or even undesirable behavior. In this paper we investigate whether a robot's ability to self-explain its behavior affects the users' perception and assessment of this behavior. We propose an explanation model based on humans' folk-psychological concepts and test different explanation strategies in specifically designed HRI scenarios with robot behaviors perceived as intentional, but differently surprising or desirable. All types of explanation strategies increased the understandability and desirability of the behaviors. While merely stating an action had similar effects as giving a reason for it (an intention or need), combining both in a causal explanation helped the robot to better justify its behavior and to increase its understandability and desirability to a larger extent.

In this paper, we take a first step in exploring the effect of xR devices on the experiences of users in the context of human-agent interactions. We report on the design and heuristic evaluation of an embodied conversational agent integrated into either a Virtual or Mixed Reality environment. Our evaluation with four experts show that Virtual Reality may elicit a better experience from users, while Mixed Reality may evoke a better sense of social presence due to natural user embodiment and spatial references to physical objects in the real environment.

Social robots are increasingly applied to support children’s learning, but how a robot can foster (or may hinder) learning is still not fully clear. One technique used by teachers is scaffolding, temporarily assisting learners to achieve new skills or levels of understanding they would not reach on their own. We ask if and how a social robot can be utilized to scaffold second-language learning of children at kindergarten age (4--7 years). Specifically, we explore an adapt-and-explain scaffolding strategy in which a robot acts as a peer-like tutor who dynamically adapts its behavior or the learning tasks to the cognitive and affective state of the child, and provides verbal explanations of these adaptations. An evaluation study with 40 children shows that children benefit from the learning adaptation and that the explanations have a positive effect especially for slower learners. Further, in 76% of all cases the robot managed to “re-engage” children who started to disengage from the learning interaction, helping them to achieve an overall higher learning gain. These findings demonstrate that a social robot equipped with suitable scaffolding mechanisms can increase engagement and learning, especially when being adaptive to the individual behavior and states of a child learner.

This video illustrates the large-scale experiment of the L2TOR project that will be presented at the HRI 2019 conference. The experiment aimed to investigate how 192 Dutch 5-year-old children could learn 34 English words from a NAO robot in 7 lessons. The experiment compared 4 conditions: 1) robot using iconic gestures, 2) robot without iconic gestures, 3) tablet only, and 4) a control group. The results revealed that children could learn more English words in all experimental conditions compared to the control group. The three experimental conditions did not show any significant differences regarding the learning outcomes.

We present a large-scale study of a series of seven lessons designed to help young children learn English vocabulary as a foreign language using a social robot. The experiment was designed to investigate 1) the effectiveness of a social robot teaching children new words over the course of multiple interactions (supported by a tablet), 2) the added benefit of a robot's iconic gestures on word learning and retention, and 3) the effect of learning from a robot tutor accompanied by a tablet versus learning from a tablet application alone. For reasons of transparency, the research questions, hypotheses and methods were preregistered. With a sample size of 194 children, our study was statistically well-powered. Our findings demonstrate that children are able to acquire and retain English vocabulary words taught by a robot tutor to a similar extent as when they are taught by a tablet application. In addition, we found no beneficial effect of a robot's iconic gestures on learning gains.

In the future, people and intelligent technical systems will cooperate in situations in which the goals, means, or actions are not completely prespecified, but develop in the course of a process that entails also the finding of new goals and strategies. This position paper presents an account of how these processes can be described such that they enable support of human decision-making and action coordination in such open, under-specified scenarios. We discuss how technical cognitive systems can assist these processes by bringing together techniques for multi-modal processing, information retrieval, situated action planning and autonomous action generation, with novel capabilities of recognizing and anticipating task-related (cognitive-intentional, procedural, affective) states of the actors, and for cooperative goal refinement and action coordination among the actors. A foremost requirement is to automatically provide markers for the indication of necessary strategy changes that reconFigure the space of actions, where it is to be expected that such strategy changes may require explanation and mediation. To that end, cognitive systems and robots must be endowed with new kinds of models of (explicit or implicit) cooperative processes.

This paper presents a study in which children, four to six years old, were taught words in a second language by a robot tutor. The goal is to evaluate two ways for a robot to provide scaffolding for students: the use of iconic gestures, combined with adaptively choosing the next learning task based on the child»s past performance. The results show a positive effect on long-term memorization of novel words, and an overall higher level of engagement during the learning activities when gestures are used. The adaptive tutoring strategy reduces the extent to which the level of engagement is diminishing during the later part of the interaction.

Eye gaze of is an informative social signal in interactions with other humans and also with virtual agents (VA). But for a successful communication, users have to accurately perceive the VA's point of gaze (POG). In our study, participants sitting opposite to a VA at a table indicated its POG by positioning a token on the table surface. We measured the perceptual accuracy within and between participants as well as the participants' response times and eye movements for five horizontally aligned gaze targets. We demonstrated that perceiving the VA met perceptual benchmarks from human lookers: a) variances within and between participants were only slightly larger, b) the VA's visual angle was linearly overestimated, and c) variances increased with the visual angle. Finally, participants showed large individual differences but were consistent in their own gaze behaviour and response times across trials and gaze targets.

Dialog modeling of making suggestions in human-agent interaction is a challenge due to the socially delicate nature of a suggestion and ensuing interactional negotiations. A basic first dialog model for making suggestions was tested in the context of schedule management assistance by an embodied conversational agent with elderly and mildly cognitively impaired persons. Analysis showed that users responded according to human social structures with most response types bearing potential challenges concerning the system's language understanding and the users' intention interpretation:next to explicit answers, users produced implicit versions for acceptance or resistance and further requests for information or modifications. Thus, an enhanced dialog model with a newly added clarification sequence and a new multi-conditional entry sequence was tested in a second study with the autonomous system. Initial observations show a promising performance of the dialog model.

In this paper we explore whether a social robot can learn, in and from a task-oriented interaction with a human user, how to employ different social behaviors to achieve interactional goals under specific situational circumstances. We present a multimodal behavior generation architecture that maps high-level interactional functions and behaviors onto low-level behaviors executable by a robot. While high-level behaviors are selected based on the state of the user as well as the interaction, reinforcement learning is used within each behavior to optimize its local mapping onto lower-level behaviors. The approach is implemented and applied in a scenario in which a social robot (Furhat) assists a human player in solving a Memory game by guiding the attention of the user to target objects. Results of an evaluation study demonstrate that participants are able to solve the Memory faster with the adaptive, assistive robot.

In this paper, we present an approach to adaptive language tutoring in child-robot interaction. The approach is based on a dynamic probabilistic model that represents the inter-relations between the learner's skills, her observed behaviour in tutoring interaction, and the tutoring action taken by the system. Being implemented in a robot language tutor, the model enables the robot tutor to trace the learner's knowledge and to decide which skill to teach next and how to address it in a game-like tutoring interaction. Results of an evaluation study are discussed demonstrating how participants in the adaptive tutoring condition successfully learned foreign language words.

Many approaches for motion processing or motion analysis employ Dynamic Time Warping (DTW) for temporally aligning an input movement with a reference movement. DTW, however, does not work online since it requires the complete input trajectory. Its online extension Open-End DTW can lead to poor alignments. In this paper we propose Weight-Optimized Open-End DTW, which combines path-length weighting and joint weights optimized from training data. We demonstrate our method to work online and to outperform Open-End DTW in terms of alignment quality.

We present the flexdiam dialogue management architecture, which was developed in a series of projects dedicated to tailoring spoken interaction to the needs of users with cognitive impairments in an everyday assistive domain, using a multimodal front-end.This hybrid DM architecture affords incremental processing of uncertain input, a flexible, mixed-initiative information grounding process that can be adapted to users' cognitive capacities and interactive idiosyncrasies, and generic mechanisms that foster transitions in the joint discourse state that are understandable and controllable by those users, in order to effect a robust interaction for users with varying capacities.

Latency between a user's movement and visual feedback is inevitable in every Virtual Reality application, as signal transmission and processing take time. Unfortunately, a high end-to-end latency impairs perception and motor performance. While it is possible to reduce feedback delay to tens of milliseconds, these delays will never completely vanish. Currently, there is a gap in literature regarding the impact of feedback delays on perception and motor performance as well as on their interplay in virtual environments employing full-body avatars. With the present study at hand, we address this gap by performing a systematic investigation of different levels of delay across a variety of perceptual and motor tasks during full-body action inside a Cave Automatic Virtual Environment. We presented participants with their virtual mirror image, which responded to their actions with feedback delays ranging from 45 to 350 ms. We measured the impact of these delays on motor performance, sense of agency, sense of body ownership and simultaneity perception by means of psychophysical procedures. Furthermore, we looked at interaction effects between these aspects to identify possible dependencies. The results show that motor performance and simultaneity perception are affected by latencies above 75 ms. Although sense of agency and body ownership only decline at a latency higher than 125 ms, and deteriorate for a latency greater than 300 ms, they do not break down completely even at the highest tested delay. Interestingly, participants perceptually infer the presence of delays more from their motor error in the task than from the actual level of delay. Whether or not participants notice a delay in a virtual environment might therefore depend on the motor task and their performance rather than on the actual delay.

We analyzed autonomous conversational spoken interaction as a modality for assistive systems in initial groups of older adults and people with cognitive impairments; we had previously explored this in a WOz setup. Solving a simple task in the domain of week planning, subjects readily interacted with the system. Performance was generally good, but dependent on successful adherence to a specific terse conversational style. Enforcing these patterns in a socially acceptable way during conversation is the next goal for the system.

We present a multimodal coaching system that supports online motor skill learning. In this domain, closed-loop interaction between the movements of the user and the action instructions by the system is an essential requirement. To achieve this, the actions of the user need to be measured and evaluated and the system must be able to give corrective instructions on the ongoing performance. Timely delivery of these instructions, particularly during execution of the motor skill by the user, is thus of the highest importance. Based on the results of an empirical study on motor skill coaching, we analyze the requirements for an interactive coaching system and present an architecture that combines motion analysis, dialogue management, and virtual human animation in a motion tracking and 3D virtual reality hardware setup. In a preliminary study we demonstrate that the current system is capable of delivering the closed-loop interaction that is required in the motor skill learning domain.

Rottschäfer S, Buschmeier H, van Welbergen H, Kopp S. Online Lombard-adaptation in incremental speech synthesis. In: Proceedings of INTERSPEECH 2015. Dresden, Germany; 2015: 80-84.

Virtual Reality (VR) has the potential to support motor learning in ways exceeding beyond the possibilities provided by real world environments. New feedback mechanisms can be implemented that support motor learning during the performance of the trainee and afterwards as a performance review. As a consequence, VR environments excel in controlled evaluations, which has been proven in many other application scenarios.

In this paper, we present a hybrid grammar formalism designed to learn structured models of natural iconic gesture performances that allow for compressed representation and robust recognition. We analyze a dataset of iconic gestures and show how the proposed Feature-based Stochastic Context-Free Grammar (FSCFG) can generalize over both structural and feature-based variations among different gesture performances.

It is established that driver distraction is the result of sharing cognitive resources between the primary task (driving) and any other secondary task. In the case of holding conversations, a human passenger who is aware of the driving conditions can choose to interrupt his speech in situations potentially requiring more attention from the driver, but in-car information systems typically do not exhibit such sensitivity. We have designed and tested such a system in a driving simulation environment. Unlike other systems, our system delivers information via speech (calendar entries with scheduled meetings) but is able to react to signals from the environment to interrupt when the driver needs to be fully attentive to the driving task and subsequently resume its delivery. Distraction is measured by a secondary short-term memory task. In both tasks, drivers perform significantly worse when the system does not adapt its speech, while they perform equally well to control conditions (no concurrent task) when the system intelligently interrupts and resumes.

When a passenger speaks to a driver, he or she is co-located with the driver, is generally aware of the situation, and can stop speaking to allow the driver to focus on the driving task. In-car dialogue systems ignore these important aspects, making them more distracting than even cell-phone conversations. We developed and tested a "situationally-aware" dialogue system that can interrupt its speech when a situation which requires more attention from the driver is detected, and can resume when driving conditions return to normal. Furthermore, our system allows driver-controlled resumption of interrupted speech via verbal or visual cues (head nods). Over two experiments, we found that the situationally-aware spoken dialogue system improves driving performance and attention to the speech content, while driver-controlled speech resumption does not hinder performance in either of these two tasks

Humans frequently join words and gestures for multimodal communication. Such natural co-speech gesturing goes far beyond what can be currently processed by gesture-based interfaces and especially its coordination with speech still poses open challenges for basic research and multimodal interfaces alike. How can we develop computational models for processing and generating natural speech-gesture behavior, in a flexible, fast and adaptive manner similar to humans? In this talk I will review approaches and methods applied to this problem and I will argue that such models need to (and can) based on a deeper understanding of what shapes co-speech gesturing in a particular situation. I will present work that connects empirical analyses with computational modeling and evaluation to unravel the cognitive, embodied and socio-interactional mechanisms underlying the use of speech- accompanying gestural behavior, and to develop deeper models of these mechanisms for interactive systems such as virtual characters, humanoid robots, or multimodal interfaces.

Humans perform iconic gestures to refer to entities through embodying their shapes. For instance, people often gesture the outline of an object (e.g. a circle for a ball) when referring to it during communication. In this paper, we present a gesture-based object recognition algorithm that enables natural human-computer interaction involving iconic gestures. Based on our analysis of multiple gesture performances, we propose a new 3D motion description of iconic gestures, called Histograms of Guiding Strokes (HoGS), which successfully summarizes hand dynamic during gestures. Our gesture-based object recognition algorithm compares favorably to human judgment performance and outper-forms most conventional gesture recognition approaches. 1

Buschmeier H, Malisz Z, Wlodarczak M, Kopp S, Wagner P. 'Are you sure you're paying attention?' – 'Uh-huh'. Communicating understanding as a marker of attentiveness. In: Proceedings of INTERSPEECH 2011. International Speech Communication Association; 2011: 2057-2060.

Gesture is an important feature of social interaction, frequently used by human speakers to illustrate what speech alone cannot provide, e.g. to convey referential, spatial or iconic information. Accordingly, humanoid robots that are intended to engage in natural human-robot interaction should produce speech-accompanying gestures for comprehensible and believable behavior. But how does a robot's non-verbal behavior influence human evaluation of communication quality and the robot itself? To address this research question we conducted two experimental studies. Using the Honda humanoid robot we investigated how humans perceive various gestural patterns performed by the robot as they interact in a situational context. Our findings suggest that the robot is evaluated more positively when non-verbal behaviors such as hand and arm gestures are displayed along with speech. These findings were found to be enhanced when the participants were explicitly requested to direct their attention towards the robot during the interaction.

In this paper, we present a calibration-free head gesture recognition system using a motion-sensor-based approach. For data acquisition we conducted a comprehensive study with 10 subjects. We analyzed the resulting head movement data with regard to separability and transferability to new subjects. Ordered means models (OMMs) were used for classification, since they provide an easy-to-use, fast, and stable approach to machine learning of time series. In result, we achieved classification rates of 85-95% for nodding, head shaking and tilting head gestures and good transferability. Finally, we show first promising attempts towards online recognition.

One of the crucial aspects in building sociable, communicative robots is to endow them with expressive nonverbal behaviors. Gesture is one such behavior, frequently used by human speakers to illustrate what they express in speech. The production of gestures, however, poses a number of challenges with regard to motor control for arbitrary, expressive hand-arm movement and its coordination with other interaction modalities. We describe an approach to enable the humanoid robot ASIMO to flexibly produce communicative gestures at run-time, building upon the Articulated Communicator Engine (ACE) that was developed to allow virtual agents to realize planned behavior representations on the spot. We present a control architecture that tightly couples ACE with ASIMO's perceptuo-motor system for multi-modal scheduling. In this way, we combine conceptual representation and planning with motor control primitives for meaningful arm movements of a physical robot body. First results of realized gesture representations are presented and discussed

The generation of communicative, speech-accompanying robot gesture is still largely unexplored. We present an approach to enable the humanoid robot ASIMO to flexibly produce speech and co-verbal gestures at run-time, while not being limited to a pre-defined repertoire of motor actions. Since much research has already been dedicated to this challenge within the domain of virtual conversational agents, we build upon the experience gained from the development of a speech and gesture production model used for the virtual human Max. We propose a robot control architecture building upon the Articulated Communicator Engine (ACE) that was developed to allow virtual agents to flexibly realize planned multi-modal behavior representations on the spot. Our approach tightly couples ACE with ASIMO's perceptuo-motor system, combining conceptual representation and planning with motor control primitives for speech and arm movements of a physical robot body. First results of both gesture production and speech synthesis using ACE and the MARY text-to-speech system are presented and discussed.

Feedback utterances are an important part of any dialog be- tween humans. When two or more persons talk, they use short backchannel utterances to signal understanding and interest in the conversation. Surprisingly little is known about the rela- tionship between the accompanying prosody and the meaning of feedback perceived by the dialog partner. We present a study of 12 synthesized German ja (yes) interjections that shows the influence of prosodic features on emotional and pragmatic per- ception of this kind of feedback. Listeners perceived utterances as bored, hesitant, or happy and agreeing depending on the prosodic parameters used for synthesis.

When talking about spatial domains, humans frequently accompany their explanations with iconic gestures to depict what they are referring to. For example, when giving directions, it is common to see people making gestures that indicate the shape of buildings, or outline a route to be taken by the listener, and these gestures are essential to the understanding of the directions. Based on results from an ongoing study on language and gesture in direction-giving, we propose a framework to analyze such gestural images into semantic units (image description features), and to link these units to morphological features (hand shape, trajectory, etc.). This feature-based framework allows us to generate novel iconic gestures for embodied conversational agents, without drawing on a lexicon of canned gestures. We present an integrated microplanner that derives the form of both coordinated natural language and iconic gesture directly from given communicative goals, and serves as input to the speech and gesture realization engine in our NUMACK project.

Abstract Conversational agents are supposed to combine speech with non‐verbal modalities for intelligible multimodal utterances. In this paper, we focus on the generation of gesture and speech from XML‐based descriptions of their overt form. An incremental production model is presented that combines the synthesis of synchronized gestural, verbal, and facial behaviors with mechanisms for linking them in fluent utterances with natural co‐articulation and transition effects. In particular, an efficient kinematic approach for animating hand gestures from shape specifications is presented, which provides fine adaptation to temporal constraints that are imposed by cross‐modal synchrony. Copyright © 2004 John Wiley & Sons, Ltd.

Virtual conversational agents are supposed to combine speech with non-verbal modalities for intelligible and believable utterances. However, the automatic synthesis of co-verbal gestures is still struggling with several problems like naturalness in procedurally generated animations, flexibility in pre-defined movements, and synchronization with speech. In this paper we focus on generating complex multimodal utterances including gesture and speech from XML-based descriptions of their overt form. We describe a coordination model that reproduces coarticulation and transition effects in both modalities. In particular, an efficient kinematic approach to creating gesture animations from shape specifications is presented, which provides fine adaptation to temporal constraints that are imposed by cross-modal synchrony.

In this paper an operational model for the automatic generation of lifelike gestures of an anthropomorphic virtual agent is described. The biologically motivated approach to controlling the movements of a highly articulated figure provides a transformation of spatiotemporal gesture specifications into an analog representation of the movement from which the animations are directly rendered. To this end, knowledge-based computer animation techniques are combined with appropriate methods for trajectory formation and articulated figure animation.