Knowledge graphs have been used to improve robot dialogues by providing more sophisticated world knowledge. We now propose a new role for knowledge graphs in GenAl-based HRI that aims to reduce dialogue errors by better conversational grounding. This approach uses both domain knowledge graphs and dialogue history graphs, constructing shared knowledge via entity linking. We present first steps towards these aims, and also address sustainability by supporting the use of smaller models.

Dates are often treated as exact measures, yet their precision is limited. This applies not only to physical clocks but also to human memory. In conversations, we typically share timeframes relevant to the topic, not pinpoint accuracy. Precise birth times, for example, hold little conversational value compared to their biographical significance. Similarly, couples cherish their wedding day but might recall only the year they bought their house. This paper explores the advantages of incorporating fuzzy dates into personal knowledge graphs constructed through dialogues within the LifeLine feature of the “my eViTA” smartphone app. Fuzzy dates acknowledge the inherent imprecision of human memory and prioritize contextually relevant timeframes for a more natural and insightful knowledge representation.

Knowledge models are fundamental to dialogue systems for enabling conversational interactions, which require handling domainspecific knowledge.Ensuring effective communication in information-providing conversations entails aligning user understanding with the knowledge available to the system.However, dialogue systems often face challenges arising from semantic inconsistencies in how information is expressed in natural language compared to how it is represented within the system's internal knowledge.To address this problem, we study the potential of large language models for conversational grounding, a mechanism to bridge information gaps by establishing shared knowledge between dialogue participants.Our approach involves annotating human conversations across five knowledge domains to create a new dialogue corpus called BridgeKG.Through a series of experiments on this dataset, we empirically evaluate the capabilities of large language models in classifying grounding acts and identifying grounded information items within a knowledge graph structure.Our findings offer insights into how these models use in-context learning for conversational grounding tasks and common prediction errors, which we illustrate with examples from challenging dialogues.We discuss how the models handle knowledge graphs as a semantic layer between unstructured dialogue utterances and structured information items.

In many cases of human-human communication, humans interact with others while assuming their emotions and impressions based on not only verbal information but also non-verbal information. Similarly, during the human-robot interaction, predicting the impressions that a person has of the robot is important for the robot to change the behavior and realize good interaction. In this work, we try to use gaze and gesture annotation data in human-robot interaction from AICO-Corpus and show LSTM approach has the potential for the prediction about impressions of interaction with a robot. We also analyzed the types of nonverbal information that influence the impressions towards the robot in English and Japanese respectively.

The paper discusses two current approaches to conversational AI, using large language models and knowledge graphs, and compares types of errors that occur in human-robot interactions based on these approaches. It provides example dialogues and describes solutions to several error types including false implications, ontological errors, theory of mind errors, and handling of speech recognition errors. The paper addresses issues of particular concern for earning user trust.

Speech is an intuitive way to interact with social robots: spoken language dialogues can help users to express their intents in a natural and flexible manner. In recent years, there has been remarkable progress in artificial intelligence related to spoken dialogue technology, including speech recognition and natural language processing. In this paper, we present the integration of the open source speech recognition, natural language processing, and dialogue management components into a robot software platform, and also report on a preliminary experiment of the integrated system using real users. Gesturing of the robot, which is also important in human-robot interaction, is combined with the spoken content of the robot utterance and included in the dialogue management component. As the dialogue domain we chose mealtime discussions on food and recipes, since spoken communication with a companion robot in such scenarios is considered natural and useful.

The paper describes an approach that combines work from three fields with previously separate research commu-nities: social robotics, conversational AI, and graph databases. The aim is to develop a generic framework in which a variety of social robots can provide high-quality information to users by accessing semantically-rich knowledge graphs about multiple different domains. An example implementation uses a Furhat robot with Rasa open source conversational AI and knowledge graphs in Neo4j graph databases.

Abstract With life expectancy growing rapidly over the past century, societies are being increasingly faced with a need to find smart living solutions for elderly care and active ageing. The e-VITA project, which is a joint European (H2020) and Japanese (MIC) funded project, is based on an innovative approach to virtual coaching that addresses the crucial domains of active and healthy ageing. In this paper we describe the role of spoken dialogue technology in the project. Requirements for the virtual coach were elicited through a process of participatory design in workshops, focus groups, and living labs, and a number of use cases were identified for development using the open-source RASA framework. Knowledge Graphs are used as a shared representation within the system, enabling an integration of multimodal data, context, and domain knowledge.

The paper addresses aspects of cooperative and uncooperative behaviour in natural language dialogue between humans and social robots. The principles of cooperation in human-human dialogues are taken as the basis for cooperative behaviour in human-robot dialogues. Several approaches are described that can improve human-robot cooperation. These include more flexible recognition of user intents, more flexible searches using knowledge graphs, generating more cooperative responses using semantic metadata, and generating more human-friendly responses using Wikipedia. These approaches are demonstrated in a series of videos.

The paper discusses ethical issues in dialogue engagement to build a long-term trusting relationship between a human and a robot agent over a period of time. Engagement refers not only to frequent interactions but also to recognition of the partner and their non-verbal communication which indicates level of interest, activity, and affective state. In applications related to real-world care-taking and instructional tasks, a robot agent’s ability to recognize human partners and react appropriately can prove crucial for smooth interaction. Recognizing the partner’s face and remembering their individual interactions can contribute positively to the acceptance of social robot applications by enabling adaptation to the user’s needs and building long-term relations. For this purpose, we describe work on short-term and long-term memories in the robot’s dialogue manager, and the use of facial landmark recognition to identify the user.

Peer reviewed

This paper describes data collection and the first explorative research on the AICO Multimodal Corpus. The corpus contains eye-gaze, Kinect, and video recordings of human-robot and human-human interactions, and was collected to study cooperation, engagement and attention of human participants in task-based as well as in chatty type interactive situations. In particular, the goal was to enable comparison between human-human and human-robot interactions, besides studying multimodal behaviour and attention in the different dialogue activities. The robot partner was a humanoid Nao robot, and it was expected that its agent-like behaviour would render humanrobot interactions similar to human-human interaction but also high-light important differences due to the robot’s limited conversational capabilities. The paper reports on the preliminary studies on the corpus, concerning the participants’ eye-gaze and gesturing behaviours,which were chosen as objective measures to study differences in their multimodal behaviour patterns with a human and a robot partner.

Social robots are receiving more attention through increased research and development, and they are gradually becoming a part of our daily lives. In this study, we investigated how social robots are accepted by robot users. We applied the theoretical lens of the boundary-crossing robot concept, which describes the role shift of robots from tools to agents. This concept highlights the impact of social robots on the everyday lives of humans, and can be used to structure the development of perceived interactions between robots and human users. In this paper, we report on the results of a web questionnaire study conducted among users of interactive devices (humanoid robots, animal robots, and smart speakers). Their acceptance and roles in daily life are compared from both functional and affective perspectives, with respect to their perceived roles as boundary-crossing robots.

This paper describes experiments to learn laughter co-occurrences with dialogue contributions. The dialogue data belongs to the special type of First Encounter Dialogues where the interlocutors meet each other for the first time and where laughter mainly functions as a sign of politeness or relief of embarrassment. The earlier studies have shown that there is a correlation between the speaker's utterance content (topic) and non-verbal communication (laughter and body movement) while in this paper we seek to learn the correlations via a neural model. The results show that there seems to be a weak correlation in our data.

the use of eye-gaze patterns in evaluating the partner's understanding process. The goal of the research is to understand better how humans focus their attention when interacting with a robot and to build a model for natural gaze patters to improve the robot's engagement and interaction capabilities.

The demo shows ERICA, a highly realistic female android robot, and WikiTalk, an application that helps robots to talk about thousands of topics using information from Wikipedia. The combination of ERICA and WikiTalk results in more natural and engaging human-robot conversations.

This paper concerns interaction with social robots and focuses on the evaluation of a robot application that allows users to access interesting information from Wikipedia. The evaluation method compares the users' expectations with their experience with the robot, and takes into account their self-declared previous experience with robots. The results show that most participants had an overall positive experience, even though the averages indicate a slight negative tendency related to expectations of the robot's behavior and being understood by the robot. Interestingly, the most experienced users seem to be the most critical.

Eye gaze has been used broadly in interactive intelligent systems. The research area has grown in recent years to cover emerging topics that go beyond the traditional focus on interaction between a single user and an interactive system. This special issue presents five articles that explore new directions of gaze-based interactive intelligent systems, ranging from communication robots in dyadic and multiparty conversations to a driving simulator that uses eye gaze evidence to critique learners’ behavior.

At SIGDIAL-2013 our talking robot demonstrated Wikipedia-based spoken information access in English.Our new demo shows a robot speaking different languages, getting content from different language Wikipedias, and switching languages to meet the linguistic capabilities of different dialogue partners.

This paper presents a summary of the seventh workshop on Eye Gaze in Intelligent Human Machine Interaction. The Gaze-in 2014 workshop is a part of a series of workshops held around the topics related to gaze and multimodal interaction. The workshop web-site can be found at http://hhhuang.homelinux.com/gaze_in/.

This paper presents a summary of the sixth workshop in Eye Gaze in Intelligent Human Machine Interaction. The GazeIn'13 workshop is a part of a series of workshops held around the topics related to gaze and multimodal interaction.

The demo shows WikiTalk, a Wikipedia-based open-domain information access dialogue system implemented on a talking humanoid robot. The robot behaviour integrates speech, nodding, gesturing and face-tracking to support interaction management and the presentation of information to the partner.

Eye gaze is an important means for controlling interaction and coordinating the participants' turns smoothly. We have studied how eye gaze correlates with spoken interaction and especially focused on the combined effect of the speech signal and gazing to predict turn taking possibilities. It is well known that mutual gaze is important in the coordination of turn taking in two-party dialogs, and in this article, we investigate whether this fact also holds for three-party conversations. In group interactions, it may be that different features are used for managing turn taking than in two-party dialogs. We collected casual conversational data and used an eye tracker to systematically observe a participant's gaze in the interactions. By studying the combined effect of speech and gaze on turn taking, we aimed to answer our main questions: How well can eye gaze help in predicting turn taking? What is the role of eye gaze when the speaker holds the turn? Is the role of eye gaze as important in three-party dialogs as in two-party dialogue? We used Support Vector Machines (SVMs) to classify turn taking events with respect to speech and gaze features, so as to estimate how well the features signal a change of the speaker or a continuation of the same speaker. The results confirm the earlier hypothesis that eye gaze significantly helps in predicting the partner's turn taking activity, and we also get supporting evidence for our hypothesis that the speaker is a prominent coordinator of the interaction space. Such a turn taking model could be used in interactive applications to improve the system's conversational performance.

This is the fourth workshop in a series of workshops on Eye Gaze in Intelligent Human Machine Interaction, in which we have discussed a wide range of issues for eye gaze; technologies for sensing human attentional behaviors, roles of attentional behaviors as social gaze in human-human and human-humanoid interaction, attentional behaviors in problem-solving and task-performing, gaze-based intelligent user interfaces, and evaluation of gaze-based user interfaces. In addition to these topics, this year's workshop focuses on eye gaze in multimodal interpretation and generation. Since eye gaze is one of the facial communication modalities, gaze information can be combined with other modalities or bodily motions to contribute to the meaning of utterance and serve as communication signals.

Abstract Spoken interactions are known for accurate timing and alignment between interlocutors: turn-taking and topic flow are managed in a manner that provides conversational fluency and smooth progress of the task. This paper studies the relation between the interlocutors’ eye-gaze and spoken utterances, and describes our experiments on turn alignment. We conducted classification experiments by Support Vector Machine on turn-taking using the features for dialogue act, eye-gaze, and speech prosody in conversation data. As a result, we demonstrated that eye-gaze features are important signals in turn management, and seem even more important than speech features when the intention of utterances is clear. Index Terms : eye-gaze, dialogue, interaction, speech analysis, turn-taking 1. Introduction The role of eye-gaze in fluent communication has long since been acknowledged ([2]; [7]). Previous research has established close relations between eye-gaze and conversational feedback ([3]), building trust and rapport, as well as focus of shared attention ([15]). Eye-gaze is also important in turn-taking signalling: usually the interlocutors signal their wish to give the turn by gazing up to the interlocutor, leaning back, and dropping in pitch and loudness, and the partner can, accordingly, start preparing to take the turn. There is evidence that lack of eye contact decreases turn-taking efficiency in video-conferencing ([16]), and that the coupling of speech and gaze streams in a word acquisition task can improve performance significantly ([11]). Several computational models of eye-gaze behaviour for artificial agents have also been designed. For instance, [9] describe an eye-gaze model for believable virtual humans, [13] demonstrate gaze modelling for conversational engage-ment, and [10] built an eye-gaze model to ground information in interactions with embodied conversational agents. Our research focuses on turn-taking and eye-gaze alignment in natural dialogues and especially on the role of eye-gaze as a means to coordinate and control turn-taking. In our previous work [5,6] we noticed that in multi-party dialogues the participants head movement was important in signalling turn-taking, maybe because of its greater visibility than eye-gaze. (This is in agreement with [12], who noticed that in virtual environments, head tracking seems sufficient when people turn their heads to look but if the person is not turning their head to look at an object, then eye-tracking is important to discern the gaze of a person.) The main objective in the current research is to explore the relation between eye-gaze and speech, in particular, how the annotated turn and dialogue features and automatically recognized speech properties affect in turn management. Methodologically our research relies on experimentation and observation: signal-level measurements and analysis of gaze and speech are combined with human-level observation of dialogue events (dialogue acts and turn-taking). We use our three-party dialogue data that is analysed with respect to the interlocutors’ speech, and annotated with dialogue acts, eye-gaze, and turn-taking features [6]. The experiments deal with the classification of turn-taking events using the analysed features and the results show that eye-gaze speech information significantly improves the accuracy compared with the classification with dialogue act information only. However, what is also interesting that the difference between gaze and speech features is not significant, i.e. eye-gaze and speech are important signals in turn management, but their effect is parallel rather than complementary. Moreover, eye-gaze seems to more important than speech when the intention of the utterance is clear. The paper is structured as follows. We first describe the research on turn-taking and the alignment of speech and gaze in Section 2. We then present our data and speech analysis in Section 3, and experimental results as well as discussion concerning their importance in Section 4. Section 5 draws conclusions and points to future research.

We present evaluation results of a multimodal route navigation system that allows interaction using speech and tactile/visual modes. Various functional aspects of the system were studied, related especially to the IO-modalities and their use as means of communication. We compared the users’ expectations before the evaluation with their actual experience of the system, and found significant differences among various user groups.

The tutorial will introduce the major topics, established practices and methodologies in dialogue management research. Evaluation criteria and usability aspects for useful and enjoyable interactive systems will also be discussed. The tutorial is based on the framework of Constructive Dialogue Management, and focuses especially on the technological and theoretical challenges in designing adaptive and intelligent conversational systems.

This paper describes the user expertise model in AthosMail, a mobile, speech-based e-mail system.The model encodes the system's assumptions about the user expertise, and gives recommendations on how the system should respond depending on the assumed competence levels of the user.The recommendations are realized as three types of explicitness in the system responses.The system monitors the user's competence with the help of parameters that describe e.g. the success of the user's interaction with the system.The model consists of an online and an offline version, the former taking care of the expertise level changes during the same session, the latter modelling the overall user expertise as a function of time and repeated interactions.

One of the much discussed topics in building spoken dialogue systems is how to take the users into account when designing practical systems: given the more complex environment in which we have to interact with various automatic services, it is obvious that the systems are not only required to function impeccably in regard to their technical specification, but they should also fulfil requirements concerning appropriate user needs. In this paper, some usability issues related to human factors in interface design are discussed from the point of view of communicative competence and adaptation.