Due to recent advances in large language models and robotics, social robots will potentially play an important role in people’s daily lives soon, and are expected to improve dynamic multi-party group discussions in social scenarios. In this paper, we developed a system to assist dynamic group discussion with our social robot Haru. Our system is composed of three modules: a Dialogue Assistance module via integrating Haru with large language models which facilitates Haru to be an embodied chatbot; a Balancing and Welcoming Behavior module to improve users’ engagement and welcome new users to join the discussion with verbal behaviors; an Autonomous Eye Gazing module to show politeness during group discussion, e.g., gazing to the talking user or the less-engaging user to encourage her, looking to the new comer when she joins the discussion, gazing via eyeball movement when the current speaking user is close to the previous one. The autonomous eye gazing behavior was first trained via deep reinforcement learning in simulation and transferred to physical Haru in the real world. Results of our user study with 50 subjects show the significant performance of our system in assisting dynamic group discussion.

The use of social robots is rapidly expanding across various domains, including education and healthcare. To achieve human-like interactions, these robots should possess well-rounded personalities. A carefully designed personality enhances a social robot’s persuasiveness and increases its appeal during interactions with humans. This paper explores how personality traits can be effectively leveraged to generate conversational responses for social robots, making human-robot interactions more engaging. We introduce a knowledge base that profiles the multifaceted dimensions of the social robot Haru. To generate contextually appropriate responses, we employ a retrieval-augmented generation (RAG) approach to retrieve relevant personality traits. Additionally, we propose a method that integrates result filtering and prompt engineering to ensure consistency in Haru’s responses. To evaluate the effectiveness of our approach, we conduct a preliminary annotation survey assessing the retrieved personality traits and generated responses. The results demonstrate that our method improves conversational flow and enhances response faithfulness to retrieved personality traits. A demo of our approach can be seen at this URL: https://www.youtube.com/watch?v=5wCQDBeSkG8.

Social robots for children have focused mainly on conventional education domains such as teaching language, science, and math, while applications focusing on the enhancement of cultural competency are quite scarce. In this paper, we present a prototype of a robot-mediation framework for cross-cultural communication. This framework paves the way for a social robot to act as a mediator between groups of schoolchildren from different countries. First, we conducted a participatory design activity by an interdisciplinary team, resulting in the extraction of the design, robot’s roles, and technical requirements. Based on these requirements, we built the robot-mediation system prototype. We conducted a pilot study using the system with groups of high school children in Japan and Australia and our results show the potential of the system to drive children’s interest in communicating, sharing, and discussing cultural themes with their remote peers through the social robot.

Socially assistive robots are potentially to be integrated with human daily lives in the near future, and expected to be able to improve group dynamics when interacting with groups of people in social settings. In this paper, we developed a system with desktop robot Haru to assist group discussions. The system consists of three modules: a dialogue assistance module which facilitates Haru to speak to users and answer questions in a free way; a dialogue balance module to encourage participation of users in the discussion with verbal behaviors; an autonomous gazing behavior module trained via deep reinforcement learning in simulation and deployed on physical Haru in reality, which can show politeness during group discussion, e.g., gazing to the speaking member, looking to the middle when both members are talking or silent, looking at the least spoken person when encouraging her. Results of user study with 40 subjects show the significant effectiveness of our system in assisting group discussion.

In this paper, building on recent advances in the fields of gaming AI and social robotics, we present a new approach to facilitate the social robot Haru to imitate game strategies from human players’ demonstrated trajectories and evaluative feedback in a real-time two-player game. Our research shows that Haru is able to learn and imitate human different game strategies from human players in a human time scale. In addition, our results show that human evaluative feedback plays an important role in allowing Haru to obtain a better performance via our method than human player’s demonstrations. Finally, results of our user study indicate that Haru imitating human player’s game strategies via our method is perceived to be more human-like and have better game performance and experience than self-learning from pre-defined reward functions via traditional deep reinforcement learning.

Social robots are gradually integrating into human’s daily lives. Storytelling by social robots could bring a different experience to users through non-verbal and emotional capabilities compared to text-only one. However, as user needs and preferences over storytelling might change over time during long-term interaction with social robots, it is important for social robots to learn from social interactions with human users in real-time. In this paper, we propose to allow our social robot Haru to learn personalized storytelling styles for different human user’s emotional states via human-centered reinforcement learning using the reward provided and delivered by directly interaction with the user explicitly. Results of our user study show that Haru can learn to adapt its storytelling style for detected human emotional states in a few number of interactions, and was perceived to have a better storytelling performance, experience and impact than a neutral one.

Conveying a robot's target mood is crucial to successful social interactions. The robot's expressive performance must be appropriate, persuasive, and consistent. However, this is challenging when interactions contain a mixture of scripted and improvised content, such as those generated by language models. In this paper, we take on the task of teaching robots to stay in character, that is to say, exhibit consistency in mood during interactions. We start by defining a communication strategy module that allows for the top-down specification of a target robot mood for a given task, goal, or context. We then propose a mood steering framework for enforcing robot mood consistency throughout an interaction that supports several target moods. Our framework consists of two components: 1. expressivity steering specifies the speech and behavior to be used by the robot to convey a target mood, and 2. language model steering ensures that improvised language is consistent with the robot's target mood. As a first step toward identifying effective communication strategies, we implement grumpy and cheerful strategies for a collaborative storytelling game and compare them to a neutral baseline. Evaluation in a collaborative storytelling game shows that our approach generates robot behavior that successfully conveys the robot's target mood throughout gameplay and language model steering generates story contributions that capture the target mood without quality degradation and raises important issues for communication strategy design.

This paper describes the development of a 3- stage signalling framework to trigger a social robot's bottom- up reactive behavior inspired by a biological model. In the first stage, low-level firing of stimuli due to external sources is constructed through perception grounding. This is followed by a saliency classifier which fires-up high level salient signals that require attention and are used to trigger the robot's reactive behavior. The whole framework evolves primarily on the knowledge ontology that defines the characteristics of the social robot and the querying mechanism that correlates the perceived stimuli with the ontology to trigger the reactive behavior. We evaluated the performance of our system with timing metrics and we achieved good results for our application.

Conversation can play an essential role in forging bonds between humans and social robots, but participants need to feel like they are being listened to, remembered, and cared about in order to effectively build rapport. In this paper, we propose a novel strategy for conducting small talk with a social robot. Our approach is known as the Tiers of Friendship. It is centered around three core design elements: 1) Persuasive content and character is provided through topic modules created by professional creative writers to ensure engaging conversational content and a compelling personality for the social robot. 2) Conversational memory is achieved by allowing topic modules to specify required information that can be learned through conversation or recalled from previous interactions and organizing topic modules into a hierarchy that enforces information requirements between topics. 3) Dynamicity in conversation is promoted through topic navigation that supports fluid transitions to topics of human interest and employs elements of random ordering to create fresh conversation experiences. In this paper, we show how the Tiers of Friendship can be used to generate conversation content for a social robot that encourages the development of rapport. We describe a working implementation of a small talk system for a social robot based on the Tiers of Friendship that combines off-the-shelf ASR and NLU components and custom robot behavior components implemented via behavior trees on ROS. Finally, in order to evaluate our approach's effectiveness, we conduct an elicitation survey that evaluates conversations in terms of perceived engagement, personality traits, and rapport expectation and discuss the implications for social robotics.

Collaborative storytelling has long been a goal of social robotics, however, much of this research is limited in interactivity or assumes that story content is curated. In this paper, we present a working fully-automatic collaborative storytelling robot, which can collaborate with a person to create a unique, improvised story by using a large-scale neural language model to dynamically generate continuations to a story. Because effective storytelling requires engaging the emotions of participants, we explore several modalities of procedurally-generated expressivity: 1. an expressive text-to-speech voice with several delivery styles, 2. physical and verbal reactions performed by the robot, and 3. an external display used to show instructions and graphics during storytelling.To understand the issues associated with improvised collaborative storytelling with a social robot, we conduct an online survey and elicitation study with a group of online observers of collaborative storytelling gameplay, comparing several expressivity strategies in terms of storytelling-related characteristics, expressivity characteristics, and personality traits as measured by RoSAS. This evaluation showed that expressivity strategies using both emotive voice and performed reactions were perceived to be more competent storytellers and more strongly associated with positive personality traits.

Storytelling plays a central role in human socializing and entertainment, and research on conducting storytelling with robots is gaining interest. However, much of this research assumes that story content is curated. In this paper, we expand the recently-proposed task of collaborative storytelling, where an intelligent agent and a person collaborate to create a unique story by taking turns adding to it, for application to social robot and consider the design implications that arise. Since latency can be detrimental to human-robot interaction, we examine the performance-latency trade-offs of an existing generate-and-rank-based approach to collaborative storytelling by finding the optimal ranker’s sample size that strikes the best balance between quality and computational cost. We improve on existing evaluation that was previously based on system-generated stories by having human participants play the collaborative storytelling game with our system and comparing the stories they create with our system to a naive baseline. Finally, we conduct a pilot elicitation survey that sheds light on issues to consider when adapting our collaborative storytelling system to a social robot. Our evaluation shows that participants have a positive view of collaborative storytelling with a social robot and consider rich, emoting capabilities to be key to enjoyment.

In this paper, we explore the storytelling potential of a robot. We exploit the use of creative contents that maximize the embodied communication affordance of the empathic robot Haru. We identify the elements in storytelling such as narration, agency, engagement and education and synthesized these into the robot. Through effective design we investigated the possible answers that could leverage the limitations and the challenges in developing storytelling applications through a robotic medium. Our preliminary findings show that the use of an embodied agent such as a robot in storytelling only has meaning when its communicative affordance (i.e. embodiment, expressiveness, and other modalities) is tapped, adding new dimension to the experience. Otherwise, traditional storytelling delivery (e.g. tablet) without the use of embodiment will suffice. Hence, robots need to be performers rather than just mere props in storytelling.

Realistic recordings of soundscapes often have multiple sound events co-occurring, such as car horns, engine and human voices. Sound event retrieval is a type of contentbased search aiming at finding audio samples, similar to an audio query based on their acoustic or semantic content. State of the art sound event retrieval models have focused on single-label audio recordings, with only one sound event occurring, rather than on multi-label audio recordings (i.e., multiple sound events occur in one recording). To address this latter problem, we propose different Deep Learning architectures with a Siamesestructure and a Pairwise Presence Matrix. The networks are trained and evaluated using the SONYC-UST dataset containing both single- and multi-label soundscape recordings. The performance results show the effectiveness of our proposed model.

Storytelling plays a central role in human socializing and entertainment. However, much of the research on automatic storytelling generation assumes that stories will be generated by an agent without any human interaction. In this paper, we introduce the task of collaborative storytelling, where an artificial intelligence agent and a person collaborate to create a unique story by taking turns adding to it. We present a collaborative storytelling system which works with a human storyteller to create a story by generating new utterances based on the story so far. We constructed the storytelling system by tuning a publicly-available large scale language model on a dataset of writing prompts and their accompanying fictional works. We identify generating sufficiently human-like utterances to be an important technical issue and propose a sample-and-rank approach to improve utterance quality. Quantitative evaluation shows that our approach outperforms a baseline, and we present qualitative evaluation of our system’s capabilities.

Grounding, or localizing, a textual phrase in an image is a challenging problem that is integral to visual language understanding. Previous approaches to this task typically make use of candidate region proposals, where end performance depends on that of the region proposal method and additional computational costs are incurred. In this paper, we treat grounding as a regression problem and propose a method to directly identify the region referred to by a textual phrase, eliminating the need for external candidate region prediction. Our approach uses deep neural networks to combine image and text representations and refines the target region with attention models over both image subregions and words in the textual phrase. Despite the challenging nature of this task and sparsity of available data, in evaluation on the ReferIt dataset, our proposed method achieves a new state-of-the-art in performance of 37.26% accuracy, surpassing the previously reported best by over 5 percentage points. We find that combining image and text attention models and an image attention area-sensitive loss function contribute to substantial improvements.

We address the problem of acquiring the ontological categories of unknown terms through implicit confirmation in dialogues.We develop an approach that makes implicit confirmation requests with an unknown term's predicted category.Our approach does not degrade user experience with repetitive explicit confirmations, but the system has difficulty determining if information in the confirmation request can be correctly acquired.To overcome this challenge, we propose a method for determining whether or not the predicted category is correct, which is included in an implicit confirmation request.Our method exploits multiple user responses to implicit confirmation requests containing the same ontological category.Experimental results revealed that the proposed method exhibited a higher precision rate for determining the correctly predicted categories than when only single user responses were considered.

Named entity recognition is a challenging task that has traditionally required large amounts of knowledge in the form of feature engineering and lexicons to achieve high performance. In this paper, we present a novel neural network architecture that automatically detects word- and character-level features using a hybrid bidirectional LSTM and CNN architecture, eliminating the need for most feature engineering. We also propose a novel method of encoding partial lexicon matches in neural networks and compare it to existing approaches. Extensive evaluation shows that, given only tokenized text and publicly available word embeddings, our system is competitive on the CoNLL-2003 dataset and surpasses the previously reported state of the art performance on the OntoNotes 5.0 dataset by 2.13 F1 points. By using two lexicons constructed from publicly-available sources, we establish new state of the art performance with an F1 score of 91.62 on CoNLL-2003 and 86.28 on OntoNotes, surpassing systems that employ heavy feature engineering, proprietary lexicons, and rich entity linking information.

This paper proposes a spiking-neural-network-based robot controller inspired by the control structures of biological systems. Information is routed through the network using facilitating dynamic synapses with short-term plasticity. Learning occurs through long-term synaptic plasticity which is implemented using the temporal difference learning rule to enable the robot to learn to associate the correct movement with the appropriate input conditions. The network self-organizes to provide memories of environments that the robot encounters. A Pioneer robot simulator with laser and sonar proximity sensors is used to verify the performance of the network with a wall-following task, and the results are presented.

Online video presents a great opportunity for up-and-coming singers and artists to be visible to a worldwide audience. However, the sheer quantity of video makes it difficult to discover promising musicians. We present a novel algorithm to automatically identify talented musicians using machine learning and acoustic analysis on a large set of "home singing" videos. We describe how candidate musician videos are identified and ranked by singing quality. To this end, we present new audio features specifically designed to directly capture singing quality. We evaluate these vis-a-vis a large set of generic audio features and demonstrate that the proposed features have good predictive performance. We also show that this algorithm performs well when videos are normalized for production quality.

Since the problem of textual entailment recognition requires capturing semantic relations between diverse expressions of language, linguistic and world knowledge play an important role. In this article, we explore the effectiveness of different types of currently available resources including synonyms, antonyms, hypernym-hyponym relations, and lexical entailment relations for the task of textual entailment recognition. In order to do so, we develop an entailment relation recognition system which utilizes diverse linguistic analyses and resources to align the linguistic units in a pair of texts and identifies entailment relations based on these alignments. We use the Japanese subset of the NTCIR-9 RITE-1 dataset for evaluation and error analysis, conducting ablation testing and evaluation on hand-crafted alignment gold standard data to evaluate the contribution of individual resources. Error analysis shows that existing knowledge sources are effective for RTE, but that their coverage is limited, especially for domain-specific and other low-frequency expressions. To increase alignment coverage on such expressions, we propose a method of alignment inference that uses syntactic and semantic dependency information to identify likely alignments without relying on external resources. Evaluation adding alignment inference to a system using all available knowledge sources shows improvements in both precision and recall of entailment relation recognition.

This paper presents a Spiking Neural Network (SNN) architecture for mobile robot navigation. The SNN contains 4 layers where dynamic synapses route information to the appropriate neurons in each layer and the neurons are modeled using the Leaky Integrate and Fire (LIF) model. The SNN learns by self-organizing its connectivity as new environmental conditions are experienced and consequently knowledge about its environment is stored in the connectivity. Also a novel feature of the proposed SNN architecture is that it uses working memory, where present and previous sensor states are stored. Results are presented for a wall following application.

We present data-driven methods for supporting musical creativity by capturing the statistics of a musical database. Specifically, we introduce a system that supports users in exploring the high-dimensional space of musical chord sequences by parameterizing the variation among chord sequences in popular music. We provide a novel user interface that exposes these learned parameters as control axes, and we propose two automatic approaches for defining these axes. One approach is based on a novel clustering procedure, the other on principal components analysis. A user study compares our approaches for defining control axes both to each other and to an approach based on manually-assigned genre labels. Results show that our automatic methods for defining control axes provide a subjectively better user experience than axes based on manual genre labeling.

We propose a method to alleviate the problem of referential granularity for Japanese zero pronoun resolution. We use dictionary definition sentences to extract ‘representative’ arguments of predicative definition words; e.g. ‘arrest’ is likely to take police as the subject and criminal as its object. These representative arguments are far more informative than ‘person’ that is provided by other valency dictionaries. They are auto-extracted using both Shallow parsing and Deep parsing for greater quality and quantity. Initial results are highly promising, obtaining more specific information about selectional preferences. An architecture of zero pronoun resolution using these representative arguments is described.