Due to recent advances in large language models and robotics, social robots will potentially play an important role in people’s daily lives soon, and are expected to improve dynamic multi-party group discussions in social scenarios. In this paper, we developed a system to assist dynamic group discussion with our social robot Haru. Our system is composed of three modules: a Dialogue Assistance module via integrating Haru with large language models which facilitates Haru to be an embodied chatbot; a Balancing and Welcoming Behavior module to improve users’ engagement and welcome new users to join the discussion with verbal behaviors; an Autonomous Eye Gazing module to show politeness during group discussion, e.g., gazing to the talking user or the less-engaging user to encourage her, looking to the new comer when she joins the discussion, gazing via eyeball movement when the current speaking user is close to the previous one. The autonomous eye gazing behavior was first trained via deep reinforcement learning in simulation and transferred to physical Haru in the real world. Results of our user study with 50 subjects show the significant performance of our system in assisting dynamic group discussion.

The development and integration of robots capable of expressing gaze directionality through eye-head movements are crucial for effective human-robot interaction, especially for those with eye designs on 2D screens. Our proposed mutual eye-head gaze model aligns eye movements with head/body rotation, incorporating an attention engine for estimating the most saliency location, and a retina-fovea engine for precise gaze alignment. Additionally, the eye-head engine controls head movements, enhancing the robot’s ability to perform responsive coordinated eye-head gaze behaviors. This improvement leads to enhanced human subjective perception of direct gaze from the robot, ultimately holding potential for advancing human-robot interaction in social dynamics and human-centered robot development research.

This work presents a framework for a visual-auditory attention-driven robot eye-head gaze movement, which combines visual and auditory inputs to determine the direction of gaze movement for a social robot. The framework computes the most salient changes in position by considering both visual and auditory cues. The proposed system was implemented on Haru, a tabletop social robot, where eye-head gaze movement was controlled using visual input from a camera positioned above the eyes and auditory input from a seven-channel microphone. This allowed for eye movement on a two-dimensional flat screen and body rotation towards the person who is speaking. This framework provides a representation of the robot’s attentional gaze that leverages both visual and auditory cues, resulting in more natural and responsive coordinated eye-head gaze movements of the social robot. The potential benefits include improved communication, increased engagement, and a stronger sense of connection with the robot.

This paper describes the development of a 3- stage signalling framework to trigger a social robot's bottom- up reactive behavior inspired by a biological model. In the first stage, low-level firing of stimuli due to external sources is constructed through perception grounding. This is followed by a saliency classifier which fires-up high level salient signals that require attention and are used to trigger the robot's reactive behavior. The whole framework evolves primarily on the knowledge ontology that defines the characteristics of the social robot and the querying mechanism that correlates the perceived stimuli with the ontology to trigger the reactive behavior. We evaluated the performance of our system with timing metrics and we achieved good results for our application.

In this paper, we explore the storytelling potential of a robot. We exploit the use of creative contents that maximize the embodied communication affordance of the empathic robot Haru. We identify the elements in storytelling such as narration, agency, engagement and education and synthesized these into the robot. Through effective design we investigated the possible answers that could leverage the limitations and the challenges in developing storytelling applications through a robotic medium. Our preliminary findings show that the use of an embodied agent such as a robot in storytelling only has meaning when its communicative affordance (i.e. embodiment, expressiveness, and other modalities) is tapped, adding new dimension to the experience. Otherwise, traditional storytelling delivery (e.g. tablet) without the use of embodiment will suffice. Hence, robots need to be performers rather than just mere props in storytelling.