We propose a proactive teleoperation support system to help high-power individuals exercise power appropriately during hierarchical interactions. In our study, seniors (high-power individuals) remotely operated an avatar robot as operators, while juniors (low-power individuals) exercised as exercisers. To ground the design, we conducted an observational study of hierarchical interactions, identifying three recurring challenges: perceived loss of power, escalation of negativity, and abandonment of power. Based on these findings, we proposed interaction policies and integrated them into a teleoperation system that provides operators with guidance for the appropriate use of power, aiming to prevent conflict and foster positive communication. We evaluated this system in a 2-hour study with 17 senior–junior pairs, comparing it to a baseline teleoperation system where operators controlled the avatar robot directly without such guidance. Results showed that our system significantly improved operator satisfaction and reduced workload. Exercisers reported greater enjoyment and acceptance, with similar exercise counts in both conditions. Interviews revealed that the system broadened operators’ communication strategies and fostered more positive, supportive environments for low-power participants.

Irony can have remarkable positive effects on human behavior. However, those effects depend on the appropriate use of irony. Existing theoretical frameworks in psychology and sociology discuss the use of irony, including its effects on emotional states and behaviors. However, these frameworks often rely on subjective intuition, rendering them impractical for robotic systems. Additionally, our investigation of prior research in human-robot interaction (HRI) reveals a lack of systematic, context-sensitive frameworks for irony usage. This gap limits the ability of robots to dynamically evaluate the effectiveness of irony and adapt its use based on user states and interaction contexts. This study introduces a novel operational framework for irony usage in HRI, addressing these gaps through dynamic, real-time guidelines that enable robots to effectively and adaptively deploy irony. Given the lack of direct support from existing theoretical frameworks, our research adopts an exploratory approach. Specifically, we conducted semi-structured interviews to investigate successful irony in interpersonal interactions that can positively impact humans, evaluating it from the perspective of those who received irony. In the investigation, 25 irony receivers were interviewed, and the essential factors in successful irony usage were extracted from those interviews. The interview results indicated that successful irony is typically used when the receiver is in an inefficient state but in a good mood and energetic. Based on these findings, guidelines for the successful use of irony by a robotic system were established. In the proposed system, the robot acts as an exercise partner, encouraging users to adhere to their ideal plans. The robot uses irony on users when they do not make sufficient efforts, but only if they are in a good mental state and energetic. In an experiment involving 22 participants, the proposed system outperformed a baseline system, where the robot used irony whenever participants exhibited inefficiency. Participants using the proposed system exercised for 59 minutes on average, compared to 48 minutes with the baseline system. Moreover, they felt that the robot with the proposed robot-behavior model better understood their states, and they also felt more enjoyment.

Although most people interact with social robots in daily environments in a friendly manner, some do not. They sometimes destroy, insult, bully, or abuse them. Uncovering such antisocial attitudes and behaviors is a painful but critical research topic in the process of deploying social robots in real society. In this study, we describe the robot harassment we observed in a three-year exhibition with an android robot. During the exhibition, visitors talked freely with the android robot, which autonomously handled the conversation topics. Many people enjoyed chatting with it in a friendly manner, although a few behaved rudely toward it. A coder labeled 26/971 scenes as “harassment” interactions and organized them into four subcategories: unwelcome flirting, rude attitudes, unsolicited comments about appearance, and inappropriate touching. We introduce these details and discuss possible research topics necessary to reduce such harassment behaviors.

Automatic robotic facial expression generation is crucial for human–robot interaction (HRI), as handcrafted methods based on fixed joint configurations often yield rigid and unnatural behaviors. Although recent automated techniques reduce the need for manual tuning, they tend to fall short by not adequately bridging the gap between human preferences and model predictions—resulting in a deficiency of nuanced and realistic expressions due to limited degrees of freedom and insufficient perceptual integration. In this work, we propose a novel learning-to-rank framework that leverages human feedback to address this discrepancy and enhanced the expressiveness of robotic faces. Specifically, we conduct pairwise comparison annotations to collect human preference data and develop the Human Affective Pairwise Impressions (HAPI) model, a Siamese RankNet-based approach that refines expression evaluation. Results obtained via Bayesian Optimization and online expression survey on a 35-DOF android platform demonstrate that our approach produces significantly more realistic and socially resonant expressions of Anger, Happiness, and Surprise than those generated by baseline and expert-designed methods. This confirms that our framework effectively bridges the gap between human preferences and model predictions while robustly aligning robotic expression generation with human affective responses.

Challenges in multimodal task-oriented dialogue between humans and systems, particularly those involving audio and visual interactions, have not been sufficiently explored or shared, forcing researchers to define improvement directions individually without a clearly shared roadmap. To address these challenges, we organized a competition for multimodal task-oriented dialogue systems and constructed a large competition-based dataset of 1,865 minutes of Japanese task-oriented dialogues. This dataset includes audio and visual interactions between diverse systems and human participants. After analyzing system behaviors identified as problematic by the human participants in questionnaire surveys and notable methods employed by the participating teams, we identified key challenges in multimodal task-oriented dialogue systems and discussed potential directions for overcoming these challenges.

In human-human conversation, interpersonal consideration for the interlocutor is essential, and similar expectations are increasingly placed on dialogue systems. This study examines the behavior of dialogue systems in a specific interpersonal scenario where a user vents frustrations and seeks emotional support from a long-time friend represented by a dialogue system. We conducted a human evaluation and qualitative analysis of 15 dialogue systems under this setting. These systems implemented diverse strategies, such as structuring dialogue into distinct phases, modeling interpersonal relationships, and incorporating cognitive behavioral therapy techniques. Our analysis reveals that these approaches contributed to improved perceived empathy, coherence, and appropriateness, highlighting the importance of design choices in socially sensitive dialogue.

This study explores the potential benefits of robots having the capability to anticipate people’s mental states in an exercise context. We designed 80 utterances for a robot with associated gestures that exhibit a range of emotional characteristics and then performed a 23-person data collection to investigate the effects of these robot behaviors on human mental states during exercise. The results of cluster analysis revealed that (1) utterances with similar meanings had the same effect and (2) the effects of a certain cluster on different people depend on their emotional state. On the basis of these findings, we proposed a robotic system that anticipates the effect of utterances on the individual’s future mental state, thereby choosing utterances that can positively impact the individual. This system incorporates three main features: (1) associating the relevant events detected by sensors with a user’s emotional state; (2) anticipating the effects of robot behavior on the user’s future mental state to choose the next behavior that maximizes the anticipated gain; and (3) determining appropriate times to provide coaching feedback, using predefined rules in the motion module for timing decisions. To evaluate the proposed system’s overall performance comprehensively, we compare robots equipped with the system’s unique features to those lacking these features. We design the baseline condition that lacks these unique features, opting for periodic random selection of utterances for interaction based on the current context. We conducted a 21-person experiment to evaluate the system’s performance. We found that participants perceived the robot to have a good understanding of their mental states and that they enjoyed the exercises more and put in more effort due to the robot’s encouragement.

This study explored a tele-operation notification interface that supports a multitasking operator independently perceiving remote information and shifting their tasks. We proposed a noise-reducing auditory notification method, which artificially replicated the cocktail party effect by decreasing the volume of the background noise and increasing the volume of visitor footsteps. Experimental results show that our proposed method provided an effective interface for operators to independently notice human presence.

Facial mimicry is crucial for human-like robots in human-robot interaction. The challenge is that the high diversity of facial expressions proposes difficulties in programming a robotic face to mimic human facial expressions using traditional methods. In this paper, we present a data-driven method to retarget human facial expressions to robotic faces without human effort. Our data collection is fully automatic, where only a robotic face and Apple ARKit are involved to sample actuator commands and record the resulting facial blendshape values. We trained a neural network that predicts blendshape values from commands, which is then used as a surrogate model to optimize command values to resemble given facial expressions. Experiments show that the proposed method has achieved lower error in terms of facial blendshape values than baselines. Moreover, the response time can be reduced to 0.2 seconds via TCP/IP through WiFi, offering great potential for real-time application. Our method is a novel framework for retargeting facial expressions to robotic faces, which can be incorporated into various human-robot interaction systems.

Human robot interaction studies have investigated how various non-verbal expressions of robots can enhance feelings of familiarity and trust in users’ and establish good relationships between users and robots. In the field of art, this approach has been taken one step further by creating robot artworks and demonstrations through which people feel as if they are experiencing heart-to-heart communication with a robot. Although robotic behavior that conveys such a feeling is useful for human coexistence and providing a sense of security and trust, no engineering methodology can yet achieve such behavior. This study attempts to explain what kind of robot behavior is connected to such feelings by analyzing demonstrations of such a robot based on information theory. We found that the intensity of these feelings can be partially explained by transfer entropy between humans and the robot’s body movements. We expect this research to clarify how to design a robot’s behavior so that it can provide heart-to-heart communication with people as well as how to construct a robot that can build solid relationships with people in daily life.

This paper presents an autonomous conversational mobile robot Butsukusa that can describe its own observations and internal states during patrolling tasks. The proposed robot can observe the surrounding environment using the recognition module for objects, humans, environment, localization, and speech and then move autonomously around an indoor living space. Interaction skills via language are required for the robot to perform in such human-centered spaces. To investigate a better communication protocol with users, we evaluate various language generation patterns based on different observations and interaction patterns. The evaluation results indicate that the importance of describing the robot's observation results and internal states, as well as the necessity of an appropriate description, depends on the situation.

We propose a wearable tactile sensor suit, which can be regarded as tactile sensor networks, for monitoring natural body dynamics to be exploited as a computational resource for estimating the posture of a human or robot that wears it. We emulated the periodic motions of a wearer (a human and an android robot) using a novel sensor suit with a 9-channel fabric tactile sensor on the left arm. The emulation was conducted by using a linear regression (LR) model of sensor states as readout modules that predict the next wearer's movement using the current sensor data. Our result shows that the LR performance is comparable with other recurrent neural network approaches, suggesting that a fabric tactile sensor network can monitor the natural body motions, and further, this natural body dynamics itself can be used as an effective computational resource.

Humans use not only verbal information but also non-verbal information in daily communication. Among the non-verbal information, we have proposed methods for automatically generating hand gestures in android robots, with the purpose of generating natural human-like motion. In this study, we investigate the effects of hand gesture models trained/designed for different speakers on the impression of the individuality through android robots. We consider that it is possible to express individuality in the robot, by creating hand motion that are unique to that individual. Three factors were taken into account: the appearance of the robot, the voice, and the hand motion. Subjective evaluation experiments were conducted by comparing motions generated in two android robots, two speaker voices, and two motion types, to evaluate how each modality affects the impression of the speaker individuality. Evaluation results indicated that all these three factors affect the impression of speaker individuality, while different trends were found depending on whether or not the android is copy of an existent person.

This paper addresses the effects of a subtle pause in reactions during human-robot touch interactions. Based on the human scientific literature, people's reaction times to touch stimuli range from 150 to 400 msec. Therefore, we decided to use a subtle pause with a similar length for reactions for more natural human-robot touch interactions. On the other hand, in the human-robot interaction research field, a past study reports that people prefer reactions from a robot in touch interaction that are as quick as possible, i.e., a 0- second reaction time is slightly preferred to 1- or 2- second reaction times. We note that since the resolution of the study's time slices was every second, it remains unknown whether a robot should take a pause of hundreds of milliseconds for a more natural reaction time. To investigate the effects of subtle pauses in touch interaction, we experimentally investigated the effects of reaction time to people's touch with a 200-msec resolution of time slices between 0 second and 1 second: 0 second, 200, 400, 600, and 800 msec. The number of people who preferred the reactions with subtle pauses exceeded the number who preferred the 0- second reactions. However, the questionnaire scores did not show any significant differences because of individual differences, even though the 400-msec pause was slightly preferred to the others from the preference perspective.

This work studies a dialogue strategy aimed at building people's motivation for talking with autonomous conversational robots. Even though spoken dialogue systems continue to develop rapidly, the existing systems are insufficient for continuous use because they fail to motivate users to talk with them. One reason is that users fail to realize that the intentions of the system's utterances are based on its values. Since people recognize the values of others and modify their own values in human-human conversations, we hypothesize that a dialogue strategy that makes users saliently feel the difference of their own values and those of the system will increase motivation for the dialogues. Our experiment, which evaluated human-human dialogues, supported our hypothesis. However, an experiment with human-android dialogues failed to produce identical results, suggesting that people did not attribute values to our android. For a conversational robot, we need additional techniques to convince people to believe a robot speaks based on its own values and opinions.

We are dealing with the problem of generating natural human-like motions during speech in android robots, which have human-like appearances. So far, automatic generation methods have been proposed for lip and head motions of tele-presence robots, based on the speech signal of the tele-operator. In the present study, we aim for extending the speech-driven motion generation methods for laughing speech, since laughter often occurs in natural dialogue interactions and may cause miscommunication if there is mismatch between audio and visual modalities. Based on analysis results of human behaviors during laughing speech, we proposed a motion generation method given the speech signal and the laughing speech intervals. Subjective experiments were conducted using our android robot by generating five different motion types, considering several modalities. Evaluation results show the effectiveness of controlling different parts of the face, head and upper body (eyelid narrowing, lip corner/cheek raising, eye blinking, head motion and upper body motion control).

The development of an android with convincingly lifelike appearance and behavior has been a long-standing goal in robotics, and recent years have seen great progress in many of the technologies needed to create such androids. However, it is necessary to actually integrate these technologies into a robot system in order to assess the progress that has been made towards this goal and to identify important areas for future work. To this end, we are developing ERICA, an autonomous android system capable of conversational interaction, featuring advanced sensing and speech synthesis technologies, and arguably the most humanlike android built to date. Although the project is ongoing, initial development of the basic android platform has been completed. In this paper we present an overview of the requirements and design of the platform, describe the development process of an interactive application, report on ERICA's first autonomous public demonstration, and discuss the main technical challenges that remain to be addressed in order to create humanlike, autonomous androids.

We developed a method to automatically generate humanlike trunk motions (neck and waist motions) of a conversational android from its speech in real time. It is based on a spring-damper dynamical model to simulate a human's trunk movement involved in speech. Differing from the existing methods based on machine learning, our system can easily modulate the motions generated due to speech patterns since the parameters in the model correspond to muscle stiffness. The experimental result showed that the android motions generated by our model could be perceived as more natural and motivate participants to talk with the android more, compared with simple copying of human motions.

The goal of this study is to develop a humanoid robot that can conduct a continuous conversation with people. Spoken dialogue systems have recently been rapidly developed, but the existing systems are insufficient for continuous use because they fail to inspire the user's motivation to talk with them. This is because a user is unable to feel that a robot possesses its own intentions, and thus the robot must acquire its own values to convey its sense of intentionality to users through its spoken communications. This paper focuses on a dialogue strategy aimed at building people's motivation under the assumption that the robot has a values-based dialogue system. People's motivation can be influenced by the intentionality as well as by the affinity of the robot. We hypothesized that there is a good disagreement/agreement ratio in a conversation to efficiently balance people's feelings of intentionality and affinity. The result of a psychological experiment using an android robot partially supported our hypothesis.

We developed a tele-operated robot system where the head motions of the robot are controlled by combining those of the operator with the ones which are automatically generated from the operator's voice. The head motion generation is based on dialogue act functions which are estimated from linguistic and prosodic information extracted from the speech signal. The proposed system was evaluated through an experiment where participants interact with a tele-operated robot. Subjective scores indicated the effectiveness of the proposed head motion generation system, even under limitations in the dialogue act estimation.

We introduce a communication medium that encourages users to fall in love with their counterparts. Hugvie, the huggable tele-presence medium, enables users to feel like hugging their counterparts while chatting. In this paper, we report that when a participant talks to his communication partner during their first encounter while hugging Hugvie, he mistakenly feels as if they are establishing a good relationship and that he is being loved rather than just being liked.

Robot avatar systems give the feeling we share a space with people who are actually at a distant location. Since our cognitive system specializes in recognizing a human, avatars of the distant people can make us strongly feel that we share space with them, provided that their appearance has been designed to sufficiently resemble humans. In this paper, we investigate the minimal requirements of robot avatars for distant people to feel their presence. Toward this aim, we give an overview of the chronological development of Dogu̅, which are human figurines made in ancient Japan. This survey of the Dogu̅ shows that the torso, not the face, was considered the primary element for representing a human. It also suggests that some body parts can be represented in a simple form. Following the development of Dogu̅, we use a conversation task to examine what kind of body representation is necessary to feel a distant person's presence. The experimental results show that the forms for the torso and head are required to enhance this feeling, while other body parts have less impact. Finally, we summarize design considerations for communication avatars.

Recent research has focused on such physical communication media as teleoperated robots, which provide a feeling of being with people in remote places. Recent invented media resemble cute animals or imaginary creatures that quickly attract attention. However, such appearances could distort tele-communications because they are different from human beings. This paper studies the effect on the speaker's personality that is transmitted through physical media by regarding appearances as a function that transmits the speaker's information. Although communication media's capability to transmit information reportedly influences conversations in many aspects, the effect of appearances remains unclear. To reveal the effect of appearance, we compared three appearances of communication media: stuffed-bear teleoperated robot, human-like teleoperated robot, and video chat. Our results show that communication media whose appearance greatly differs from that of the speaker distorts the personality perceived by interlocutors. This paper suggests that the design of the appearance of physical communication media needs to be carefully selected.

Touch is an important means for communication among humans. Sport instructors or dance teachers often use touch to adjust students' postures in a very intuitive way. Using tactile instructions appears thus to be a very appealing modality for developing humanoid robot motions as well. Spontaneous interpretation of tactile instructions given by users reveals itself to be a complex task for artificial systems. This paper describes a proof of concept system for robot motion creation based on tactile interaction. The system is interesting for two reasons. Firstly, it shows the feasibility of using tactile instructions for motion development. Secondly, it can be used as a tool for studying the way humans intuitively use touch to communicate. This, in turn, will allow the development of better algorithms for predicting the meaning of tactile instructions. Results of a pilot experiment are discussed, and a first set of features of tactile communication, yielded by the analysis of the data collected, is identified.

In order for humans and robots to engage in direct physical interaction several requirements have to be met. Among others, robots need to be able to adapt their behavior in order to facilitate the interaction with a human partner. This can be achieved using machine learning techniques. However, most machine learning scenarios to-date do not address the question of how learning can be achieved for tightly coupled, physical touch interactions between the learning agent and a human partner. This paper presents an example for such human in-the-loop learning scenarios and proposes a computationally cheap learning algorithm for this purpose. The efficiency of this method is evaluated in an experiment, where human care givers help an android robot to stand up.

A natural human-robot communication is supported by a person's interpersonal behavior for a robot. The condition to elicit interpersonal behavior is thought to be related to a mechanism to support natural communication. In the present study, we hypothesize that motion diversity produced independently of a subject's intention contributes to the human-like nature of the motions of an android that closely resembles a human being. In order to verify this hypothesis, we construct a model of motion diversity through the observation of human motion, specifically, a touching motion. Psychological experiments have shown that the presence of motion diversity in android motion influences the impression toward the android.

In order to develop a robot that has a human-like presence, the robot must be given a very human-like appearance and behavior, and a sense of perception that enables it to communicate with humans. We have developed an android robot called ldquoRepliee Q2rdquo that closely resembles a human being; however, the sensors mounted on its body are not sufficient to allow human-like communication due to factors such as sensing range and spatial resolution. To overcome this problem, we endowed the environment surrounding the android with perceptive capabilities by embedding a variety of sensors into it. This sensor network provides the android with human-like perception by constantly and extensively monitoring human activities in a less obvious manner. This paper reports on an android system that is integrated with a sensor network system embedded in the environment. A human-android interaction experiment shows that the integrated system provides relatively human-like interaction.

Communication skills are necessary for robots to get by in daily life. Previous studies have shown that the chameleon effect, which increases the likeability of a person who mimicks another during a conversation, appears not only with humans but also with CG agents. However, if we use humans, they cannot accurately mimic. In addition, if we use a CG agent, it doesnpsilat have a very humanlike appearance. It remains doubtful whether the CG agent reappears in a human-human communication. Thus, we use an android with a very humanlike appearance and approach the principle of human-human communication. In this paper, we compared mimic and non-mimic conditions when an android conversed with subjects in face-to-face. Likeability toward an android increased when the android mimicked its partner. In addition, we confirmed the possibility of using an android to explore human-human communication. These results contribute to verbal communication for robots in daily life.

In order to develop a robot working in daily situations, it is necessary to discover the principles relevant to establishing and maintaining social interaction between human and robot. One important issue is discovering how to naturally animate a robot to maintain social interaction. This study tackles the issue through implementing natural motions in the android which closely resembles those of human beings. This paper proposes a method to implement postures that look human by mapping the three-dimensional positions of a human subject body onto the android.

The "uncanny valley" must be avoided from the viewpoint of communication robot design, while this is an essential phenomenon for discovering the principles relevant to establishing social interaction between human and robot. Studying the uncanny valley allows us to explore the boundary of human-likeness. We have empirically and experimentally obtained evidence to show the uncanny valley. We have also obtained experimental evidence to suggest that the uncanny valley varies owing to the development of individuals. We refer to this variable uncanny valley as the age-dependent uncanny valley. We assume that the uncanny valley is induced by a lateral inhibition effect, and is referred to herein as the lateral inhibition hypothesis of the uncanny valley;. The present paper presents evidence concerning the uncanny valley and describes the likelihood of the present hypothesis.

One of the main aims of humanoid robotics is to develop robots that are capable of interacting naturally with people. However, to understand the essence of human interaction, it is crucial to investigate the contribution of behavior and appearance. Our group's research explores these relationships by developing androids that closely resemble human beings in both aspects. If humanlike appearance causes us to evaluate an android's behavior from a human standard, we are more likely to be cognizant of deviations from human norms. Therefore, the android's motions must closely match human performance to avoid looking strange, including such autonomic responses as the shoulder movements involved in breathing. This paper proposes a method to implement motions that look human by mapping their three-dimensional appearance from a human performer to the android and then evaluating the verisimilitude of the visible motions using a motion capture system. This approach has several advantages over current research, which has focused on copying a person's moving joint angles to a robot: (1) in an android robot with many degrees of freedom and kinematics that differs from that of a human being, it is difficult to calculate which joint angles would make the robot's posture appear similar to the human performer; and (2) the motion that we perceive is at the robot's surface, not necessarily at its joints, which are often hidden from view.

This paper explores memory-based approaches to the recognition of human behavior that relies on a database of previously categorized instances of sensory data. To overcome the curse of dimensionality, we examine two related methods that both rely on a hierarchical division of the sensory space using a decision tree. The first approach iteratively applies linear discriminant analysis to divide the sensory space in half in order to construct a binary tree for recognizing behaviors. We have verified the effectiveness of this approach for real-time behavior recognition using infrared sensors distributed in a desk environment and compared its results to those of Quinlan's C4.5. The second approach applies the well-known ID3 algorithm to the construction of a decision tree based on an information criterion. We use it to recognize browsing behavior at a video rental shop. Inferences are derived directly from the binarized pixel data of four wide-view cameras. Both systems offer behavior recognition rates in excess of 90%.

This paper proposes an interpretation system for recognizing human motion behaviors and constructing the behavior rules called Behavior Grammar. The system recognizes human motion behaviors based on the gestures, locations, directions, and distances by using a distributed omnidirectional vision system (DOVS). The DOVS consisting of multiple omnidirectional cameras is a prototype of a perceptual information infrastructure for monitoring and recognizing the real world. The sequences of interpreted behaviors are represented as a behavior graph to extract behavior rules. This paper shows how the system realizes robust and real-time visual recognition based on View and Motion based Aspect Models (VAMBAM) and the resultant behavior graph.

Visual attention is one of the key issues for robots to accomplish the given tasks, and the existing methods specify the image features and attention control scheme in advance according to the task and the robot. However, in order to cope with environmental changes and/or task variations, the robot should construct its own attention mechanism. As the first step towards selective attention, this paper presents a method for image feature generation by visio-motor map learning for a mobile robot. The teaching data construct the visio-motor mapping that constrains the image feature generation and state vector estimation as well. The resultant image feature and state vector are nothing but task-oriented. The method is applied to indoor navigation and soccer shooting tasks, and a discussion is given.

Most of existing robot learning methods have considered the environment where their robots work unchanged, therefore, the robots have to learn from scratch if they encounter new environments. This paper proposes a method which adapts robots to environmental changes by efficiently transferring a learned policy in the previous environments into a new one and effectively modifying it to cope with these changes. The resultant policy (a part of state transition map) does not seem optimal in each individual environment, but may absorb the differences between multiple environments. We apply the method to a mobile robot navigation problem of which task is to reach the target avoiding obstacles based on uninterpreted sonar and visual information. Experimental results show the validity of the method and discussion is given.