Although most people interact with social robots in daily environments in a friendly manner, some do not. They sometimes destroy, insult, bully, or abuse them. Uncovering such antisocial attitudes and behaviors is a painful but critical research topic in the process of deploying social robots in real society. In this study, we describe the robot harassment we observed in a three-year exhibition with an android robot. During the exhibition, visitors talked freely with the android robot, which autonomously handled the conversation topics. Many people enjoyed chatting with it in a friendly manner, although a few behaved rudely toward it. A coder labeled 26/971 scenes as “harassment” interactions and organized them into four subcategories: unwelcome flirting, rude attitudes, unsolicited comments about appearance, and inappropriate touching. We introduce these details and discuss possible research topics necessary to reduce such harassment behaviors.

Challenges in multimodal task-oriented dialogue between humans and systems, particularly those involving audio and visual interactions, have not been sufficiently explored or shared, forcing researchers to define improvement directions individually without a clearly shared roadmap. To address these challenges, we organized a competition for multimodal task-oriented dialogue systems and constructed a large competition-based dataset of 1,865 minutes of Japanese task-oriented dialogues. This dataset includes audio and visual interactions between diverse systems and human participants. After analyzing system behaviors identified as problematic by the human participants in questionnaire surveys and notable methods employed by the participating teams, we identified key challenges in multimodal task-oriented dialogue systems and discussed potential directions for overcoming these challenges.

In human-human conversation, interpersonal consideration for the interlocutor is essential, and similar expectations are increasingly placed on dialogue systems. This study examines the behavior of dialogue systems in a specific interpersonal scenario where a user vents frustrations and seeks emotional support from a long-time friend represented by a dialogue system. We conducted a human evaluation and qualitative analysis of 15 dialogue systems under this setting. These systems implemented diverse strategies, such as structuring dialogue into distinct phases, modeling interpersonal relationships, and incorporating cognitive behavioral therapy techniques. Our analysis reveals that these approaches contributed to improved perceived empathy, coherence, and appropriateness, highlighting the importance of design choices in socially sensitive dialogue.

This study explores the potential benefits of robots having the capability to anticipate people’s mental states in an exercise context. We designed 80 utterances for a robot with associated gestures that exhibit a range of emotional characteristics and then performed a 23-person data collection to investigate the effects of these robot behaviors on human mental states during exercise. The results of cluster analysis revealed that (1) utterances with similar meanings had the same effect and (2) the effects of a certain cluster on different people depend on their emotional state. On the basis of these findings, we proposed a robotic system that anticipates the effect of utterances on the individual’s future mental state, thereby choosing utterances that can positively impact the individual. This system incorporates three main features: (1) associating the relevant events detected by sensors with a user’s emotional state; (2) anticipating the effects of robot behavior on the user’s future mental state to choose the next behavior that maximizes the anticipated gain; and (3) determining appropriate times to provide coaching feedback, using predefined rules in the motion module for timing decisions. To evaluate the proposed system’s overall performance comprehensively, we compare robots equipped with the system’s unique features to those lacking these features. We design the baseline condition that lacks these unique features, opting for periodic random selection of utterances for interaction based on the current context. We conducted a 21-person experiment to evaluate the system’s performance. We found that participants perceived the robot to have a good understanding of their mental states and that they enjoyed the exercises more and put in more effort due to the robot’s encouragement.

This study explored a tele-operation notification interface that supports a multitasking operator independently perceiving remote information and shifting their tasks. We proposed a noise-reducing auditory notification method, which artificially replicated the cocktail party effect by decreasing the volume of the background noise and increasing the volume of visitor footsteps. Experimental results show that our proposed method provided an effective interface for operators to independently notice human presence.

Human robot interaction studies have investigated how various non-verbal expressions of robots can enhance feelings of familiarity and trust in users’ and establish good relationships between users and robots. In the field of art, this approach has been taken one step further by creating robot artworks and demonstrations through which people feel as if they are experiencing heart-to-heart communication with a robot. Although robotic behavior that conveys such a feeling is useful for human coexistence and providing a sense of security and trust, no engineering methodology can yet achieve such behavior. This study attempts to explain what kind of robot behavior is connected to such feelings by analyzing demonstrations of such a robot based on information theory. We found that the intensity of these feelings can be partially explained by transfer entropy between humans and the robot’s body movements. We expect this research to clarify how to design a robot’s behavior so that it can provide heart-to-heart communication with people as well as how to construct a robot that can build solid relationships with people in daily life.

We propose a wearable tactile sensor suit, which can be regarded as tactile sensor networks, for monitoring natural body dynamics to be exploited as a computational resource for estimating the posture of a human or robot that wears it. We emulated the periodic motions of a wearer (a human and an android robot) using a novel sensor suit with a 9-channel fabric tactile sensor on the left arm. The emulation was conducted by using a linear regression (LR) model of sensor states as readout modules that predict the next wearer's movement using the current sensor data. Our result shows that the LR performance is comparable with other recurrent neural network approaches, suggesting that a fabric tactile sensor network can monitor the natural body motions, and further, this natural body dynamics itself can be used as an effective computational resource.

We developed a method to automatically generate humanlike trunk motions (neck and waist motions) of a conversational android from its speech in real time. It is based on a spring-damper dynamical model to simulate a human's trunk movement involved in speech. Differing from the existing methods based on machine learning, our system can easily modulate the motions generated due to speech patterns since the parameters in the model correspond to muscle stiffness. The experimental result showed that the android motions generated by our model could be perceived as more natural and motivate participants to talk with the android more, compared with simple copying of human motions.

We developed a tele-operated robot system where the head motions of the robot are controlled by combining those of the operator with the ones which are automatically generated from the operator's voice. The head motion generation is based on dialogue act functions which are estimated from linguistic and prosodic information extracted from the speech signal. The proposed system was evaluated through an experiment where participants interact with a tele-operated robot. Subjective scores indicated the effectiveness of the proposed head motion generation system, even under limitations in the dialogue act estimation.

We propose a huggable communication device called Hugvie that encourages children to concentrate on listening by reducing their stress and strengthening the feeling that the storyteller is close. We observed a group of preschool children who listened to a story and conclude that Hugvie has the potential to facilitate attention on stories. This indicates its usefulness to relieve the educational problem where children show disobedient or restless behavior during class. We discuss Hugvie's effect on learning and memory and potential applications to special-needs children.

We introduce a communication medium that encourages users to fall in love with their counterparts. Hugvie, the huggable tele-presence medium, enables users to feel like hugging their counterparts while chatting. In this paper, we report that when a participant talks to his communication partner during their first encounter while hugging Hugvie, he mistakenly feels as if they are establishing a good relationship and that he is being loved rather than just being liked.