Educational robots prove effective in children’s learning, but designing robots that support parent-child dialogue during home learning remains unclear. This study proposes a semi-autonomous para-operated robot that parents partially control while engaging with their child. We hypothesized that unlike autonomous robots which may reduce parental engagement [1], this approach would increase parental involvement through active operation. We conducted an experiment with parent-child pairs comparing no robot, autonomous robot, and the proposed semi-autonomous robot conditions. Results showed the proposed robot significantly increased parental utterances without increasing parental burden, demonstrating its effectiveness in facilitating parent-child home learning dialogue.

In personality trait analysis, self-evaluation through questionnaire methods is widely used due to ease of implementation and analysis. However, respondents can easily provide false evaluation scores based on their intentions. This study proposes a method to obtain truthful evaluation scores from dialogue records through dialogue simulation with autonomous agents for questionnaire items. Since dialogue requires immediate responses and detailed explanations, we hypothesized this method would be less prone to deception compared to traditional questionnaire methods. We developed a comprehensive system using Large Language Models that handles everything from setting dialogue situations based on questionnaire items to conducting dialogues with autonomous agents, and automatically evaluating dialogue records. In preliminary experiments, we found that evaluation scores from the proposed method when participants were prompted to lie were close to truthful questionnaire responses when participants were prompted to answer truthfully. These findings suggest the effectiveness of psychological measurement through dialogue with autonomous agents.

Agents that migrate across different devices can provide seamless support for users in various contexts. However, little is known about how users determine whether a migrating agent remains the same individual across different devices. To explore the key factors influencing users’ perception of agent continuity before and after migration, we conducted two experiments. In Experiment 1, we examined how users’ perception is influenced when two visually identical agents differ in any of the three factors: voice, memory, or speaking style. Experiment 2 tested whether maintaining the factors identified as critical in Experiment 1 would enable users to perceive the agent as the same individual after it migrated from a robot to headphones. Results suggest that while changes in the factors can disrupt perceived continuity, preserving them enables consistent identity perception before and after migration, even when the agent’s appearance changes.

Co-located operated robots, where an operator, a robot, and a user interact in the same physical space, are an emerging paradigm in human-robot interaction. This para-operation approach aims to reduce operator workload and enhance communication. A key challenge is balancing robot autonomy with operator intent; fully manual systems lack flexibility, while fully autonomous ones risk unintended actions. To address this, we propose a semi-autonomous system that uses an augmented reality (AR) interface for intuitive control. We developed a prototype where an operator’s AR glasses display contextual action choices for the robot. A preliminary user study was conducted in a simulated retail scenario. Results from interviews showed that the desired dialogue strategy depends on the operator’s proficiency. Novices preferred robot-led interaction for guidance, while experienced users favored operator-led support for nuanced control. These findings highlight the need for adaptive dialogue strategies in AR-based para-operation, offering initial insights for this collaborative framework.

An ITACO agent is a conversational agent that migrates across multiple devices to provide user support through the most suitable embodiment while maintaining a continuous relationship with the user. However, because the agent must operate on diverse devices with unique configurations and behaviors, scaling the system significantly increases implementation complexity and development effort. To address this, we propose a platform that serves as a foundational framework to streamline ITACO agent development. Our system simplifies the process of integrating new devices and defining their behaviors, enabling scalable, efficient development of migrating agents. In this paper, we present the design and implementation of this platform and report on a preliminary user study conducted to evaluate its viability. The primary contribution of our work is the core design of a foundational system that facilitates the future development of ITACO agents.

Teleoperated customer service robots have attracted attention to improve customer service efficiency. However, operators experience boredom during long-duration operation due to monotony and idle time, leading to decreased task motivation. This study proposes and evaluates a method to reduce operator boredom through dialogue with the robot to be operated. Field experiments demonstrated that operator-robot dialogue significantly reduced boredom and contributed to maintaining task engagement during long-duration operation.

In recent years, numerous studies have been conducted on dialogue robots powered by large language models,enabling sophisticated interactions such as providing guidance and engaging in small talk. However, the interaction performance remains imperfect, and the robots sometimes cause problems during interactions. In this study, we aim to automatically detect such anomalies in human-robot interactions by creating a dataset and developing anomaly detection models. To this end, we created a dataset by manually annotating videos of in-the-wild interactions collected from our field experiment designed to test a framework of parallel conversations in which a human intervenes when a problem occurs in the interaction. Using this dataset, we trained classification models to construct anomaly detection models. We then conducted another field experiment in which the model’s detection results were presented as alerts to operators within the parallel conversation framework. The results confirmed that providing alerts on the basis of the anomaly detection model was useful for facilitating operator intervention.

Interactive systems that enable people to easily access mental health care in their daily lives are being studied. Self-instruction training, which is a kind of cognitive-behavioral therapy, is expected to be effective for general-purpose mental health care, but the difficulty for users to create self-instructional texts by themselves has hindered its widespread use. Furthermore, it is not sufficient to simply use a computer-assisted method for the training in which the computer automatically generates the self-instructional texts. This is because previous studies have shown that users do not feel self-instructional texts that are not created by themselves are their thoughts, and consequently cannot internalize the content of the texts, reducing the effectiveness of the training. In this study, we focused on the interface design that facilitates users to internalize automatically generated self-instructional texts, toward the realization of a self-instructional training support system. We proposed an interface that shows the real-time captured video of a user's face like a mirror and superimposes a speech bubble containing a self-instructional text to be connected to the user's face and confirmed the interface facilitates the user's internalization of the text through online experiments. The contribution of this paper is to provide a novel interface design for computer-assisted self-instructional training.

This study investigates the effects of both operator and avatar gender on the perceived ease of communication and creepiness of the avatars. Utilizing a mixed-methods 2 by 2 approach (male operator, female operator; male avatar, female avatar), we conducted an experiment involving 148 participants. The results indicate significant variations in perceived avatar impressions influenced by the gender alignment between the operator and the avatar. Specifically, female-operator-female-avatar group rated as the easiest to talk to, most interesting and least in the creepiness metric. These results suggest that for low-friction conversational encounters using female-operator-female-avatars might be most suitable.

Cybernetic avatars (CAs), robots or CG agents operated remotely by humans, are increasingly used for remote communication, offering potential for applications such as telework. However, the usability of these systems, particularly regarding the operator’s sense of agency (SoA) and sense of ownership (SoO), remains underexplored. We conducted an experiment using a commercially employed CA system. The results showed that operators can experience SoA and SoO over on-screen CG avatars, supporting the system’s basic usability. Additionally, the results suggest that limiting operator hand movements might better maintain SoA in environments with a delay in the reflection of operators’ movement on CAs. These findings have preliminary implications for CA system usage, highlighting the need to consider trade-offs between operation and system responsiveness.

Teleoperated social robots have been widely studied and employed, yet their operation interfaces provide limited information to the operator, causing difficulties in grasping the dialogue situation compared to face-to-face interactions. Particularly with conventional interfaces, operators face challenges with vocal adaptation due to the inability to assess the distance between the robot and the interlocutor. Consequently, operators struggle to determine the appropriate volume level for clear communication, leading to insecurity about their vocalization’s effectiveness. In this study, we propose a novel approach to address this issue: a voice volume gauge that enables operators to adjust their voice volume by displaying the difference between their current volume and the estimated optimal volume on the interface. We implemented the gauge and conducted a subjective evaluation experiment, which demonstrated no usability concerns for operators and a reduction in their insecurity levels. Furthermore, preliminary objective evaluation results based on speech analysis suggest that utilizing the gauge may bring operators’ vocalizations closer to those in face-to-face dialogues. The contribution of this paper lies in providing a simple interface design method to aid in solving the vocal adaptation problem for teleoperated robot applications.

Various customer service robots’ teleoperation interfaces (I/Fs) have been developed for human-robot collaboration. However, a previous study has indicated a lack of operator enjoyment. This study aims to design an I/F that elicits operator enjoyment and identifies the factors that contribute to this enjoyment. We developed two I/Fs: a high degree of freedom I/F and a gradual flexibility degree of freedom I/F based on gamification to elicit operator enjoyment. Using these I/Fs, we conducted a field study in a real shopping mall to investigate the operators’ experiences. As a result, the operators in this experiment greatly enjoyed using our I/Fs. The results showed that our I/Fs could elicit operator enjoyment with the following three I/F factors suggested as potentially important design guidelines: ease of use through anonymity, moderate restriction of freedom, and sharing experiences with a group of operators.

Dialogue robots powered by large language models can generate advanced utterances. However, the interaction performance is not yet perfect, and the robots sometimes cause problems during interactions. In this study, to detect anomalies in human-robot interactions, we created a dataset and constructed anomaly detection models. For the dataset creation, we collected videos of human-robot interactions in a framework where humans intervene when a dialogue breakdown occurs and labeled the scenes where humans intervened as anomalies. Using this dataset, we built classification models and deep metric learning models utilizing encoders for video, audio, and multimodal information. The results showed that we could successfully train the models and achieve an accuracy and F1-score of over 80%. The performance of the deep metric learning models surpassed that of the classification models, thus demonstrating the importance of separating the differences between classes. We also clarified the importance of audio information.

While research and applications of operated social robots that can be used as operators’ avatars have been conducted, a para-operated social robot has also been proposed, in which the robot and its operator exist in the same space, and the operator, robot, and interlocutor can interact with each other. From the findings of human-human-robot interaction research, it is known that the robot’s behavior influences the interpersonal relationships of the people involved in the dialog. However, there is no knowledge of how the operator is perceived by the interlocutor when the operations by the operator are unveiled. This paper reports preliminary results from two online surveys conducted to investigate the following question considering the robot-operator distance; Does the impression of the operator improve in unveiled para-operation when the operator operates the robot and has it say favorable utterances? The results of the two video surveys conducted at different distances between the robot and the operator show that the impression of the operator was improved by the robot’s favorable humorous utterances when the distance was close, but not when the distance was further apart. We consider that the findings contribute to the potential application of unveiled para-operated robots and suggest the importance of considering the distance factor in the influence of robots on human-human relationships.

In recent years, a framework of parallel conversations has been proposed to facilitate efficient conversations through cooperation between humans and dialogue systems. This approach aims to enable simultaneous conversations with multiple users by enabling the system to handle basic conversation and human operators to intervene when problems arise in the system’s conversation. Previous studies on parallel conversations have primarily focused on delegating simple exchanges such as greetings and acknowledgments to the system, with humans taking over for more complex interactions like providing guidance. Recent advancements in large language models may change this situation, enabling dialogue systems to engage in more advanced interactions. In this study, to examine which interventions will be made when large language models are utilized, we placed six dialogue robots based on large language models in an actual facility and conducted a field experiment involving parallel conversations for about a month. Our analysis of the collected data on dialogues and interventions showed that the most frequent interventions were made for supporting interactions when the system failed to react to the user utterances, indicating the limitations of using large language models alone and clarifying our next steps for facilitating smoother parallel conversations.

Dialogue systems are widely utilized in chatbots and call centers. However, it is often difficult for such systems to deliver fully autonomous dialogue. For users to have a better dialogue experience, a framework for human-system collaborative dialogue is proposed in which a human operator takes over the dialogue when needed, engaging in conversation with the user instead of the system (we call this process intervention). Operators join the dialogue in the middle; therefore, it is believed that dialogue summarization can be helpful for interventions. However, it is currently unclear whether dialogue summarization is actually useful. Therefore, in this study, we aim to investigate the usefulness of dialogue summaries for interventions through a field experiment conducted at an actual facility combining an aquarium and a zoo. The results of the field experiment revealed that dialogue summaries were more useful for intervention than dialogue history. Furthermore, we found no differences in the word categories included in the operator utterances during interventions irrespective of whether the dialogue history or dialogue format summary was presented to the operators, suggesting that dialogue format summary has content similar to that of dialogue history but improves the usefulness in intervention.