Controlling multiple agents in parallel is an important challenge in human-robot interaction, with growing applications in remote collaboration, telepresence, and multi-agent coordination. However, empirical work on distributed embodiment remains limited, particularly in scenarios where independently controlled avatars operate in distinct environments. To address this gap, we propose the concept of Dual Body Sensation (DBS): a novel theoretical construct of distributed embodiment, describing a state in which a single operator simultaneously experiences embodiment for two independent robots in distinct environments during their control. We conducted a within-subject experiment to investigate the cognitive feasibility of DBS using a $2\times 3$ factorial design that varied visual presentation (independent view, superimposed view) and tactile presentation (no tactile, bilateral tactile, unified tactile). The results indicated that DBS emerged across all conditions and was significantly enhanced under the unified tactile condition. The combination of the superimposed view and unified tactile conditions also increased the sense of body ownership and the sense of agency, while the superimposed view condition reduced cybersickness and cognitive load. These findings provide empirical support for DBS and inform the design of future multi-agent interaction systems that support flexible distribution of bodily control and awareness.

This paper proposes RIMER, a dialogue facilitation system specifically designed to support situation-aware remote reminiscence therapy using a shoulder-mounted robot. Reminiscence therapy is a psychological intervention that promotes cognitive improvement by encouraging individuals to recall and talk about their past experiences. Traditional reminiscence therapy typically involves face-to-face conversations guided by pre-selected photographs. In contrast, RIMER dynamically captures a local companion’s current surroundings through a camera mounted on the robot and generates situated questions based on the observed scene and dialogue history. This allows a remote therapy recipient to naturally recall memories associated with what they are presently seeing and recent conversations, enabling more spontaneous and contextually relevant reminiscence. The situated therapy is conducted remotely, without the need for physical co-presence. To evaluate the effectiveness of RIMER, we conducted a comparative study with three groups: one using RIMER with reminiscence support, one with facilitation but without reminiscence, and one with no facilitation. We measured dialogue volume during the sessions and analyzed the effects of each condition on facilitation and memory recall. Results showed that the group using RIMER exhibited more conversation and stronger memory recall than the other groups.

This workshop aims to support early-career researchers in the field of Human-Agent Interaction (HAI), especially those from non-native English-speaking backgrounds. Despite increasing global engagement, many talented researchers face challenges in presenting their work internationally due to language barriers and limited opportunities. By providing a platform to present research in English in a supportive and peer-driven environment, this workshop will foster accessibility, visibility, and confidence among participants. Presentations will be followed by extended and moderated Q&A sessions to encourage inclusive discussion. The workshop also invites guest talks from early-career researchers who exemplify innovative HAI research, promoting role models for the next generation. Overall, the workshop seeks to lower entry barriers and empower underrepresented voices in the global HAI community.

Communication robots have the potential to contribute to effective human-XAI interaction as an interface that goes beyond textual or graphical explanations. However, it is not clear how we can develop an adaptive strategy to use a robot’s physical and vocal expressions depending on the context in dynamic interactions. This paper proposes a method for a communication robot to decide where to emphasize XAI-generated explanations with physical expressions. In the method, a user model predicts the effect of emphasizing certain points on a user and aims to minimize the expected difference between predicted user decisions and AI-suggested ones. We conducted a user study to investigate how emphasis selection with our method affects the performance of user decisions. The results suggest that our method guides a part of users to better decisions when the performance of the AI suggestion is high.

Explainable AI (XAI) has been developed to make AI understandable to humans by providing explanations of its outputs. However, multiple types of AI explanations displayed on a screen could distort users’ trust in AI and their decision to rely on it. This could lead to poor task performance. In this study, we experimentally investigated the influence of AI explanations on trust and acceptance of AI results using a visual task. As a result, we found that participants increased their trust and acceptance of AI results with multiple types of explanations even though this did not improve task performance. These results, showing over-trust and over-reliance in human-agent interaction, were discussed along with cognitive load and cognitive bias caused by XAI.

This paper addresses the challenge of selecting explanations for XAI (Explainable AI)-based Intelligent Decision Support Systems (IDSSs). IDSSs have shown promise in improving user decisions through XAI-generated explanations along with AI predictions, and the development of XAI made it possible to generate a variety of such explanations. However, how IDSSs should select explanations to enhance user decision-making remains an open question. This paper proposes X-Selector, a method for selectively presenting XAI explanations. It enables IDSSs to strategically guide users to an AI-suggested decision by predicting the impact of different combinations of explanations on a user’s decision and selecting the combination that is expected to minimize the discrepancy between an AI suggestion and a user decision. We compared the efficacy of X-Selector with two naive strategies (all possible explanations and explanations only for the most likely prediction) and two baselines (no explanation and no AI support) in virtual stock-trading support scenarios. The results suggest the potential of X-Selector to guide users to AI-suggested decisions and improve task performance under the condition of a high AI accuracy.

This study investigated how a robot head motion towards an AI attention heatmap during a visual identification task influences a human user’s trust in eXplainable AI (XAI). The findings revealed that the robot head motion presented in a video increased the user’s acceptance of AI-generated results compared to the robot eye gaze displayed in a static image with or without the AI attention heatmap. However, displaying the heatmap improved task performance more than displaying no heatmap with or without the robot. Overall, these results suggest a possibility that showing a robot head motion towards an AI attention heatmap in a movie can serve as an interpretable XAI for visual tasks.

This study investigated how displaying an AI attention heatmap influences human acceptance of the AI’s suggestions in accordance with the interpretability of the heatmap. We conducted an experiment using a visual task where the participants were required to decide whether to accept or reject an AI’s suggestions. The participants could see the suggestions with an AI attention heatmap, the heatmap with the AI pointing to it (displayed as a laser dot cursor), the heatmap with a robot pointing (a robot using a stick to point to the AI heatmap displayed on a tablet), or no heatmap. The experimental results revealed that human acceptance of AI suggestions differed depending on the interpretability of the heatmap, especially when the heatmap was displayed with AI pointing. Also, additional analysis revealed an effect on acceptance due to the AI pointing to the heatmap that was found only in a high-task difficulty situation. An AI pointing to its attention heatmap is considered to trigger people to reason about particular AI processes and accept its suggestions. This study showed that an AI pointing to its attention heatmap could be used to control human behaviors in human-agent interaction.

Transparency in decision-making is an important factor for AI-driven autonomous systems to be trusted and relied on by users. Studies in the field of visual information processing typically attempt to make an AI system's behavior transparent by showing bounding boxes or heatmaps as explanations. However, it has also been found that an excessive amount of explanations sometimes causes information overload and brings negative results. This paper proposes SmartBBox, a method for reducing the number of bounding boxes to show while maintaining human reliance on an AI. It infers if each bounding box is worth showing by predicting its effect on human reliance. SmartBBox can autonomously learn to decide whether to show bounding boxes from humans' usage data. We implemented and tested SmartBBox in an autonomous driving scenario in which a human continuously decides whether to rely on an autonomous driving system while observing the dynamic results of object detection by the system. The results suggest that SmartBBox can reduce bounding boxes 64.8% on average from object recognition results while keeping human reliance at the same level as in the case where all the bounding boxes are presented.

This study investigated how displaying a robot’s attention heatmap while the robot pointing gesture at it influences human trust and acceptance of its outputs. We conducted an experiment using two types of visual tasks. In these tasks, the participants were required to decide whether to accept or reject the answers of an AI or robot. The participants could see the answers with an AI attention heatmap, the heatmap with AI pointing (displayed as a laser dot cursor), a robot attention heatmap with robot pointing (pointing at a certain location on the heatmap displayed on a tablet with a stick), or no heatmap. The experimental results revealed that the AI and robot pointing at their attention heatmaps lowered the participants’ acceptance of their answers when the heatmaps had low interpretability in a more difficult task. Also, the robot pointing at the heatmaps showed the possibility of increasing acceptance of its answer when the heatmaps had high interpretability in a more difficult task. In addition, the acceptance of the robot’s answers correlated with emotional trust in the robot. This study demonstrates that a robot pointing gesture at its attention heatmap could be used to control human behaviors and emotional trust in human-robot interactions.

When people are talking together in front of digital signage, advertisements that are aware of the context of the dialogue will work the most effectively. However, it has been challenging for computer systems to retrieve the appropriate advertisement from among the many options presented in large databases. Our proposed system, the Conversational Context-sensitive Advertisement generator (CoCoA), is the first attempt to apply masked word prediction to web information retrieval that takes into account the dialogue context. The novelty of CoCoA is that advertisers simply need to prepare a few abstract phrases, called Core-Queries, and then CoCoA automatically generates a context-sensitive expression as a complete search query by utilizing a masked word prediction technique that adds a word related to the dialogue context to one of the prepared Core-Queries. This automatic generation frees the advertisers from having to come up with context-sensitive phrases to attract users’ attention. Another unique point is that the modified Core-Query offers users speaking in front of the CoCoA system a list of context-sensitive advertisements. CoCoA was evaluated by crowd workers regarding the context-sensitivity of the generated search queries against the dialogue text of multiple domains prepared in advance. The results indicated that CoCoA could present more contextual and practical advertisements than other web-retrieval systems. Moreover, CoCoA acquired a higher evaluation in a particular conversation that included many travel topics to which the Core-Queries were designated, implying that it succeeded in adapting the Core-Queries for the specific ongoing context better than the compared method without any effort on the part of the advertisers. In addition, case studies with users and advertisers revealed that the context-sensitive advertisements generated by CoCoA also had an effect on the content of the ongoing dialogue. Specifically, since pairs unfamiliar with each other more frequently referred to the advertisement CoCoA displayed, the advertisements had an effect on the topics about which the pairs spoke. Moreover, participants of an advertiser role recognized that some of the search queries generated by CoCoA fit the context of a conversation and that CoCoA improved the effect of the advertisement. In particular, they learned how to design of designing a good Core-Query at ease by observing the users’ response to the advertisements retrieved with the generated search queries.

When a user manipulates a system, a user input through an interface, or an operation, is converted to the user’s intended action according to the mapping that links operations and actions, which we call “operation mapping”. Although many operation mappings are created by designers assuming how a typical user would operate the system, the optimal operation mapping may vary from user to user. The designer cannot prepare in advance all possible operation mappings. One approach to solve this problem involves autonomous learning of an operation mapping during the operation. However, existing methods require manual preparation of scenes for learning mappings. We propose advantage mapping, which enables the efficient learning of operation mappings. Working from the idea that scenes in which the user’s desired action is predictable are useful for learning operation mappings, advantage mapping extracts scenes according to the magnitude of entropy in the output of the action value function acquired from reinforcement learning. In our experiment, the user’s ideal operation mapping was more accurately obtained from the scenes selected by advantage mapping than from learning through actual play.

In this work, we present a system that sequentially generates advertisements within the context of a dialogue. Advertisements tailored to the user have long been displayed on the digital signage in stores, on web pages, and on smartphone applications. Advertisements will work more effectively if they are aware of the context of the dialogue between the users. Creating an advertising sentence as a query and searching the web by using that query is one way to present a variety of advertisements, but there is currently no method to create an appropriate search query for the search in accordance with the dialogue context. Therefore, we developed a method called the Conversational Context-sensitive Advertisement generator (CoCoA). The novelty of CoCoA is that advertisers simply need to prepare a few abstract phrases, called Core-Queries, and then CoCoA dynamically transforms the Core-Queries into complete search queries in accordance with the dialogue context. Here, “transforms” means to add words related to the context in the dialogue to the prepared Core-Queries. The transformation is enabled by a masked word prediction technique that predicts a word that is hidden in a sentence. Our attempt is the first to apply masked word prediction to a web information retrieval framework that takes into account the dialogue context. We asked users to evaluate the search query presented by CoCoA against the dialogue text of multiple domains prepared in advance and found that CoCoA could present more contextual and effective advertisements than Google Suggest or a method without the query transformation. In addition, we found that CoCoA generated high-quality advertisements that advertisers had not expected when they created the Core-Queries.

User interfaces have been designed to fit typical users and their usage styles as assumed by designers. However, it is impossible to cover all the possible use cases. To address this problem, we propose$Q$-Mapping, which is a method for user interfaces to acquire the operation mapping, or mapping from user operations to their effects.$Q$-Mapping has an advantage over previous techniques in that it can acquire operation mapping interactively. The core idea of$Q$-Mapping is that what a user selects as an ideal action has a tendency to be the same as the action that has the highest$Q$-value. On the basis of this concept, we defined the operation-action value function, which can be calculated from the value that a user expects to gain when a particular mapping is given in that state and is updated each time an operation occurs. We conducted a simulation experiment and a user study to investigate the$Q$-Mapping performance and the effects of the acquisition of interactive operation mapping. The simulation results showed that the changeability of operation mapping could be controlled by a coefficient called the balancing parameter. As for the user study, we found that$Q$-Mapping with a balancing parameter that decays with time was able to acquire operation mapping that was easy for users to understand. These results demonstrate the importance of balancing consistency and adaptability in the interactive acquisition of operation mapping.

To create dialogue systems that provide information a user needs to know at an opportune moment, it is important to infer the user’s mental states such as his/her beliefs and desires. There are two types of study on inferring beliefs and desires: one type infers them from actions and the other infers them from the content of utterances. However, a method to infer beliefs and desires from both kinds of inference in an integrated way has not yet been established. In this paper, we propose Multimodal Inference of Mind Simultaneous Contextualization and Interpreting (MIoM SCAIN), a system for sequentially inferring users’ beliefs and desires on the basis of their walking behaviors and the content of their utterances. In our evaluation, we compared inferences of MIoM SCAIN with those of baselines that use either walking behaviors or the content of utterances. MIoM SCAIN’s predictions showed more correlation with subjective judgements compared with the baselines, indicating that the inference of beliefs and desires from both walking behaviors and utterance content is possible.

Building an interactive artificial intelligence that can ask questions about the real world is one of the biggest challenges for vision and language problems. In particular, goal-oriented visual dialogue, where the aim of the agent is to seek information by asking questions during a turn-taking dialogue, has been gaining scholarly attention recently. While several existing models based on the GuessWhat?! dataset [10] have been proposed, the Questioner typically asks simple category-based questions or absolute spatial questions. This might be problematic for complex scenes where the objects share attributes, or in cases where descriptive questions are required to distinguish objects. In this paper, we propose a novel Questioner architecture, called Unified Questioner Transformer (UniQer), for descriptive question generation with referring expressions. In addition, we build a goal-oriented visual dialogue task called CLEVR Ask. It synthesizes complex scenes that require the Questioner to generate descriptive questions. We train our model with two variants of CLEVR Ask datasets. The results of the quantitative and qualitative evaluations show that UniQer outperforms the baseline.

Most of agents that learn policy for tasks with reinforcement learning (RL) lack the ability to communicate with people, which makes human-agent collaboration challenging. We believe that, in order for RL agents to comprehend utterances from human colleagues, RL agents must infer the mental states that people attribute to them because people sometimes infer an interlocutor's mental states and communicate on the basis of this mental inference. This paper proposes PublicSelf model, which is a model of a person who infers how the person's own behavior appears to their colleagues. We implemented the PublicSelf model for an RL agent in a simulated environment and examined the inference of the model by comparing it with people's judgment. The results showed that the agent's intention that people attributed to the agent's movement was correctly inferred by the model in scenes where people could find certain intentionality from the agent's behavior.

In interactions, estimating how much information is being shared between participants is one of the crucial aspects that make the interaction more lively and enhance each participant's sense of understanding of the others. In this paper, we propose a model to estimate how much information is being shared between participants in a conversation. In the proposed model, we considered not only the content of the utterance but also the timing of the utterance. To verify the validity of our model, we implemented a simulator of a party game called Word Wolf, which requires information sharing estimation as the part of the game, and simulated the participants' behavior. Through the simulation, we showed that utterance timing is an important backchannel when estimating information sharing.

In cooperation, the workers must know how co-workers behave. However, an agent's policy, which is embedded in a statistical machine learning model, is hard to understand, and requires much time and knowledge to comprehend. Therefore, it is difficult for people to predict the behavior of machine learning robots, which makes Human Robot Cooperation challenging. In this paper, we propose Instruction-based Behavior Explanation (IBE), a method to explain an autonomous agent's future behavior. In IBE, an agent can autonomously acquire the expressions to explain its own behavior by reusing the instructions given by a human expert to accelerate the learning of the agent's policy. IBE also enables a developmental agent, whose policy may change during the cooperation, to explain its own behavior with sufficient time granularity.