Daily life support robots must interpret ambiguous verbal instructions involving demonstratives such as "Bring me that cup," even when objects or users are out of the robot’s view. Existing approaches to exophora resolution primarily rely on visual data and thus fail in real-world scenarios where the object or user is not visible. We propose Multimodal Interactive Exophora resolution with user Localization (MIEL), which is a multimodal exophora resolution framework leveraging sound source localization (SSL), semantic mapping, visual-language models (VLMs), and interactive questioning with GPT-4o. Our approach first constructs a semantic map of the environment and estimates candidate objects from a linguistic query with the user’s skeletal data. SSL is utilized to orient the robot toward users who are initially outside its visual field, enabling accurate identification of user gestures and pointing directions. When ambiguities remain, the robot proactively interacts with the user, employing GPT-4o to formulate clarifying questions. Experiments in a real-world environment showed results that were approximately 1.3 times better when the user was visible to the robot and 2.0 times better when the user was not visible to the robot, compared to the methods without SSL and interactive questioning. The project website is https://emergentsystemlabstudent.github.io/MIEL/.

ABSTRACT This paper addresses the challenge of generating natural, interactive motion for embodied agents in social roles. We argue that such motion requires three key properties: Adaptability to dynamic environments and user interactions, switchability between different motions, and directability for expressing intentions and emotions. We propose a novel method that leverages overlapping minimum jerk trajectories to represent the motion of body parts, achieving both switchability and adaptability. Switchability is enabled by the ability to begin transitioning between minimum jerk trajectories at any moment, allowing the agent to smoothly change its motion. Adaptability is achieved by defining trajectory goals within a goal coordinate system relative to the target object or person. This dynamic goal ensures that motions remain contextually relevant as targets move. By representing motion as a sequence of minimum jerk trajectory goals, our approach shares similarities with keyframe animation, offering potential for directability. We demonstrate the efficacy of our method through various motion examples, evaluating its adaptability and switchability. Results indicate that our approach holds promise for generating natural and interactive agent behavior, albeit with some limitations in switchability requiring further investigation.

Robots that assist humans in their daily lives should be able to locate specific instances of objects in an environment that match a user’s desired objects. This task is known as instance-specific image goal navigation (InstanceImageNav), which requires a model that can distinguish different instances of an object within the same class. A significant challenge in robotics is that when a robot observes the same object from various 3D viewpoints, its appearance may differ significantly, making it difficult to recognize and locate accurately. In this paper, we introduce a method called SimView, which leverages multi-view images based on a 3D semantic map of an environment and self-supervised learning using SimSiam to train an instance-identification model on-site. The effectiveness of our approach was validated using a photorealistic simulator, Habitat Matterport 3D, created by scanning actual home environments. Our results demonstrate a 1.7-fold improvement in task accuracy compared with contrastive language-image pre-training (CLIP), a pre-trained multimodal contrastive learning method for object searching. This improvement highlights the benefits of our proposed fine-tuning method in enhancing the performance of assistive robots in InstanceImageNav tasks. The project website is https://emergentsystemlabstudent.github.io/MultiViewRetrieve/.

Daily life support robots in the home environment interpret the user's pointing and understand the instructions, thereby increasing the number of instructions accomplished. This study aims to improve the estimation performance of pointing frames by using speech information when a person gives pointing or verbal instructions to the robot. The estimation of the pointing frame, which represents the moment when the user points, can help the user understand the instructions. Therefore, we perform pointing frame estimation using a time-series model, utilizing the user's speech, images, and speech-recognized text observed by the robot. In our experiments, we set up realistic communication conditions, such as speech containing everyday conversation, non-upright posture, actions other than pointing, and reference objects outside the robot's field of view. The results showed that adding speech information improved the estimation performance, especially the Transformer model with Mel-Spectrogram as a feature. This study will lead to be applied to object localization and action planning in 3D environments by robots in the future. The project website is https://emergentsystemlabstudent.github.io/PointingImgEst/.

With a spread of inexpensive and easy-to-use motion tracking methods such as cameras and trackers for virtual realities (VRs), real-time motion-tracked avatars are becoming increasingly common in virtual environments, particularly social VRs, virtual performers (e.g., virtual YouTubers), and VR games. These applications frequently involve interactions among multiple avatars or between avatars and objects for communication or gameplay. However, at present, most applications do not sufficiently consider the effects of contact for avatars, thus leading to penetration or unnatural behavior. To achieve natural avatar motion in such scenarios, the player must perform as if contact has occurred, even though, in reality, there is no contact with the player’s body. While physics simulation can solve the contact issue, the basic use of physics simulation causes tracking delay. We therefore solve this tracking delay problem by employing Featherstone’s algorithm, a reduced-coordinate method for forward dynamics, and by utilizing the Newton-Euler formulation for inverse dynamics to compute tracking forces and torques. To evaluate the delay, we measured the difference between the joint rotations of the simulated model and the input joint rotations. The evaluation with our method indicates the difference is less than 1.4 micro radians for all joints in real time with a small computational cost. Moreover, accurate tracking enables the fixation of both feet. The proposed method provides accurate tracking and soft reactions to contact.

To enable a robot to provide support in a home environment through human-robot interaction, exophora resolution is crucial for accurately identifying the target of ambiguous linguistic instructions, which may include a demonstrative, such as “Take that one”. Unlike endophora resolution, which involves predicting the corresponding word from given sentences, exophora resolution necessitates comprehensive utilization of external real-world information to identify and disambiguate the target from the on-site environment. This study aims to resolve ambiguity in language instructions containing a demonstrative through exophora resolution, utilizing real-world multimodal information. The robot accomplishes this by using three types of information: 1) object categories, 2) demonstratives, and 3) pointing, as well as knowledge about objects obtained from the robot’s pre-exploration of the environment. We evaluated the accuracy of object identification under multiple conditions by identifying a user-indicated object in a field that mimics a home environment. Our results demonstrate that our proposed method of exophora resolution using multimodal information can identify the target with two to three times higher accuracy than baseline methods in cases where information is missing.

In this study, we aim to improve the experience of virtual reality (VR) shooting games by employing a 3D haptic guidance method using necklace-type and belt-type haptic devices. Such devices help to modulate the vibrations generated by and synchronized with musical signals according to the azimuth and height of a target in 3D space, which is expected to improve the gaming experience by providing 3D guidance and enhancing the music-listening experience. For the first step, we evaluated the method's potential by conducting an experiment in which participants were asked to shoot a randomly spawned target moving in 3D VR space. The experiment applied four conditions: the proposed method (Haptic), displaying 3D radar (Vision) to represent the visualization method, no guidance (None), and a combination of Haptic and Vision (VisHap). Outcomes related to the success rate and accomplishment time (of the shooting task), the number of head rotations, and participant responses to a follow-up questionnaire revealed that Haptic performed significantly better than None but was inferior to Vision, indicating that the proposed method succeeded in terms of effectively providing 3D guidance. VisHap performed roughly as well as Vision and was preferred to other conditions in most cases, indicating the general usefulness of the proposed method. Meanwhile, the findings from the questionnaire suggest that although the modular vibrations improved the music-listening experience during the shooting task, the impact on the overall gaming experience is unclear. This warrants further research.

The real-time performance of motion-captured avatars in virtual space is becoming increasingly popular, especially within applications including social virtual realities (VRs), virtual performers (e.g., virtual YouTubers), and VR games. Such applications often include contact between multiple avatars or between avatars and objects as communication or gameplay. However, most current applications do not solve the effects of contact for avatars, causing penetration or unnatural behavior to occur. In reality, no contact with the player’s body occurs; nevertheless, the player must perform as if contact occurred. While physics simulation can solve the contact issue, the naive use of physics simulation causes tracking delay. We propose a novel avatar tracking controller with feedforward control. Our method enables quick, accurate tracking and flexible motion in response to contacts. Furthermore, the technique frees avatar performers from the loads of performing as if contact occurred. We implemented our method and experimentally evaluated the naturalness of the resulting motions and our approach’s effectiveness in reducing performers’ loads.

We propose "TEllipsoid", an ellipsoidal display for video conference systems that can provide not only accurate eye-gaze transmission but also practicality in conferences, namely the convenience to use and the preservation of the identity of the displayed face.The display comprises an ellipsoidal screen, a small projector, and a convex mirror, where the bottom-installed projector projects the facial image of a remote participant onto the screen via the convex mirror. The facial image is made from photos shot from 360 degrees around the participant. Moreover, the image is modified to improve identity. The gaze representation is implemented by projecting the 3D model of eyeballs onto a virtual ellipsoidal screen.We evaluated the gaze transmissibility of the display in conference situations. As a result of experiments, we concluded that accurate gaze transmission is available in conferences when the angular distance of the adjacent participants is more than 38.5 degrees.

Recent developments in robotics and virtual reality (VR) are making embodied agents familiar, and social behaviors of embodied conversational agents are essential to create mindful daily lives with conversational agents. Especially, natural nonverbal behaviors are required, such as gaze and gesture movement. We propose a novel method to create an agent with human-like gaze as a listener in multi-party conversation, using Hidden Markov Model (HMM) to learn the behavior from real conversation examples. The model can generate gaze reaction according to users' gaze and utterance. We implemented an agent with proposed method, and created VR environment to interact with the agent. The proposed agent reproduced several features of gaze behavior in example conversations. Impression survey result showed that there is at least a group who felt the proposed agent is similar to human and better than conventional methods.

Embrace is an essential part of human-social interactions. It also gives positive health benefits as much as other touch gestures. However, in the recent years, studies of embrace recognition has been largely ignored when compared to recognition of touch gestures such as pat or rub. There are different kinds of embrace, with humans and pets, which can express different meanings. As embraces can be as important as touches, we are interested in what kind of embraces we can model, specially for human-robot interaction. In this paper, we investigate an unsupervised embrace pose recognition system based on a soft-stuffed robot platform. Our proposed method includes a hardware implementation of soft fabric-based capacitive touch sensors and a software algorithm comprising of k-means clustering based on locational features extracted from a sliding window. The result shows that our proposed method can model embrace patterns as different clusters. The method is capable of recognizing and clustering unseen data to a similar cluster's patterns, though there is a limitation when modeling two poses whose touches are similar but different in the alignment of the robot. In the next step, we plan to improve and test the proposed method in real-time environment, and make adjustments to our sensing system to cope with found limitations. If successful, the proposed method will be integrated with a touch gesture recognizer into a gesture recognition system for creating interactive and affective responses with stuffed-toy robot, which can become a medium for robot therapy or our own pets at home.

Narrative or episodic memory is a way of how humans recall a story. Narrative has also a strong connection with human emotions and attention. We combine the analysis of facial expressions and attention to find a way to generate an episodic memory timeline which matches each individual person. We use face shape deformations and eye gaze to understand when a person has a change of facial expression and associate this with a narrative event. To evaluate our assumption, we performed an experiment where several subjects watch a short video while their physical reactions were recorded; then our software generates an episodic memory timeline based on those reactions. We found that, even with the same video, the episodic memory timeline for each subject was unique. We found as well, that physical reactions and participant's memory recalls, were consistent with the generated episodic memory timeline.

A novel stuffed-toy robot that expresses its intentions through the sense of force and touch is proposed. The robot is so soft that one might wish to embrace it. Compliance of the robot can also be determined by holding its hands, similar to newborn babies or small animals.

In this paper, we propose a technique for a high quality haptic display working with a low update rate rigid body dynamics simulator. The proposed method uses two dynamics simulators. One has a low update rate for the whole virtual world and the other has a high update rate for the neighboring objects of the haptic pointer. In addition, the method calculates accelerance matrices of neighboring objects with regard to the contact forces added to these objects. We carried out a simulation and an experiment to check the effectiveness of the proposed method.

Genetic Algorithms (GAs) generally maintain diverse solutions of good quality in multi-objective problems, while Particle Swarm Optimization (PSO) shows rapid convergence to the optimum solution. Previous studies indicated that search abilities can be improved by simply coupling these two algorithms; GA compensates for the low diversity of PSO, while PSO compensates for the high computational costs of GA. In this study, the configurations of the two methods when used in a fully coupled hybrid algorithm were investigated to achieve an improvement in diversity and convergence simultaneously for application to real-world engineering design problems. The new hybrid algorithm was validated using standard test function problems, and it was demonstrated that the new hybrid algorithm showed better performance than the simply coupled hybrid algorithm, as well as both pure GA and pure PSO. Especially, the new hybrid algorithm shows robust search ability regardless of initial population selection. This feature is very important in real-world engineering design problems, which do not allow multiple optimization runs to be implemented due to heavy computational costs. The new method was applied to optimization of a diesel engine combustion chamber to reduce exhaust emissions, such as NOx and soot. The results demonstrated the applicability of the present method to real-world design problems. In addition, important geometry design variables controlling the emission performance were investigated to obtain useful knowledge about low emission diesel engine design.

Virtual machine monitors (VMMs), including hypervisors, are a popular platform for implementing various security functionalities. However, traditional VMMs require numerous components for providing virtual hardware devices and for sharing and protecting system resources among virtual machines (VMs), enlarging the code size of and reducing the reliability of the VMMs.This paper introduces a hypervisor architecture, called parapass-through, designed to minimize the code size of hypervisors by allowing most of the I/O access from the guest operating system (OS) to pass-through the hypervisor, while the minimum access necessary to implement security functionalities is completely mediated by the hypervisor. This architecture uses device drivers of the guest OS to handle devices, thereby reducing the size of components in the hypervisor to provide virtual devices. This architecture also allows to run only single VM on it, eliminating the components for sharing and protecting system resources among VMs.We implemented a hypervisor called BitVisor and a parapass-through driver for enforcing storage encryption of ATA devices based on the parapass-through architecture. The experimental result reveals that the hypervisor and ATA driver require approximately 20 kilo lines of code (KLOC) and 1.4 KLOC respectively.

Abstract Various forms of art and entertainment involve many different characters, and advances in human interfaces have necessitated physical interactions in order to develop an improved sense of reality. In this paper we propose a method for generating the motions of characters using multidimensional keyframe animation in parallel with real‐time physical simulation. The method generates characters capable of physical interaction, and also allows animators to use traditional methods for designing character motion. We have implemented the system and confirmed its effectiveness experimentally.

In recent years, as the data processed by governmental or commercial organizations increases, cases involving information leak have risen. It is difficult to control information on many distributed end-point computers using conventional security mechanisms. Therefore, we have been proposed a novel secure VMM (Virtual Machine Monitor) architecture which is used as a foundation of security policy enforcement on distributed computers. This paper especially introduces Role-based Access Control (RBAC) to theID management framework in a secure VMM system. Our proposal will reduce costs for distributed policies updates. Proposed RBAC mechanism employs attribute certificates (ACs) to handle user’s roles. This paper shows design and prototype implementation based on PKI-based ID card and proven open source VMM software, QEMU.

We create a framework for physical interaction with virtual human. The virtual human reacts to the input of the user in realtime. The virtual human has an ocular system model, which mimics human ocular system, to generate natural and human-like gaze motions and to create inputs for the cognitive model of the virtual human. The visual input for virtual human is generated from the dynamics simulator for the virtual world. The gaze target is chosen by vying of bottom-up and top-down attentions. The motion controller for the eyes and head is based on observations and models of human gaze motion. We create a virtual boxing game as an application of proposed framework. Proposed framework succeeds to create natural gaze reaction. In addition, by changing the balance of the bottom-up and top-down attention, looks of concentration of the virtual human can be tuned.

Haptic interfaces enable us to interact with a virtual world using our sense of touch. This paper presents a method for realizing haptic interaction with water. Our method displays forces acting on rigid objects due to water with a high frame rate (500 Hz). To achieve this, we present a fast method for simulating the dynamics of water. We decompose the dynamics into two parts. One is a linear flow expressed by a wave equation used to compute water waves. The other is a more complex and non-linear flow around the object. The fluid forces due to the non-linear flow is precomputed by solving Navier-Stokes equations, and stored in a database, named the Fluid Resistance Map. The precomputed non-linear flow and the linear flow are combined to compute the forces due to water.

In this paper, our initial work on the use of visualization and virtual reality for education especially for providing three-dimensional concepts of molecular structure was presented. Our system is composed of a hybrid screen, multi-projectors, PC cluster, 6DOF haptic device etc. It offers a channel to reach into the molecular space in an immersive environment, allows user to employ an intuitive 6DOF haptic device Spidar-G to control the perspective. For educators, it can serve as a good instructional aid for helping students to understand molecular structure through effective visual representation and interactive manipulation.

Abstract This paper proposes a new method for real‐time rigid body simulations for haptic interactions based on a penalty method regarding contact volume. Analytical methods for calculation of contact forces require too much time to maintain fast update rates for haptic controls. In addition, they prohibit direct connection of haptic interfaces. Penalty methods, which employ spring‐damper models for calculation of contact forces, offer a very rapid rate of iterations. In addition, they permit direct connection of haptic interfaces. Penalty methods are good for haptic interactions. However, previous penalty methods do not regard distribution of contact forces over the contact area. For that reason, they can't calculate normal and friction forces on face‐face contacts correctly. We propose a distributed spring‐damper model on a contact area to solve these problems. We analyze three‐dimensional geometries of the intersecting portion on the polyhedral objects. Then, we integrate forces and torques of distributed spring‐damper models. We implement a proposed simulator and compare it with a point‐based penalty method and constraint method. The comparison shows that the proposed simulator improves accuracy of the simulation of face‐face contact and friction forces and the simulation speed. In addition, we attach a six degree‐of‐freedom (6‐DOF) haptic interface to the simulator. Users can feel 6‐DOF force feedback and input 6‐DOF motions. Categories and Subject Descriptors (according to ACM CCS): I.6.5 [Model Development]: Modeling methodologies, H.5.2 [User Interfaces]: Haptic I/O, I.3.6 [Methodology and Techniques]: Interaction techniques

This paper proposes a new approach for adding haptic feedback on a workbench. The integration of a stringed haptic device is discussed.

Abstract The workbench is an interesting semi‐immersive configuration for interactive tasks. However, haptic feedback, i.e.force and tactile feedback, is one important cue which is missing. To the authors' knowledge, the sole proposedsolution consists in installing an arm force feedback device on one‐screen workbenches. This solution, however,has several drawbacks. The arm can perturb the stereoscopic display, cross virtual objects or hide parts of thevisualization space. Furthermore, the interaction space is limited by the size of the arm, which may also damagethe screen or perturb the electromagnetic tracking system. Some of these difficulties may even be worth with a two‐screenworkbench. This paper discusses an alternative solution, which consists in integrating a stringed hapticdevice on a workbench. This approach is less invasive, more flexible and well‐suited to a two‐screen workbench.

We demonstrate a new intuitive force feedback device for advanced VR applications. Force feedback for the device is tension based and is characterized by 7 degrees of freedom (DOF); 3 DOF for translation, 3 DOF for rotation, and 1 DOF for grasp. The SPIDAR-G (Space Interface Device for Artificial Reality with Grip) will allow users to interact with virtual objects naturally by manipulating two hemispherical grips located in the center of the device frame. We show how to connect the strings between each vertex of grip and each extremity of the frame in order to achieve force feedback. In addition, methodologies will be discussed for calculating translation, orientation and grasp using the length of 8 strings connected to the motors and encoders on the frame. The SPIDAR-G exhibits smooth force feedback, minimized inertia, no backlash, scalability and safety. Such features are attributed to strategic string arrangement and control that results in stable haptic rendering. The design and control of the SPIDAR-G is described in detail and the space graphic user interface system based on the proposed SPIDAR-G system is demonstrated. Experimental results validate the feasibility of the proposed device and reveal its application to virtual reality.