This study explores interactive scripted performance with a character robot to create an engaging storytelling experience. By involving human participants as performers alongside the robot, we investigated whether this approach could enhance engagement in the storytelling experience. We developed a character robot system capable of executing predefined phrases and movements to foster immersion, thereby enhancing engagement. We conducted interactive scripted performance events at after-school care facilities. The results indicate that interactive scripted performance encouraged participants to engage in the experience actively. Moreover, the findings suggest that the developed character robot enhanced participants’ story immersion and encouraged physical engagement.

Science Fiction Prototyping (SFP) is a method that uses science fiction to imagine future technologies and foster innovation. It is considered effective for exploring human-robot relationships and envisioning better robot designs. However, robot embodiment influences human perception, which plays a crucial role in interaction. Simply imagining future scenarios with robots through SFP may overlook these aspects. We propose an approach called Experiential Science Fiction Prototyping (ESFP), which adds a phase of experiencing the story to the traditional SFP process. To explore the effects of ESFP, we conducted a workshop with Japanese teenagers under the theme of designing a robot that contributes to a sense of “ibasho”—a Japanese concept referring to a space or relationship where one feels accepted and comfortable. ESFP unfolds in three phases: Storytelling, where participants envision future lives with robots and create stories; Experience, where they bring these stories to life through interaction with a physical robot; and Discussion, where they reflect on the story they created and experienced. The results suggested that, through the experiential phase, participants developed new ideas about interaction with robots and expanded their imagination about future relationships. Experiencing the story helped participants connect more closely with the envisioned robot interactions and inspired new reflections and expectations. This study contributes by proposing the ESFP method, detailing its implementation, and discussing its potential through a case study.

A robotic hand for social interaction should be capable of comfortable touch with humans. However, it is difficult to mount skin, tactile sensors, and driving mechanism required for human contact, especially holding hands, on a slender finger. In addition, in order to unitize the hand for easy use with any robot and maintainability, the mechanism must be contained within the small space of the fingers and palms. In this paper, we propose a human-sized five-fingered robotic hand named WARABI Hand. It is covered with multi-layored rubber skin to realize human-like soft and pleasant feel. Force sensors on each finger link detect contact with humans and adjust gripping force. We conducted experiments in which a humanoid equipped with WARABI Hand grasped forearm, held hands, and interlocked fingers with a person. The performance for object grasping was also evaluated. We demonstrated that our proposed hand is useful for interaction with humans including receiving and handing over things.

In order to make robots more useful in a variety of environments, they need to be highly portable so that they can be transported to wherever they are needed, and highly storable so that they can be stored when not in use. We propose "on-site robotics", which uses parts procured at the location where the robot will be active, and propose a new solution to the problem of portability and storability. In this paper, as a proof of concept for on-site robotics, we describe a method for estimating the kinematic model of a robot by using inertial measurement units (IMU) sensor module on rigid links, estimating the relative orientation between modules from angular velocity, and estimating the relative position from the measurement of centrifugal force.At the end of this paper, as an evaluation for this method, we present an experiment in which a robot made up of wooden sticks reaches a target position. In this experiment, even if the combination of the links is changed, the robot is able to reach the target position again immediately after estimation, showing that it can operate even after being reassembled. Our implementation is available on https://github.com/hiroya1224/urdf_estimation_with_imus.

It is important for daily life support robots to detect changes in their environment and perform tasks. In the field of anomaly detection in computer vision, probabilistic and deep learning methods have been used to calculate the image distance. These methods calculate distances by focusing on image pixels. In contrast, this study aims to detect semantic changes in the daily life environment using the current development of large-scale vision-language models. Using its Visual Question Answering (VQA) model, we propose a method to detect semantic changes by applying multiple questions to a reference image and a current image and obtaining answers in the form of sentences. Unlike deep learning-based methods in anomaly detection, this method does not require any training or fine-tuning, is not affected by noise, and is sensitive to semantic state changes in the real world. In our experiments, we demonstrated the effectiveness of this method by applying it to a patrol task in a real-life environment using a mobile robot, Fetch Mobile Manipulator. In the future, it may be possible to add explanatory power to changes in the daily life environment through spoken language.

Holding hands can give people a sense of security. In this study, we developed a five-fingered robotic hand that can hold hands with a person and a guidance system that uses the developed hand to hold hands with them. In this system, a robot remembers the location where people have taught it and guides people to that point. We conducted an experiment to evaluate the sense of security in guidance with hand-holding compared with guidance without hand-holding. Participants watched the videos on these two conditions and answered a questionnaire. The results confirmed that holding hands can lead to a sense of security.

The purpose of this paper is the robotic hanging manipulation of an object of various shapes that is not limited to a specific category. To achieve this, we propose a method that allows the estimator to learn many different shapes with hanging points without any manual annotation. A random shape generator using GAN solves the limitation of the number of 3D models and can handle objects of various shapes. In addition, hanging is repeated in the dynamics simulation, and hanging points are automatically generated. A large amount of training data is generated by rendering random-textured objects with hanging points in the random simulation environment. A deep neural network trained with these data was able to estimate hanging points of an unknown category object in the real world and achieved hanging manipulation by a robot.

The learning-from-observation (LfO) framework aims to map human demonstrations to a robot to reduce programming effort. To this end, an LfO system encodes a human demonstration into a series of execution units for a robot, which are referred to as task models. Although previous research has proposed successful task-model encoders, there has been little discussion on how to guide a task-model encoder in a scene with spatio-temporal noises, such as cluttered objects or unrelated human body movements. Inspired by the function of verbal instructions guiding an observer’s visual attention, we propose a verbal focus-of-attention (FoA) system (i.e., spatiotemporal filters) to guide a task-model encoder. For object manipulation, the system first recognizes the name of a target object and its attributes from verbal instructions. The information serves as a where-to-look FoA filter to confine the areas in which the target object existed in the demonstration. The system then detects the timings of grasp and release that occurred in the filtered areas. The timings serve as a when-to-look FoA filter to confine the period of object manipulation. Finally, a task-model encoder recognizes the task models by employing the FoA filters. We demonstrate the robustness of the verbal FoA in attenuating spatio-temporal noises by comparing it with an existing action localization network. The contributions of this study are as follows: (1) to propose a verbal FoA for LfO, (2) to design an algorithm to calculate FoA filters from verbal input, and (3) to demonstrate the effectiveness of a verbal FoA in localizing an action by comparing it with a state-of-the-art vision system.

In this paper, we aim to recognize function points of category-agnostic objects and perform object manipulation. To recognize function points of various shapes, it is necessary to train with a large amount of training data. Also, it is necessary to take into account not only visual information but also physics and interaction between objects. To solve these problems, we are working on the automatic generation of training data by detecting function points from a physical simulation. In the proposed system, we add simulation with target task operation and goal state, which allows a robot to acquire the target function point recognizer. We also use GAN to generate various random shapes and render them with random domains, and train Deep Neural Networks on these data. These enable the robot to recognize function points of unseen objects in the real world and realize manipulation.

In this paper, we propose an approach for online whole-body control of position-controlled humanoid robot with 3D multi-contact to cope with contact wrench constraints and joint overload. In our method, robots are controlled under contact wrench constraints with three features: 1) internal wrench control to reduce joint load and prolong the time in which the high-load postures can be maintained 2) feasible utilization of self-collision to reduce joint load by turning off joint servo gains 3) handling degenerated degree of freedom by solving a quadratic optimization problem integrating wrench distribution and inverse kinematics in which internal wrench is controlled only in controllable directions.With our methods, HRP2-JSKNTS could pick up an object under a desk with squatting with the back of the upper leg on the back of the lower leg without sliding at the right arm. We also evaluated the effectiveness of our control to reduce joint load with another experiment.

For robots to autonomously achieve manipulation tasks in various scenes, advanced operational skills such as tool use, learning from demonstration, and multi-robot/human-robot cooperation are necessary. In this research, we devise a method for robots to realize such operational skills in a unified manner by evaluating physical consistency (referred to as “physics reasoning”) based on the formulation of the manipulation statics constraints. First, we propose manipulation planning and estimation methods in which the operational feasibility and properties' likelihood are derived by physics reasoning. In addition, we propose a framework to manipulate an object with unknown physical properties by executing planning and estimation both sequentially and in parallel. We demonstrate the effectiveness of the proposed methods by performing experiments in which real humanoid robots achieve various manipulation tasks with advanced operational skills.

We propose a human mimetic foot structure for musculoskeletal humanoids. We designed the foot structure by inspiring from human foot abilities of the multi-bone connected structure for flexibility and the distributed force sensor system. The foot has multi-DOFs structure including toe DOF that is composed of fingers. The distributed force sensing system is composed of 12 an-axis force sensors. In order to demonstrate those effectiveness, we implement the foot into musculoskeletal humanoid Kengoro and conduct several experiments. As a result, we confirmed effectiveness of the foot from tiptoe motion and balancing behavior by utilizing the foot characteristics.