Systems providing various virtual experiences through virtual reality (VR) are being utilized in many fields such as sports, education, entertainment, and healthcare. Many of these systems offer unique and rare experiences that are not easily accessible in the real world, aimed at improving motor and cognitive skills, as well as enhancing enjoyment. However, from the perspective of information systems that improve user happiness and well-being quality, the field is still in a developmental state, and the specific design methods for virtual experiences remain unclear. Our aim is not to enhance athletic performance or skills, but rather to construct a VR system that supports individuals in discovering their own pleasures and aspirations. This paper reports on the implementation of a prototype system aligned with the concept of supporting the discovery of pleasures and aspirations, focusing on a travel experience system as the prototype for this VR system.

The difficulty is a fundamental factor of the user’s motivation and engagement in some tasks. Dynamic difficulty adjustment (DDA) systems provide users with an optimal level of challenge. Previously, some studies developed a DDA system that can set the task’s difficulty to any level. However, these studies lack the investigation of the influence of the difficulty levels on the psychological aspect. For this purpose, we consider a difficulty setting that consists of stepwise difficulty levels (e.g., hard, normal, and easy) set to adapt to each user’s skill and evaluate it using self-efficacy. In the experiment, we employ a Kendama task in a VR space where the difficulty level can be easily adjusted. The result shows that the difficulty levels in our method can be set according to the user’s skill. Moreover, we experimentally clarify a strong correlation between successful experiences in imagination and the enhancement of self-efficacy in the difficulty setting, which means that adapting difficulty levels to the user’s skill has the potential to enhance self-efficacy effectively.

This study examined the impact of a "white lie" designed to boost motivation during virtual reality exercise on enjoyment and mental fatigue. Participants engaged in a ball-throwing or ball-targeting task and were randomly assigned to groups with or without the white lie. Results indicated that both groups experienced similar levels of enjoyment and fatigue, suggesting the white lie had minimal effect on these factors. All participants, regardless of group, reported high levels of enjoyment, with 17 out of 18 indicating they had fun, no significant differences in mental fatigue were found between groups while participants generally favored the white lie. However, the positive experience across all participants highlights the potential of Virtual Reality for promoting exercise engagement.

Assistive agents have to adapt training tasks according to the needs of the users. Virtual Reality can be helpful in understanding the best support strategies to improve people’s motivation, performance and confidence in their capabilities, defined as self-efficacy. When given a virtual body, users can develop a sense of embodiment towards it. We developed a ball-throwing training system that is able to support users by enhancing the sphere’s trajectory without them being aware of it, to increase their confidence. We analyzed the influence of the support system on sense of embodiment and found that it persisted despite the alterations.

In systems where agents teach skills to users in environments such as VR or digital applications, the efficiency of skill improvement is crucial. For designing such skill-teaching and assistive systems, this study focuses on applying Bandura’s concept of self-efficacy. Demonstrations and assistance by the agent are considered vicarious experiences, which is a crucial component for improving self-efficacy. This study investigates and analyzes the impact of vicarious experiences on self-efficacy, specifically examining the sequence of vicarious and self experiences and the resulting differences in performance. We developed a virtual reality (VR) juggling experiment to investigate how altering the sequence of vicarious and self experiences impacts self-efficacy. The results showed that the group for which the performance was tested after the vicarious experience showed reduced self-efficacy. This suggests that the sequence in which vicarious experiences are presented can impact self-efficacy. Because Bandura’s theory did not specify the order of experiences, our findings would contribute to considering lesson designs and how assistive agents must behave to enhance users’ self-efficacy.

To build and improve the interaction capabilities of robots, the quality of interaction should be evaluated based on human subjectivity, not only on the time required or the degree of accomplishment. However, it remains unclear which evaluation metrics humans prioritize when robots guide their behavior. To investigate the factors that humans prioritize when evaluating the quality of human-robot interaction in guiding human behavior, we asked the evaluators to answer what they based their evaluation of the quality of interaction on in an open-ended form and listed the evaluation items. In addition, based on the scoring results of the collected evaluation items, we conducted a factor analysis to uncover the underlying evaluation metrics. The study revealed four evaluation metrics including not just the appropriateness of attribute explanation and instruction conciseness, but also the ability to guide human users based on their behavior.

In recent years, the human Digital Twin has been discussed as new technology. When we discuss a world in which one’s self-avatar autonomously performs social activities in cyberspace, the questions arise whether or not the behavior of the avatars feels like one’s own, and whether or not we can approve of the self-avatars’ social activities on behalf of ourselves. We define such feeling as the sense of self-individuality. In this study, we focused on the situation in which self-avatars perform presentations on behalf of ourselves to investigate the effect of the modification experience on the presentation motions by self-avatars on the sense of self-individuality. We conducted VR-based experiments in which the motion modification intervention was performed on self-avatars over eight weeks by 24 experiment participants. As a result, we found that the sense of self-individuality was improved as the number of modifications and interventions increased. However, we found that the intensity of motion modification did not correlate with the improvement of the sense of self-individuality in this experiment condition. We also found that the sense of self-individuality was reduced when others intervened in the motion. From these results, we clarified that the experience of motion modification on self-avatars is significant when designing the behavior of avatars acting on behalf of ourselves in human Digital Twin. Further investigation is required to clarify the effect of the long-term intervention on behavior to distinguish between the mere exposure effect.

This article presents a new data-driven framework for analyzing periodic physical human–robot interaction (pHRI) in latent state space. The model representing pHRI is critical for elaborating human understanding and/or robot control during pHRI. Recent advancements in deep learning technology would allow us to train such a model on a dataset collected from the actual pHRI. Our framework is based on a variational recurrent neural network (VRNN), which can process time-series data generated by a pHRI. This study modifies VRNN to explicitly integrate the latent dynamics from robot to human and to distinguish it from a human state estimate module. Furthermore, to analyze periodic motions, such as walking, we integrate VRNN with a new recurrent network based on reservoir computing (RC), which has random and fixed connections between numerous neurons. By boosting RC into a complex domain, periodic behavior can be represented as phase rotation in the complex domain without decaying the amplitude. A rope rotation/swinging experiment was used to validate the proposed framework. The proposed framework, trained on the collected experiment dataset, achieved the latent state space in which variation in periodic motions can be distinguished. The best prediction accuracy of the human observations and robot actions was obtained in such a well-distinguished space.

In this paper, we propose a generative model that learns a sequence-to-sequence (Seq2Seq) translation between human whole-body motions and linguistic descriptions by natural language. Our model merges the Seq2Seq model with the training strategy of sequence generative adversarial nets (SeqGAN), which extends a GAN framework to solve the problem that the gradient cannot pass back to the generator network. This model considers a generator, trained using a policy gradient method, as a stochastic parameterized policy. In the policy gradient, we employ a Monte Carlo (MC) search to receive the final reinforcement learning (RL) reward from the discriminator. The proposed generative network is trained on the KIT Motion-Language Dataset, which is one of the few large-scale datasets available and includes 3,911 human motions and 6,278 natural language descriptions. During the experiments, we evaluated the effectiveness of our model by comparing its various configurations and parameter settings. Finally, our model achieves a remarkably high performance, outperforming an existing state-of-the-art method under the same dataset split for fair comparison. In addition, the qualitative results of the motion-to-language translation demonstrate that our model can generate semantically and grammatically correct sentences with detailed linguistic descriptions from human motions.

It is important to evaluate and analyze human behaviors and reactions to build robots that can interact with people. While immersive virtual reality (VR) is one of the useful tools for the evaluation, it has been concerned that the human behavior may differ between VR and real environments due to the differences in cognitive characteristics. In this study, we investigated the differences in human behavior by focusing on common situations in the home environment; for example, a change in head position and posture is required when searching for an object (e.g., peeking motion) and when high accuracy is required when manipulating an object (e.g., placing an object on a shelf). We found that object manipulation requires accuracy which in turn, requires more time in the VR environment. In addition, the amount of change in head posture during object search and manipulation behavior was larger in the VR environment. Finally, we discuss the possibility of using the findings of this study to design robot competitions to assess the quality of human-robot interaction (HRI).

Robot competition such as RoboCup@Home is one of the most effective ways to evaluate the performance of human-robot interaction; however, it takes a lot of costs for real robot maintenance and the practice of evaluation sessions. We have proposed a simulation software to evaluate human-robot interaction in daily life environment based on immersive virtual reality. In this paper, we design a task named `human navigation' in which the evaluation requires a subjective impression by the users. Through a substantiative experiment, we confirmed that the proposed task and system reduced the cost for the practice of the competition.

Humans interact in rich and diverse ways with the environment. However, the representation of such behavior by artificial agents is often limited. In this work we present motion concepts, a novel multimodal representation of human actions in a household environment. A motion concept encompasses a probabilistic description of the kinematics of the action along with its contextual background, namely the location and the objects held during the performance. We introduce a novel algorithm which learns and recognizes motion concepts from action demonstrations, named Online Motion Concept Learning (OMCL). The algorithm is evaluated on a virtual-reality household environment with the presence of a human avatar. OMCL outperforms standard motion recognition algorithms on an one-shot recognition task, attesting to its potential for sample-efficient recognition of human actions.

In Virtual Reality, the need to understand how subjects perceive their representation is gaining attention and importance. We present a contribution to a better understanding of the sense of embodiment by assessing two of its main components, body ownership and agency, through an experiment involving alien motion. The key aspect of the experimental protocol is to integrate a condition with some personalized alien finger movement while the subject is asked to remain still. Body ownership appears to be significantly reduced, but not agency. We also propose a metric to assess quantitatively that the view of the alien movement induces more finger posture variation compared to the reference context in the still condition.

Motion capturing and analysis is gaining attention as a way to achieve intelligent systems that support human daily-life activities. Utilizing immersive VR systems is one way to facilitate the observation and accumulation of human behavior. Although recent improvements in 3D head-mounted displays and hand tracking controllers enable us to cheaply capture natural human behavior in various scenarios, there is concern about the inconsistency of human behavior between a real physical environment and a VR one. The restriction of the field of view (FOV) is a major factor in such inconsistency, such as underestimation of distance. The effect of FOV restriction on human behavior in complex scenarios (e.g., daily-life activities requiring recognition speed and manipulation skill) in the VR environment is still unclear. In this work, we investigate whether there is any consistency of human behavior concerning recognition speed and object manipulation skill depending on the FOV restriction in both real and VR environments. We designed and carried out an object manipulation task, to perform this investigation.

Capturing realistic human behaviors is essential to learn human models that can later be transferred to robots. Recent improvements in virtual reality (VR) head-mounted displays provide a viable way to collect natural examples of human behavior without the difficulties often associated with capturing performances in a physical environment. We present a realistic, cluttered, VR environment for experimentation with household tasks paired with a semantic extraction and reasoning system able to utilize data collected in real-time and apply ontology-based reasoning to learn and classify activities performed in VR. The system performs continuous segmentation of the motions of users' hands and simultaneously classifies known actions while learning new ones on demand. The system then constructs a graph of all related activities in the environment through its observations, extracting the task space utilized by observed users during their performance. The action recognition and learning system was able to maintain a high degree of accuracy of around 92% while dealing with a more complex and realistic environment compared to earlier work in both physical and virtual spaces.

In this paper, we propose an online learning algorithm based on a Rao-Blackwellized particle filter for spatial concept acquisition and mapping. We have proposed a nonparametric Bayesian spatial concept acquisition model (SpCoA). We propose a novel method (SpCoSLAM) integrating SpCoA and FastSLAM in the theoretical framework of the Bayesian generative model. The proposed method can simultaneously learn place categories and lexicons while incrementally generating an environmental map. Furthermore, the proposed method has scene image features and a language model added to SpCoA. In the experiments, we tested online learning of spatial concepts and environmental maps in a novel environment of which the robot did not have a map. Then, we evaluated the results of online learning of spatial concepts and lexical acquisition. The experimental results demonstrated that the robot was able to more accurately learn the relationships between words and the place in the environmental map incrementally by using the proposed method.

In order to improve human-robot collaboration, it is necessary to consider how robots may be able to act in a way that is understandable to the people with whom they are working. This paper presents a preliminary experimental human-robot collaboration study with 10 human subjects. The paper analyzes the effect of a robot's emotionally expressive non-verbal behavior on human-robot teamwork. The study was modeled and performed in the immersive simulator SIGVerse. The findings of the study reveal that embodied emotional expressiveness improves the integration of human-robot activity. The results of the study show that embodied expressiveness increases the duration of the activities that have a positive value for the collaborative task. The embodied expressiveness also has a significant influence on the distance between human-robot collaborators.

This work proposes a new approach to realize embodied and multimodal HRI between virtual robot and real world human for HRI challenges in RoboCup @Home.

The high cost and complexity to maintain a robot, and the impracticability for long term large scale HRI with real robots, have induced constraints in HRI studies. The purpose of this paper is to demonstrate the feasibility of our proposal to realize embodied and multimodal HRI between virtual robot and real world human in response to the above constraints. Based on our SIGVerse simulator, a 3D virtual environment is established for the virtual robot to interact with real world human via immersive interface. The realizations of embodiment and multimodal interaction are illustrated in several HRI research scenarios, where verbal action instruction and spatial information from the body gesture are simulated.

We present the concept of Bayesian Tool Affordances as a solution to estimate the suitable action for the given tool to realize the given novel effects to the robot. We define Tool affordances as the “awareness within robot about the different kind of effects it can create in the environment using a tool”. It incorporates understanding the bi-directional association of executed Action, functionally relevant features of the Tool and the resulting effects. We propose Bayesian leaning of Tool Affordances for prediction, inference and planning capabilities while dealing with uncertainty, redundancy and irrelevant information using limited learning samples. The estimation results are presented in this paper to validate the proposed concept of Bayesian Tool Affordances.

This work aims to realize multimodal interaction with embodied contextual understanding based on the simple chatterbot system. A system framework is proposed to integrate the dialogue system into a 3D simulation platform, SIGVerse to attain multimodal interaction. The chatterbot AIML implementations are described in the achievement of the conversations with embodied contextual understanding in HRI simulations.

The aim of this work is to propose a cloud computing architecture simulation platform for social human-robot interaction. This paper explains the design and development of this system named SIGVerse in four main components, namely (1) SIGServer as the central server, (2) Agent Controller for user applications, (3) Service Provider, and (4) SIGViewer as the client terminal, and web based development interface, in addressing the two main “human” issues in human-robot interaction with cloud computing architecture, (a) Distributed development platform, and (b) Large scale of human-robot simulation. Three current applications are discussed for the validation of the cloud computing architecture in social human-robot interaction simulations.

As a step of a robotics research that requires proper binding of motion patterns and symbolic expressions, in this paper, we attempt to model how humans use motion patterns and symbolic expressions for motion coaching. Through experiments of tennis forehand swing coaching task for beginners, we observed and analyzed three kinds of motions; a learning target swing performed by human coaches, swings performed by learners, and dynamically modified swing motions by human coaches according to performances of learners. With results, we modeled relationship between differences among the three motions in a phase space and used symbolic expressions. Then we discussed how the model can be applied for realizing an efficient motion coaching system we have been proposed.

The collaboration between humans and robots can lessen the burden of automatic learning while performing difficult tasks. In this work, we explore sketching as a method to enable effective collaboration between human and robot. The system allows a human to contiguously interact with the robot to perform a task by allowing the sketching of the environment and specifying the affordances of objects and areas on the map.

Whole-body gestures and verbal expressions should be bound according to given tasks and the current situation in intelligent human-robot interaction systems. Moreover, modification of expressions, such as emphasis of motions and change in verbal expressions, plays an important roll for successfully completing tasks according to user reaction. Slight differences in motions can be conveyed by binding an emphasized motions and an verbal expression. In robotics, however, even though the synthesis of gestures and speech has been discussed, how to bind synthesized emphatic motions and verbal expressions from an engineering point of view has not been adequately discussed. Synthesis of motion and speech requires recognition of user reaction, we therefor should integrate 1) recognizing reaction, 2) planning to complete tasks, 3) modification of motions and speech, and 4) maintaining a bi-directional interaction loop consisting of processes 1)–3). Thus, using a phase space, we propose a method for binding emphatic motions and adverbial expressions, and for evaluating and controlling these four required processes by using a sole scalar parameter. In the phase space, variety of motion patterns and verbal expressions can be expressed as static points. To evaluate the feasibility of the proposed method, we also propose a bi-directional motion coaching system using the method. We show the feasibility and effectiveness of robotic motion coaching systems through experiments of actual sport coaching tasks for beginners. From the results of participants' improvements in motion learning, we discuss and conclude the factors affecting such motion coaching systems that realizes binding and controlling emphatic motions and adverbial expressions using a sole scalar parameter in a phase space.

In this paper, we propose an improved mimesis method for interpolation and extrapolation of motion patterns in the proto-symbol space towards an ultimate goal that motion pattern synthesis and recognition of humanoid robots are achieved by means of natural language. The proto-symbol space is a topological space which abstracts motion patterns by utilizing continuous hidden Markov models. An interpolation algorithm for the proto-symbol space was proposed in a previous work, but an extrapolation algorithm was not. Therefore, in this study, we propose and extrapolation method which can further clarify the physical meaning of the dimension of the proto-symbol space that is one of the most essential issues for the realization of translation between motion patterns and language using the proto-symbol space. The extrapolation method also enables the robot to recognize and synthesis various kinds of motion patterns using a fewer number of proto-symbols. The feasibility of the proposed method is demonstrated through simulation experiments.

Memorization, abstraction, and generation of a time-series of sensors and motion patterns are some of the most important functions for intelligent robots, because these memories are useful for situation recognition and behavior decision making. In conventional research, recurrent neural networks are often used for such memory functions. However, they cannot memorize a lot of patterns and its learning algorithm is unreliable. In this paper, we propose a method for the induction of behavior and situational estimation based on hidden Markov models, which is currently one of the most useful stochastic models. With the proposed method, we show the feasibility of: (1) Both recognition and association are executed at the same time, and (2) A multiple degrees of freedom and multiple sensorimotor patterns are acceptable

In this paper, we focused on concepts of behavior modifier. We aim to build a system, which acquires concepts of behavior modifier and applies it to motion generation of robots. For a form of motion representation, an existing research achievement, which statistically abstracts motion with known sets of motion examples and represents motions as points in a space, was adopted. Our system uses tendency of motions with adverbial modifier in the space to represent concepts of behavior modifier. The system has a simple representation form and has interactiveness what has potential advantage of ability to be applied to various motion concepts.

In this paper, we propose an approach where robots store shared experiences between human and the robots, and also show that the shared experiences act as an important role not only for the coexistence of the human and robots but also for the realization of the supporting ability of the robots. Enormous amount of shared experiences are effective for the understanding of users under uncertain and incomplete conditions, as well as communication among human. We propose the importance of the shared experiences for real world intelligence, and show a development research of the infrastructure for the storage of the share experiences and application method.

Memory of motion patterns as data, comparison of a new motion pattern with data, and playback of one from the data are inevitably involved in the information processing of intelligent robot systems. Such computation forms the computational foundation of learning, acquisition, recognition, and generation process of intelligent robotic systems. In this paper, we propose to apply the continuous hidden Markov model to establish the computational foundation, using which one obtains the specified number of keyframes and their probability distributions. The keyframes are optimally selected to maximize the likelihood. The probability distributions are to be used to compute comparison and playback. The proposed method is applied to the motion data of a humanoid robot as well as the time series image data, and its validity is to be discussed.

It is needed for humanoid to acquire not only just a trajectory but also aim of the behavior and symbolic information during behavior development. We (2001) have proposed the mimesis system as a framework of synchronous learning model for behavior acquisition and symbol emergence. However, the motion elements which are fundamental representation of behavior have stood on the unsuitable assumption that they are given without taking the robot embodiment and dynamics into consideration. In this paper, the design theory of motion elements with consideration of the embodiment are shown, and novel methods of realization of the mimesis for real humanoids is proposed.

Behavior recognition process and behavior generation process have a close relationship in humans' brains. It is expected that humans' brains understand the meaning of behavior and create symbols through co-development of recognition and generation processes. In this paper, we propose a novel method for the integration of behavior patterns and symbols using associative memory in order to realize the co-development processing. In the model, behavior recognition process and generation process are practiced based on a mutual dynamics. We also confirmed the feasibility of the method on humanoid simulator.

Since a biped humanoid inherently suffers from instability and always risks tipping over, ensuring high stability and reliability of walking is one of the most important goals. The paper proposes a walk control consisting of a feedforward dynamic pattern and a feedback sensory reflex. The dynamic pattern is a rhythmic and periodic motion, which satisfies the constraints of dynamic stability and ground conditions, and is generated assuming that the models of the humanoid and the environment are known. The sensory reflex is a simple, but rapid motion programmed with respect to sensory information. The sensory reflex, we propose, consists of the body posture control, the actual ZMP (zero moment point) control, and the landing time control. With the dynamic pattern and the sensory reflex, it is possible for the humanoid to walk rhythmically and to adapt itself to environmental uncertainties. The effectiveness of our proposed method was confirmed by walk experiments of an actual 26 DOF humanoid on an unknown rough terrain and in the presence of disturbances.

Mimesis is a primitive learning framework and origins of human intelligence. We have developed a behavior acquisition and understanding system based on the mimesis. This system is able to abstract observed others' behaviors into conceptual symbols, to recognize others' behavior using the primitive symbols, and to generate self motion patterns using the primitive symbols. In this paper, we mention the integration of mimesis loop which is the acquisition and development system based on mimesis, and confirmation of the feasibility against whole body motions on virtual humanoids.

We propose a behavior learning method based on Bayesian networks and experience of interaction between human and robots, which does not need a priori knowledge and can be applied to human-robot interaction models. In this method, the behavior learning based on interaction experience was established. However, developers must adjust initial sensor state of the Bayesian network according to the user preference. In this paper, we propose a new method of state space construction for user adaptation based on introspection of interaction experience using genetic algorithms. We also give two examples: 1) obstacle avoidance tasks for mobile robots; and 2) symbol grounding for natural language instruction, for realization of user's adaptation of human-robot interaction.

We propose an approach "online visual feature determination through discourse", which realizes the finding and the following task in a real environment. To segment the human's image from the complex background, it is possible to prepare many finds of basic visual features, and to combine them. The proposed approach is a solution for the problem of how to select and combine these visual features for each situation. Namely, a robot and a human make a conversation to search for the suitable visual features for the current situation. This idea enables the robot to find and follow a human successfully in a changeable background environment.