Virtual reality, which enables users to engage in physical activities in ways distinct from those in the real world, is increasingly recognized for its potential to enhance motor skill acquisition. Research on co-embodiment learning, in which instructors and learners utilize a single avatar that represents a weighted average of their movements, has demonstrated its efficacy in facilitating motor skill development. However, the current implementation of co-embodiment learning necessitates the real-time participation of instructors proficient in both virtual reality and co-embodiment, which poses challenges for its widespread adoption. To address this limitation, this study proposed a method for developing instructors trained on human motor data to effectively support motor skill learning through co-embodiment. The AI model was trained using supervised learning on data obtained from human motor learning sessions that employed co-embodiment. To evaluate the performance of the AI instructor, we compared the learning performance in co-embodiment learning with that of the AI instructor, recorded human instructor data, and a human instructor as well as in solo learning. The results showed that practicing with the AI instructor significantly improved learning efficiency compared with practicing alone or with recorded data and was comparable to that achieved by practicing with a human instructor.

Virtual reality is gaining attention as a tool to facilitate motor skill learning. Numerous studies have been conducted on virtual co-embodiment, in which the movements of the teacher and student are weighted and averaged into a single avatar for motor skill learning. Previous studies have shown that virtual co-embodiment with a human teacher enhances motor skill learning efficiency, and the behavior of the human teacher is important for effective learning. However, this system has some challenges, such as the human playing the teacher’s role must be skilled in teaching and using virtual co-embodiment, and the teacher can be adversely affected by the learner. To solve these problems, we created an AI teacher using long short-term memory, which outputs the behavior of the teacher based on the input of the learner’s behavior data and the state of the experimental environment and trained the AI teacher by supervised learning using behavior training data of a human virtual co-embodiment. We confirmed that this AI teacher can generate behaviors similar to those of the human teacher and investigated the efficiency of motor skill learning using virtual co-embodiment with the AI teacher. Co-embodiment with the AI teacher reduced performance during the learning phase but improved performance during subsequent independent task execution. We further analyzed the assist proportion of the teacher and observed that when the co-embodied with an AI teacher, the assist proportion is lower than when the co-embodied with a human, suggesting that learning efficiency may be enhanced when the co-embodied partner is a mixture of supportive and obstructive.

This study investigated the learning effect and skill retention when virtual co-embodiment, in which movements of people were weighted and averaged into a single avatar, was used to learn motor skills requiring declarative memory. Previous studies have shown that virtual co-embodiment promotes the efficiency of motor skill learning, which relies on procedural memory such as movement procedures. However, declarative memory, such as the connection between specific instructions and actions, plays an important role in learning motor skills. This study compared the learning efficiency and skill retention after one week of repeatedly performing the task of touching a specified combination of virtual spheres with both hands in accordance with the symbols presented, using virtual co-embodiment and in a condition in which the task was performed alone. The results showed that virtual co-embodiment improves learning efficiency for motor skills related to declarative memory as well as procedural memory and promotes long-term retention of skills.

Virtual reality (VR) is a promising tool for motor skill learning. Previous studies have indicated that observing and following a teacher's movements from a first-person perspective using VR facilitates motor skill learning. Conversely, it has also been pointed out that this learning method makes the learner so strongly aware of the need to follow that it weakens their sense of agency (SoA) for motor skills and prevents them from updating the body schema, thereby preventing long-term retention of motor skills. To address this problem, we propose applying "virtual co-embodiment" to motor skill learning. Virtual co-embodiment is a system in which a virtual avatar is controlled based on the weighted average of the movements of multiple entities. Because users in virtual co-embodiment overestimate their SoA, we hypothesized that learning using virtual co-embodiment with a teacher would improve motor skill retention. In this study, we focused on learning a dual task to evaluate the automation of movement, which is considered an essential element of motor skills. As a result, learning in virtual co-embodiment with the teacher improves motor skill learning efficiency compared with sharing the teacher's first-person perspective or learning alone.

Virtual reality helps us learn complex motor skills by providing a situation in which we observe or follow a teacher’s movements from a first-person perspective. However, it has been suggested that if the learners themselves do not behave actively, their body schemes will not be updated and motor skills will be acquired temporarily, but will not be retained in the long term. As a solution to this problem, “co-embodiment” in which two people embody an avatar that reflects the weighted average of their movements was proposed, and it is shown that the user can feel an excessive sense of agency (SoA) even when their control weight is small. From the perspective of motor skill learning, the learner must feel as strong a SoA as possible while performing the exercise as close to the teacher as possible. Therefore, in this study, we propose a method to transitionally change the weights in a situation where co-embodiment is used, such that a strong SoA is felt despite the high weights of control by others. Considering the two-step account of the agency model, which states that the SoA is influenced by context, we tested the hypothesis that an initially strong SoA can maintain the SoA despite a gradually decreased control weight. The experimental results support this hypothesis, and it is expected that the proposed method will enhance the effectiveness of motor skill learning using co-embodiment.

Emotional contagion is a phenomenon in which one’s emotions are transmitted among individuals unconsciously by observing others’ emotional expressions. In this paper, we propose a method for mediating people’s emotions by triggering emotional contagion through artificial bodily changes such as pseudo tears. We focused on shedding tears because of the link to several emotions besides sadness. In addition, it is expected that shedding tears would induce emotional contagion because it is observable by others. We designed an eyeglasses-style wearable device, Teardrop glasses, that release water drops near the wearer’s eyes. The drops flow down the cheeks and emulate real tears. The study revealed that artificial crying with pseudo tears increased sadness among both wearers and those observing them. Moreover, artificial crying attenuated happiness and positive feelings in observers. Our findings show that actual bodily changes are not necessary for inducing emotional contagion as artificial bodily changes are also sufficient.

In this article, we introduce a concept called "virtual co-embodiment", which enables a user to share their virtual avatar with another entity (e.g., another user, robot, or autonomous agent). We describe a proof-of-concept in which two users can be immersed from a first-person perspective in a virtual environment and can have complementary levels of control (total, partial, or none) over a shared avatar. In addition, we conducted an experiment to investigate the influence of users' level of control over the shared avatar and prior knowledge of their actions on the users' sense of agency and motor actions. The results showed that participants are good at estimating their real level of control but significantly overestimate their sense of agency when they can anticipate the motion of the avatar. Moreover, participants performed similar body motions regardless of their real control over the avatar. The results also revealed that the internal dimension of the locus of control, which is a personality trait, is negatively correlated with the user's perceived level of control. The combined results unfold a new range of applications in the fields of virtual-reality-based training and collaborative teleoperation, where users would be able to share their virtual body.

Hand interaction techniques in virtual reality often exploit visual dominance over proprioception to remap physical hand movements onto different virtual movements. However, when the offset between virtual and physical hands increases, the remapped virtual hand movements are hardly self-attributed, and the users become aware of the remapping. Interestingly, the sense of self-attribution of a body is called the sense of body ownership (SoBO) in the field of psychology, and the realistic the avatar, the stronger is the SoBO. Hence, we hypothesized that realistic avatars (i.e., human hands) can foster self-attribution of the remapped movements better than abstract avatars (i.e., spherical pointers), thus making the remapping less noticeable. In this article, we present an experiment in which participants repeatedly executed reaching movements with their right hand while different amounts of horizontal shifts were applied. We measured the remapping detection thresholds for each combination of shift directions (left or right) and avatar appearances (realistic or abstract). The results show that realistic avatars increased the detection threshold (i.e., lowered sensitivity) by 31.3 percent than the abstract avatars when the leftward shift was applied (i.e., when the hand moved in the direction away from the body-midline). In addition, the proprioceptive drift (i.e., the displacement of self-localization toward an avatar) was larger with realistic avatars for leftward shifts, indicating that visual information was given greater preference during visuo-proprioceptive integration in realistic avatars. Our findings quantifiably show that realistic avatars can make remapping less noticeable for larger mismatches between virtual and physical movements and can potentially improve a wide variety of hand-remapping techniques without changing the mapping itself.

Preventing users from walking through virtual boundaries (e.g., walls) is an important issue to be addressed in room-scale virtual environments (VEs), considering the safety and design limitations. Sensory feedback from wall collisions has been shown to be effective; however, it can disrupt the immersion. We assumed that a greater sense of presence would discourage users from walking through walls and conducted a two-factor between-subjects experiment (N = 92) that controls the anthropomorphism (realistic or abstract) and visibility (full-body or hand-only) of self-avatars. We analyzed the participants' behaviors and the moment they first penetrated the wall in game-like VEs that gradually instigated participants to penetrate the walls. The results showed that the realistic full-body self-avatar was the most effective for discouraging the participants from penetrating the walls. Furthermore, the participants with lower presence tended to walk through the walls sooner. This study can contribute to applications that require realistic user responses in VEs.

In this paper, we present a study of redirected walking (RDW) that shifts the positional relationship between visual and auditory cues during curvature manipulation. It has been shown that, when presented with incongruent visual and auditory spatial cues during a localization task, human observers integrate that information based on each cue's relative reliability, which determines their final perception of the target object's location. This multi-modal integration model is known as maximum likelihood estimation (MLE). By altering the visual location of objects that users perceive in virtual reality (VR) through auditory cues during redirection manipulation, we expect fewer users to notice the manipulation, which helps increase the usable curvature gain. Most existing studies on MLE in multi-modal integration have used random-dot stereograms as visual cues under stable motion states. In the present study, we first investigated whether this model holds while walking in VR environment. Our results indicate that in a walking state, users' perceptions of the target object's location shift toward auditory cue as the reliability of vision decreases, in keeping with the trend shown in previous studies on MLE. Based on this result, we then investigated the detection threshold of curvature gains during redirection manipulation under a condition with congruent visual-auditory cues as well as a condition in which users' location perceptions of the target object are considered to be affected by the incongruent auditory cue. We found that the detection threshold of curvature gains was higher with incongruent visual-auditory cues than with congruent cues. These results show that incongruent multimodal cues in VR may have a promising application in the area of redirected walking.

Studies have shown that the sense of body ownership towards virtual humanoid avatars with additional body parts can be successfully elicited when synchronous visuomotor and/or visuotactile feedback is given. In an interactive virtual reality (VR) application, however, it is difficult for users to uninterruptedly observe the added body parts, especially when they are attached to the backs of the avatars. Thus, the embodiment of such body parts needs to be achieved using limited synchronous visuomotor feedback. Commonly, it is specifically done by looking at a virtual mirror reflection of the avatar’s movement. However, the methodology of eliciting the sense of body ownership in such conditions remains to be studied.

Humans can estimate the shape of a wielded object through the illusory feeling of the mass properties of the object obtained using their hands. Even though the shape of hand-held objects influences immersion and realism in virtual reality (VR), it is difficult to design VR controllers for rendering desired shapes according to the perceptions derived from the illusory effects of mass properties and shape perception. We propose Transcalibur, which is a hand-held VR controller that can render a 2D shape by changing its mass properties on a 2D planar area. We built a computational perception model using a data-driven approach from the collected data pairs of mass properties and perceived shapes. This enables Transcalibur to easily and effectively provide convincing shape perception based on complex illusory effects. Our user study showed that the system succeeded in providing the perception of various desired shapes in a virtual environment.

We present PaCaPa, a handheld device that renders haptics on a user's palm when the user interacts with virtual objects using virtual tools such as a stick. PaCaPa is a cuboid device with two wings that open and close. As the user's stick makes contact with a virtual object, the wings open by a specific degree to dynamically change the pressure on the palm and fingers. The open angle of the wings is calculated from the angle between the virtual stick and hand direction. As the stick bites into the target object, a large force is generated. Our device enables three kinds of renderings: size, shape, and stiffness. We conducted user studies to evaluate the performance of our device. We also evaluated our device in two application scenarios. User feedback and qualitative ratings indicated that our device can make indirect interaction with handheld tools more realistic.

Redirection makes it possible to walk around a vast virtual space in a limited real space while providing a natural walking sensation by applying a gain to the amount of movement in a real space. However, manipulating the walking path while keeping it and maintaining the naturalness of walking when turning at a corner cannot be achieved by the existing methods. To realize natural manipulation for turning at a corner, this study proposes novel “turning gains”, which refer to the increase in real and virtual turning degrees. The result of an experiment which aims to estimate the detection thresholds of turning gains indicated that when the turning radius is 0.5 m, discrimination is more difficult compared with the rotation gains (r = 0.0m).

How does the representation of an embodied avatar influence the way in which one perceives the scale of a virtual environment? In virtual reality, it is common to embody avatars of various appearances, from abstract to realistic. It is known that changes in the realism of virtual hands affect the self-body perception, including body ownership. However, the influence of self-avatar realism on the perception of non-body objects has not been investigated. Considering the theory that the scale of the external environment is perceived relative to the size of one's body (body-based scaling), it can be hypothesized that the realism of an avatar affects not only body ownership but also the fidelity of the avatar with respect to our own body as a metric. Therefore, this study examines how avatar realism affects perceived object sizes as the size of the virtual hand changes. In the experiment, we manipulate the level of realism (realistic, iconic, and abstract) and size (veridical and enlarged) of the virtual hand and measure the perceived size of a cube. The results show that the size of the cube is perceived to be smaller when the virtual hand is enlarged compared to when it is veridical, indicating that the participants perceive the sizes of objects based on the size of the avatar, only in the case of a highly realistic hand. Our findings indicate that the more realistic the avatar, the stronger is the sense of embodiment including body ownership, which fosters scaling the size of objects using the size of the body as a fundamental metric. This provides evidence that self-avatar appearances affect how we perceive not only virtual bodies but also virtual spaces.

During group decision-making, a group with an imbalanced minority and majority typically suffers normative bias. This bias creates socio-emotional conflict and decreases group member consent and decision power. Thus, we propose a novel video-chat system, "Divided Presence," which aims to reduce this bias by equalizing the apparent number of majority and minority participants using a pseudo-population increase. Members of a discussion use computer graphics avatars on monitors instead of their actual video appearances during communication. The apparent number of discussion members increases when two avatars are assigned to an arbitrary minority member; the avatars then speak on behalf of him or her. We evaluate the system with a three-person consensus game, where a minority member is assigned two avatars. The results show that our system increases the degree of consent among the majority participants when they finally agree to the minority opinion.

PaCaPa is a handheld device that presents haptic stimuli on a user's palm when the user interacts with virtual objects using virtual tools. It is a box-shaped device and has two wings. This device can present the contact force and contact angle by opening and closing the wings based on the angle between the direction of the virtual tool and hand. By changing these haptic sensations on the palm and fingers, it enables the users to perceive different size, shape, and stiffness of virtual objects.

How does the representation of an embodied avatar influence the way in which a human perceives the scale of a virtual environment? It has been shown that the scale of the external environment is perceived relative to the size of one's body. However, the influence of avatar realism on the perceived scale has not been investigated, despite the fact that it is common to embody avatars of various representations, from iconic to realistic. This study examined how avatar realism would affect perceived graspable object sizes as the size of the avatar hand changes. In the experiment, we manipulated the realism (high, medium, and low) and size (veridical and enlarged) of the avatar hand, and measured the perceived size of a cube. The results showed that the size of the cube was perceived to be smaller when the avatar hand was enlarged for all degrees of realism of the hand. However, the enlargement of the avatar hand had a greater influence on the perceived cube size for the highly realistic avatar than for the medium-level and low-level realism conditions. This study shed new light on the importance of the avatar representation in a three-dimensional user interface field, in how it can affect the manner in which we perceive the scale of a virtual environment.

The contribution of this paper is to propose a novel Redirected Walking (RDW) technique that adopts manipulation gain in the roll direction. RDW is a technique that enables users to explore a large Virtual Environment (VE) while walking within a physically limited space by manipulating their virtual vision. Thus far, studies have determined the detection thresholds for translation, rotation, and curvature gains, but the thresholds are limited in the yaw direction. In contrast, in other research areas, movements in the yaw and roll directions have sometimes been addressed at the same time. In the present study, we investigated the detection threshold of inclination gain in the roll direction. We applied an inclination gain that was increased gradually while the participants walked 3 meters straight ahead. The results showed that users can detect an inclination gain of 1.93° on the left hand and 1.39° on the right hand.

A redirected walking (RDW) techniques enable users to walk around infinite virtual environments (VEs) in a finite physical space. In previous studies on RDW, many researchers have discussed manipulations in the yaw direction, but few have tackled with redirection in pitch and roll directions. We propose a novel VR system, which realizes pitch and roll redirections and allows users to experience walking on the 3D model of Mobius Strip in the VE.

For many automated navigation applications, the underlying localization algorithm must be able to continuously produce both accurate and stable results by using a spectrum of redundant sensing technologies. To this end, various sensors have been used for localization, such as Wi-Fi, Bluetooth, GPS, LiDAR and cameras. In particular, a class of vision-based localization techniques using Structure from Motion (SfM) has been shown to produce very accurate position estimates in the real-world with moderate assumptions about the motion of the camera and the amount of visual texture in the environment. However, when these assumptions are violated, SfM techniques can fail catastrophically (i.e., cannot generate any estimate). Recently, a deep convolutional neural network (CNN) has been applied to images to robustly regress 6-DOF camera poses at the cost of lower accuracy than SfM. In this work, we propose improving image-based localization accuracy of deep CNN by combining Bluetooth radio-wave signal readings. In our experiments, we show that our proposed dual-stream CNN can robustly regress 6-DOF poses from images and radiowave signals better than one sensing modality alone. More importantly, we show that when both modes are used, the localization accuracy of the proposed deep CNN is comparable to that of SfM and significantly more robust than SfM.

This paper presents a novel interactive system that provides users with virtual reality (VR) experiences, wherein users feel as if they are ascending/descending stairs through passive haptic feedback. The passive haptic stimuli are provided by small bumps under the feet of users; these stimuli are provided to represent the edges of the stairs in the virtual environment. The visual stimuli of the stairs and shoes, provided by head-mounted displays, evoke a visuo-haptic interaction that modifies a user's perception of the floor shape. Our system enables users to experience all types of stairs, such as half-turn and spiral stairs, in a VR setting. We conducted a preliminary user study and two experiments to evaluate the proposed technique. The preliminary user study investigated the effectiveness of the basic idea associated with the proposed technique for the case of a user ascending stairs. The results demonstrated that the passive haptic feedback produced by the small bumps enhanced the user's feeling of presence and sense of ascending. We subsequently performed an experiment to investigate an improved viewpoint manipulation method and the interaction of the manipulation and haptics for both the ascending and descending cases. The experimental results demonstrated that the participants had a feeling of presence and felt a steep stair gradient under the condition of haptic feedback and viewpoint manipulation based on the characteristics of actual stair walking data. However, these results also indicated that the proposed system may not be as effective in providing a sense of descending stairs without an optimization of the haptic stimuli. We then redesigned the shape of the small bumps, and evaluated the design in a second experiment. The results indicated that the best shape to present haptic stimuli is a right triangle cross section in both the ascending and descending cases. Although it is necessary to install small protrusions in the determined direction, by using this optimized shape the users feeling of presence of the stairs and the sensation of walking up and down was enhanced.

"Mirroring" refers to the unconscious mimicry of another person's behaviors, such as their facial expressions. Mirroring has many positive effects, such as enhancing closeness and improving the flow of a conversation, which enriches the quality of communication. Our study set out to devise a means of evoking these positive effects in a video chat without any conscious effort of participants. We constructed a videophone system, called FaceShare, which can deform the user's face into a smile in response to their partner's smiling. That is, our system generates mirroring by producing a pseudo-smile through image processing. We conducted an experiment in which pairs of participants had brief conversations via FaceShare. The results implied that mirroring using the pseudo-smile lets the mimicker, whose face is deformed according to the expressions of their partner, feel a closeness, and improves the flow of the conversation for both the mimicker and the mimickee, who sees the mimicker's deformed face.

As intelligent agents learn to behave increasingly autonomously and simulate a high level of intelligence, human interaction with them will be increasingly unpredictable. Would you accept an unexpected and sometimes irrational but actually correct recommendation by an agent you trust? We performed two experiments in which participants played a game. In this game, the participants chose a path by referring to a recommendation from the agent in one of two experimental conditions:the correct or the faulty condition. After interactions with the agent, the participants received an unexpected recommendation by the agent. The results showed that, while the trust measured by a questionnaire in the correct condition was higher than that in the faulty condition, there was no significant difference in the number of people who accepted the recommendation. Furthermore, the trust in the agent made decision time significantly longer when the recommendation was not rational.

To effectively utilize gaze in one-to-many communication, robots must direct and avert their gaze with appropriate timing. However, few studies have focused on this aspect of communication because of the difficulty in detecting the position of many gaze recipients and directing appropriate gazes to each. This paper proposes a system of robotic eyes that can send averted or direct gazes simultaneously to multiple recipients by switching between convex and hollow eye shapes, which we call Transgazer.

State space models, such as Kalman filters or Particle filters, have been applied to improve the accuracy of radio-wave-based localization. However, these models can drift radically when assumptions of the models are violated, and they do not have a mechanism to fix errors. Therefore, we propose an approach to apply supervised learning to pedestrian localization, which is based on the Inference Machines framework. During training, we collect localization ground truths using computer vision while also collecting Bluetooth signals to train a state space model for localization, which can recover from model drift. During testing, our proposed approach uses only Bluetooth signals. Our experimental results show that our approach can improve the accuracy of Bluetooth-based localization with a small number of training examples. Moreover, our multi-modal supervision can also be used to estimate additional parameters, such as device rotation, from Bluetooth signals that do not have such information.

Redirected walking allows users to explore a large virtual environment while there is a limitation of the room size. Previous works tried to present users straight path in a virtual environment while they walked on a curved path in reality. We expand a previous technique to present users a various curved path in a virtual environment while they walked on a particular curved path or a straight path with/without haptics. Furthermore, we propose a novel estimation methodology to quantify walking paths which user has thought he walked in reality. The data from our experiment shows that users feel walking a various curved path in VR as same as one-to-one mapping condition.

Humans are capable of haptically perceiving the shape of an object by simply wielding it, even without seeing it. On the other hand, typical hand-held controllers for virtual reality (VR) applications are pre-designed for general applications, and thus not capable of providing appropriate haptic shape perception when wielding specific virtual objects. Contradiction between haptic and visual shape perception causes a lack of immersion and leads to inappropriate object handling in VR. To solve this problem, we propose a novel method for designing hand-held VR controllers which illusorily represent haptic equivalent of visual shape in VR. In ecological psychology, it has been suggested that the perceived shape can be modeled using the limited mass properties of wielded objects. Based on this suggestion, we built a shape perception model using a data-driven approach; we aggregated data of perceived shapes against various hand-held VR controllers with different mass properties, and derived the model using regression techniques. We implemented a design system which enables automatic design of hand-held VR controllers whose actual shapes are smaller than target shapes while maintaining their haptic shape perception. We verified that controllers designed with our system can present aimed shape perception irrespective of their actual shapes.

Agents like human, such as humanoid robots or avatars can be felt as if they have and communicate and communicate due to manipulation of the bodily information. Meanwhile, as in the case of Internet bot, it is still difficult to communiate the emotion described as text, let alone empathizing due to degradation of information online. The current study proposes a method for experiencing emotion on the Internet by reproducing a mechanism of evoking emotion. This method evokes a number of emotions described on the Web, by changing of self-physiological perception with sensory stimuli. To investigate the feasibility of our method, we made a system named "Communious Mouse." This system rewrites the perception of self-skin temperature and pulse in a palm by presenting vibration and thermal stimulation through a mouse device for evoking emotion. The current paper discusses the feasibility of our method based on the obtained feedbacks through an exhibition of the system.

We propose a navigation interface, which enables users to control locomotion and the speed intuitively in virtual environments constructed with spherical images. In these virtual environments, users can only pass through lines where the camera has passed through during recording. Therefore, it is necessary to inform users of the accessible directions. Furthermore, velocity control is needed for close exploration in virtual environments. Therefore, we propose the Motive Compass input interface, which intuitively shows users the accessible directions and enables them to control the velocity of locomotion. We conducted a large-scale demonstration experiment in a real exhibition and showed that our interface more effectively presents accessible directions and enables users to control their velocities.

Lately, mobile augmented reality (AR) has become very popular and is used for many commercial and product promotional activities. However, in almost all mobile AR applications, the user only views annotated information or the preset motion of the virtual object in an AR environment and is unable to interact with the virtual objects as if he/she were interacting with real objects in the real environment. In this paper, in an attempt to realize enhanced intuitive and realistic object manipulation in the mobile AR environment, we propose an integrated view-input AR interaction method, which integrates user device manipulation and virtual object manipulation. The method enables the user to hold a 3D virtual object by touching the displayed object on the 2D touch screen of a mobile device, and to move and rotate the object by moving and rotating the mobile device while viewing the held object by way of the 2D screen of the mobile device. Based on this concept, we implemented three types of integrated methods, namely the Rod, Center, and Touch methods, and conducted a user study to investigate the baseline performance metrics of the proposed method on an AR object manipulation task. The Rod method achieved the highest success rate (91%). Participants' feedback indicated that this is because the Rod method is the most natural, and evoked a fixed mental model that is conceivable in the real environment. These results indicated that visualizing the manipulation point on the screen and restricting the user's interactivity with virtual objects from the user's position of view based on a conceivable mental model would be able to aid the user to achieve precise manipulation.

Surface and object recognition is of significant importance in ubiquitous and wearable computing. While various techniques exist to infer context from material properties and appearance, they are typically neither designed for real-time applications nor for optically complex surfaces that may be specular, textureless, and even transparent. These materials are, however, becoming increasingly relevant in HCI for transparent displays, interactive surfaces, and ubiquitous computing. We present SpecTrans, a new sensing technology for surface classification of exotic materials, such as glass, transparent plastic, and metal. The proposed technique extracts optical features by employing laser and multi-directional, multi-spectral LED illumination that leverages the material's optical properties. The sensor hardware is small in size, and the proposed classification method requires significantly lower computational cost than conventional image-based methods, which use texture features or reflectance analysis, thereby providing real-time performance for ubiquitous computing. Our evaluation of the sensing technique for nine different transparent materials, including air, shows a promising recognition rate of 99.0%. We demonstrate a variety of possible applications using SpecTrans' capabilities.

This paper proposes a high-definition digital display case for manipulating a virtual exhibit that has linking mechanisms. This technique enhances the understanding of dynamic exhibits. It is difficult to construct interactive contents of dynamic virtual exhibits, because measuring the mechanism invokes the risk of an exhibit's deterioration, and it takes tremendous efforts to create a fine spun computer graphics (CG) model for mechanisms. Therefore, we propose an image-based interaction method that uses image-based rendering to construct interactive contents for dynamic virtual exhibits using the interpolation between exhibit pictures with a number of deformational conditions and viewpoints. Using this method, we construct a high-definition digital showcase and exhibit the interactive content at a museum to evaluate the availability of our system.

In this paper we developed the free-viewpoint curved surface shape display, using the effect of visuo-haptic interaction. In our research, we aim to realize the simple mechanic visuo-haptic system with which we can touch various objects with our real hands. We proposed the free-viewpoint shape display system in which users can touch a virtual object with various shape through the tablet device, while walking around the object. The system figured the difference of shape between the real object and the virtual one from the current viewpoint, and calculates the amount of displacement of the touching hand image to fit to the virtual object, in real time. This modification of the movement of a hand image evokes the effect of the visuo-haptic interaction, and enable users to feel touching various shapes from various viewpoints, although actually users touch a physically static object.

In this study, we developed a system for mobile devices designed to provide a virtual experience of past scenery depicted in old photographs by superimposing them on landscapes in video see-through frames. A user is asked to capture a photograph of a landscape to be used as a keyframe and enter correspondence points between the new and old photos. The old photograph is deformed by projective transform with homography computed based on correspondence points. We then superimpose of the old photograph onto video see-through frames of current landscape. To achieve real-time and robust superimposition on mobile devices, both motion-sensor-based pose information and camera-image-keypoint-tracking-based pose information is used for device's camera pose tracking.

The main objective of this study is to create a method that can affect flavor perception, without changing the food itself, by applying thermal sensations to the skin around the nose to simulate skin temperature changes associated with pleasant and unpleasant feelings. Recent physiological research has demonstrated that skin temperature changes affect flavor perception. Based on the James--Lange theory, which claims that changes in bodily responses unconsciously evoke emotions and change our perception related to an emotional state, we hypothesized that flavor perception can be modified by affecting the skin temperature properly. To test this theory, we developed an "Affecting Tumbler," which presents thermal sensations to the skin around the nose for simulating the skin's temperature response during drinking. Our user study suggested that flavor richness and aftertaste strength were significantly improved by heating up the skin in the nasal region. These results indicate that flavor perception can be controlled with the proposed method.

This study develops a method for improving creativity by changing the facial appearance of people during video conferences. We focus on enhancing creativity during human interactions, especially cooperative work situations such as video conferences. Psychological studies have revealed that the facial appearance of people can affect creativity, i.e., "emotion can affect creativity" and "the appearance of others can affect the emotions." Based on the knowledge, we hypothesized that the media technology which modifies the facial appearance of people appropriately can enhance our creativity during cooperative work. Therefore, we develop a method to affect creativity by changing the facial appearance in real-time, such as facial expressions and facial resemblance. We test the effectiveness of this method by building a system for collaborative video conferences. The user study show that the proposed method could enhance creativity.

In this paper, we proposed a visuo-haptic system for displaying various shapes which have curve, edge, and inclined surfaces, using a simple transmutative physical device and the effect of visuo-haptic interaction. We aim to construct a perception-based shape display system to provide users with the sensation of touching virtual objects of varying shapes using only a simple mechanism. We have confirmed that the perception of each primitive shape such as curvature and angle could be modified by displacing a user's hand image on the monitor as if s/he were touching the visual shape while actually touching another shape. In this study, we constructed the method to merge these findings for displaying more various shapes, including angular ones. We built a transmutative device, which the user touches. The device does not undergo significant transformation, but its surface can be slightly bumped in and out, and displayed various shapes with various angles, length and curvature. The results of experimental trials confirmed that our method for displaying each primitive shape can also worked as designed when we combine these findings to display more complex objects using this device which transforms slightly.

Accessibility information can allow disabled people to identify suitable pathways to reach their destinations, but it is difficult to obtain new accessible pathway information rapidly because of limited local information disclosure. Thus, it is necessary to develop a comprehensive system that acquires barrier-free information from various sources and makes that information available in an intuitive form. In this study, we aimed to develop a social platform to obtain and present appropriate information depending on the user's situation, such as the user's disabilities and location, and to share the barrier-free information provided by other users.

The main contribution of this paper is to develop a method for alleviating fatigue during handling medium-weight objects and augmenting our endurance by affecting our weight perception with augmented reality technology. To assist people to lift medium-weight objects without a complex structure or various costs, we focus on the phenomenon that our weight perception during handling objects is affected by visual properties. Our hypothesis is that this illusionary effect in weight perception can be applied to reduce fatigue while handling medium-weight objects without mechatronics-based physical assistance. In this paper, we propose an augmented reality system that changes the brightness value of an object in order to reduce fatigue while handling the object. We conducted two fundamental experiments to investigate the effectiveness of the proposed system. Our results suggested that the system eliminates the need to use excess energy for handling objects and reduces fatigue during the handling task.

Currently, human-computer interaction (HCI) is primarily focused on human-centric interactions; however, people experience many nonhuman-centric interactions during the course of a day. Interactions with nature, such as experiencing the sounds of birds or trickling water, can imprint the beauty of nature in our memories. In this context, this paper presents an interface of such nonhuman interactions to observe people's reaction to the interactions through an imaginable interaction with a mythological creature. Tele Echo Tube (TET) is a speaking tube interface that acoustically interacts with a deep mountain echo through the slightly vibrating lampshade-like interface. TET allows users to interact with the mountain echo in real time through an augmented echo-sounding experience with the vibration over a satellite data network. This novel interactive system can create an imaginable presence of the mythological creature in the undeveloped natural locations beyond our cultural and imaginable boundaries. The results indicate that users take the reflection of the sound as a cue that triggers the nonlinguistic believability in the form of the mythological metaphor of the mountain echo. This echo-like experience of believable interaction in an augmented reality between a human and nature gave the users an imaginable presence of the mountain echo with a high degree of excitement. This paper describes the development and integration of nonhuman-centric design protocols, requirements, methods, and context evaluation.

This paper reports a reliving video experience system using augmented reality techniques. This system overlays a scene from video material onto the real world and allows users to experience how the camera operator captured the scene by inducing them to move as in the same way as the camera operator did. We propose novel interaction techniques that balance the competing goals of inducing users to move in the same way as the camera operator and allowing them to feel that they are observing freely. The proposed methods implement three induction techniques to induce the user to avoid unintentional movements, and to start moving the device and rotate it appropriately. We implemented this system in an exhibition system that relives the video of old railways and exhibited it at the railway museum for two weeks. The analysis results of users' operational logs and questionnaires suggest that the proposed system and interaction techniques are effective for reliving video experiences in the real world.

The main contribution of this paper is to realize a method for modifying perception of satiety and controlling nutritional intake by changing the apparent size of food with augmented reality. As a possible method for decreasing rates of obesity, we focused on controlling food intake implicitly without any effort.

In our research, we aim to construct the visuo-haptic system which can give users a sense as if they were touching on virtual objects with variety of shapes, using pseudo-haptiec effects. In this paper, we focus on modifying the identification of a position of edges on a object when touching it with a pointing finger, by displacing the visual representation of the user's hand in order to construct a novel visuo-haptic system. We compose a video see-through system, which enables us to change the perception of the shape of an object a user is visually touching, by displacing the visual representation of the user's hand as if s/he was touching the visual shape, when in actuality s/he is touching another shape. We had experiments and showed participants perceived positions of edges that was the same as the one they were visually touching, even though the positions of edges they were actually touching was different. These results prove that the perceived positions of edges could be modified, and even if the ratio of the successive distance between edges is 1 : 1, we can modify the perception of this ratio from 3 : 2 to 2 : 3.

The articles in this special section contain selected papers from the IEEE Virtual Reality Conference (VR).

The main contribution of this paper is to realize computer generated augmented flavors and establish a method to integrate gustatory information into computer human interactions. There are several reasons for the scarcity of research on gustatory information. One reason is that taste sensations are affected by a number of factors, such as vision, olfaction and memories. This produces a complex cognition mechanism for a user's gustatory sensation, and makes it difficult to build up a gustatory display which produces a specific taste on demand.

In this paper, we propose a ubiquitous augmented reality system that expands projected augmentations in wide area by using visible light communication and wireless tag. The system consists of wireless tags and a high-speed projector (DMD Discovery 3000), operating in 8kHz, attached on mobility platform. The projector transmits structured light patterns that contain different information in each pixel. Wireless tags can recognize their own locations in projected area by receiving structured light patterns projected from the projector. Since 3D coordinates of individual tag is stored in the system, projection point is estimated with 3D coordinates and pixel coordinates of each tag in projected area. According to the estimated position and orientation, augmented reality content is superimposed. The set of structured light patterns are projected in sufficient speed that human eye can only perceive the superimposed content. We have also evaluated the accuracy of superimposing augmented reality content on real world from moving projection points.

The purpose of this research is to support users' mealtime communication by changing their utterance rate during meals. By logging and analyzing the utterance rate, or the ratio of utterance length to mealtime length, we determined the relationship between the utterance rate and the distribution of the community to which each meal companion belonged. Using this relationship, we developed a real-time feedback system that presented a pre-estimated utterance rate with rate at the present time during a meal. This utterance rate was estimated according to logged utterance rate data and input data about members at the table. In order to evaluate our system, we asked six users to use it. As a result, we found out that users almost always wanted to increase the utterance rate and users' utterance rates increased as they wanted after using our feedback system.

Welcome to IEEE Virtual Reality 2011! Similar to previous years, the conference program is mostly made up of single track presentations. There are fourteen long paper presentations, eighteen short presentations, and three panels. This year, we have some new improvements to the program including the addition of four invited presentations from the highly regraded IEEE Transactions on Visualization and Computer Graphics journal.

This study tries to increase the effects of odors by controlling the odor emissions. We developed an olfactory feedback system to maintain user's concentration effectively based on biological information. The system presumes the level of user's concentration based on an electrocardiogram analysis. The analysis is carried out using variance of R-R intervals and frequency spectra. According to the presumed concentration level, the system presents an odor with an awakening effect to maintain user's concentration. We performed experiments in which subjects did simple tasks with this system, changing the method of controlling odor emissions. We compared the results of each method and evaluated the effectiveness of our olfactory feedback system and the methods of controlling odor emissions. The comparison of the results showed that presenting an odor when a user loses his/her concentration is effective to decrease errors of addition tasks.

In this paper, we propose a pseudo-gustatory display based on the cross-modal interactions that underlie the perception of flavor. Although several studies on visual, auditory, haptic, and olfactory displays have been conducted, gustatory displays have seldom been studied. This scarcity has been attributed to the fact that synthesizing arbitrary taste from basic tastants is difficult. On the other hand, it has been noted that the perception of taste is influenced by visual cues, auditory cues, smell, the trigeminal system, and touch. In our research, we aim at utilizing this influence between modalities for realizing a "pseudo-gustatory" system that enables the user to experience various tastes without changing the chemical composition of foods. Based on this concept, we built a "Meta Cookie" system to change the perceived taste of a cookie by overlaying visual and olfactory information onto a real cookie which an AR marker pattern. We performed an experiment that investigates how people experience the flavor of a plain cookie by using our system. The result suggests that our system can change the perceived taste based on the effect of the cross-modal interaction of vision, olfaction and gustation.

In our research demonstration, we show a "MetaCookie+" that enables the user to experience various tastes without changing the chemical composition of foods by using the influence between modalities. It is a pseudo-gustatory display by combining the Edible Marker system which can detect the state [number/shape/6-DOF coordinate] of each piece of bitten or divided food in real time, and the "Pseudo-gustation" method to change the perceived taste of food by changing its appearance and scent.

In this research study, we propose a divided planar-object detection method for augmented reality(AR) applications. There are mainly two types of camera-registration methods for AR applications: marker-based methods, and natural-feature-based methods. In addition, the latter methods are classified into visual SLAM and object detection methods. With respect to object detection methods, particularly for planar objects such as paper, methods for dealing with bending, folding, and occlusion are proposed. However, the division of objects has not been studied. Once an object is divided, a conventional object detection method cannot identify each of the pieces because the feature points of only a single piece are recognized as the target object, and the other feature points are regarded as outliers. The proposed system prepares a database of the target object's natural features, and applies progressive sample consensus(PROSAC), which is a robust estimation method, for iterative homography calculation to achieve the multiple planar-object detection. Moreover, the proposed method can detect shapes of pieces by simultaneously using an occlusion detection method. We demonstrate that it is possible to interact with an arbitrarily divided planar object in real time by our method to implement some AR applications.

In this research study, we propose a divided planar-object detection method for augmented reality(AR) applications. The proposed system prepares a database of the target object's natural features, and applies progressive sample consensus (PROSAC), which is a robust estimation method, for iterative homography calculation to achieve the multiple planar-object detection. Moreover, the proposed method can detect shapes of pieces by simultaneously using an occlusion detection method. We demonstrate that it is possible to interact with an arbitrarily divided planar object in real time by our method to implement some AR applications.

In this paper, we proposed the Digital Diorama system to convey background information vividly. The system superimposes computer generated diorama scene reconstructed from related image/video materials on real exhibits. In order to switch and superimpose real exhibits and past photos seamlessly, we implement a matching system for estimating the camera position where photos are taken. By applying this subsystem to 26 past photos about the steam locomotive exhibit, we succeeded in estimating their camera position. Thus, we implement and install a prototype system at estimated position to superimposing virtual scene and real exhibit in the Railway Museum.

We have developed a novel automultiscopic display, CAGRA (Computer-Aided Graphical Real-world Avatar), which can provide full parallax both horizontally and vertically without imposing any other additional equipment such as goggles on users. CAGRA adopts two axes of rotation so that it can distribute reflected light all around. In our previous work, it was proved that the display can present a black and white image of a 3D object at 1 Hz with both horizontal and vertical parallax. Here, we discuss two important problems that remain unsolved in our previous work: optimal parameter setting and the rendering method. As for finding optimal parameters, the relationship among the rotation speed of two axes and the diffusion angle of the holographic diffuser is discussed. The rendering method was implemented so that accurate view is presented in spite of the unique mechanical structure of CAGRA. Furthermore, the synchronization mechanism between projection of images and rotation of the mirror, which was also a remaining problem, is also implemented. Findings obtained through solving these problems clarify the characteristics of this novel display system, CAGRA, and lead to its further development.

Olfactory displays which exist now can only present the set of scents which was prepared beforehand because a set of "primary odors" has not been found. In this paper, we focus on development of an olfactory display using cross modality which can represent more patterns of scents than the patterns of scents prepared. Due to cross modal effect between vision and olfaction, human tends to feel not olfactory but visual stimulation as scents. First, we asked subjects to smell various aroma chemicals and evaluate their similarity. Based on the data of similarity among aromas, we built a map of smell distance. Next, we selected a few aroma chemicals from the smell distance map and implemented a visual and olfactory display. Then we conducted an assessment experiment of the display. We presented various pictures and the selected aroma chemicals to subjects and asked them what kind of scent they smelled like. In this experiment, we succeed in giving subjects feeling more patterns of scents than the number of selected aroma chemicals. Also, we succeed in making not olfactory stimulation by aroma chemicals but visual stimulation by pictures as scents. In particular, we find that the visual effect on olfactory sensation is more strong when the distance between the picture and the aroma chemicals is close than when the distance is far.

Many interactive user interfaces require the location and orientation of the user. Recently massively distributed RFID tags are being used in location aware system by attaching RFID reader to mobile robot or a person.[Bohn and Matter 2004] [Scooter and Helal 2004] These applications use RFID tags in position detection methods, in which physical coordinates are correlated with RFIDs by using a mapping table. Although such application of RFID technology has high potential, implementing such a system in physical space using conventional methods is expensive and time consuming. To resolve these problems, we have developed an RFID textile which is able to be woven commercially using a conventional loom.

Controllable particle display has been proposed, that controls the position and blinking patterns to yield a visual representation and the representation to touch these particles. Additionally, as an example of its implementation, a controllable particle display using water drops as the particles was proposed. In this system, objects are represented by projecting images upward onto falling water drops designed to form a plane surface, depending on the positions of the water drops. However, this method has a problem in terms of the brightness of the object. In this paper, we propose a method by which images are projected onto falling water drops at an angle, and users observe the images from in front of a projector.

Several indoor positioning systems have been studied to offer a public service based on one's location information. We have studied an indoor localization system using original fiducial markers. The original markers can be designed freely by users; therefore, they can be used for interior decoration. Using this system, you can obtain the three-dimensional position and pose of your camera by capturing the image of the markers arranged on the floor. However, there are some problems associated with the use of this system in a public space, such as a decline in the marker recognition rate by a change in the surrounding light condition and an instability in the marker recognition rate depending on the type of marker used and the camera angle. In this study, we enhanced the robustness of an indoor localization system used in a public space and increased the number of recognizable markers.

Wearable Forest is a clothing design that bio-acoustically interacts with distant wildlife in a remote forest through a remote-controlled speaker and microphone using a network. It expresses the bioacoustical beauty of nature in its unique aesthetic appeal for users and allows the users interact with a forest in real time through a network to acoustically experience a distant forest soundscape, thus merging man and nature without environmental destruction. This novel interactive sound system can create a sense of unity between users and a remote soundscape, enabling users to feel a sense of belonging to nature even in the midst of a city.

We present a novel approach for constructing a virtual 3D world from a sparse set of 2D photograph images. In our approach, we do not construct a large 3D world directly from the images, instead we construct several 3D models from the images and inter-connect them. Each 3D model is photograph-based 3D model that constitutes a few faces and a high-resolution photographic texture image. By arranging some photograph-based 3D models in a 3D scene, we construct a virtual 3D world. To inter-connect these 3D models and represent a unified 3D world, we render the 3D models by blending them together according to view-point position and rotation. Using our novel approach, it is possible to semi-automatically construct a virtual 3D world from fewer photograph images.

In this paper, the technology of non-invasive brain-computer interfacing (BCI) in immersing virtual environment was focused for navigation with no subject training. The common spatial patterns were for the first time applied to EEG-based BCI system in CAVE with five-screen configuration. The interfacing system was based on two brain states of left and right hand phantom movements. It was suggested that the navigation could be demonstrated with no subject training in immersing virtual environment.

Because marker-based position tracking system is inexpensive and easy to use, it has the potential to make the system more feasible for homes or businesses and broaden the current suite of augmented reality (AR) techniques. To apply marker-based systems to homes or businesses, blending markers with environments naturally is important. Our innovative approach to marker-based 3D position tracking uses seamless patterns encrypted with positional data. Although users can obtain 3D positional data by processing marker images similar to many existing marker-based systems, our markers are designed with interior decoration in mind. That way they can be installed in walls, floors, or ceilings. Unlike existing systems whose fiducial markers were designed first and foremost to be processed by computers, ours are visually attractive. By integrating positional information within the interior design, our system enables users to enjoy the benefits of position tracking without being constantly aware of the system's presence. We developed a method for making patterns in which positional information is encrypted and a method for calculating the 3D position of a user by decoding those patterns. We then constructed a system using three patterns that were made by the proposed method and evaluated that system

Recent development in mobile and ubiquitous computing technology with broadband wireless network has realizing the location based application for mobile and wearable computers. According to the wide spreading number of cellular phone with GPS receiver, it has become easy to provide location based application in outdoor scene. On the contrary, there are several positioning technologies for indoor scene, such using wireless sensors or image processing technology. However, those technologies are still not utilized in daily environment due to the cost inefficiency and the complexity in system setup. Considering the problems in previous positioning system, it is necessary to overcome the following four features to utilize the system in daily scene.

A relatively new form of Augmented Reality (AR), projector based AR using stationary projectors, makes use of rotatable projection devices to augment objects. However, the range of such systems is limited. In order to exceed these limits, we introduced the ability to switch the projection of the augmentation between several projection systems. This way the system with the best position in relation to the object to be augmented can be selected to do the projection, and thus the quality of the augmentation can be optimized. A crucial part of this process is the estimation of the achievable projection quality of all projection systems, which may also be dependent on the type of application.

In this paper, we describe a novel voice menu presentation method, the vCocktail, designed for efficient human-computer interaction in wearable computing. This method is devised to reduce the length of the serial presentation of voice menus by introducing spatiotemporal multiplexed voices with enhanced separation cues. Perception error in judging voice direction was measured first to determine appropriate directions in which and interval angles at which menu items were placed allowing the user a clear distinction among the multiple items. Then voice menu items were presented under spatiotemporally multiplexed conditions with several different settings of spatial localization, number of words, and onset interval. The results of the experiments showed that the subjects could hear items very accurately with localization cues and appropriate onset intervals. In addition, the proposed attenuating menu voice and crosstype spatial sequence of presentation increased the correct answer ratio effectively improving distinction between menu items. A correct answer ratio of 99.7 % was achieved in the case of four-item multiplexing when an attenuating voice and a 0.2 sec onset interval were used with the cross-type spatial sequence.

The electromyogram (EMG) signal, which is a kind of biomedical information, might be promising as a computer-human interface. The purpose in this paper is to make a brief report on implementation of the EMG interface in CAVE-clone display and propose the advantages of it. In general, it is expected that the environmental electromagnetic noise would disturb the novel EMG signals. We measured the EMG signals and obtained high signal-to-noise ratio in CABIN immersive multiscreen display constructed at the University of Tokyo. The observed signal-tonoise ratio would make EMG signals a reliable input channel with simple signal processing, and real time control of virtual objects was successfully performed. The advantages of the EMG signals will be proposed as a promising human interface in immersive multiscreen environments. The user would control virtual objects reflecting the activations of the user’s muscles to realize fine operations. Furthermore, by using this additional biosignal channel, the user might interact with virtual objects without definite motion.

Recently, there are various types of display systems that can present aural, visual and haptic information related to the user’s position. It is also important to present olfactory information related to the user’s position, and we focus on the spatiality of odor, which is one of its characteristics. In this research, we constructed and evaluated a wearable olfactory display to present the spatiality of odor in an outdoor environment. The prototype wearable olfactory display system treats odor in the gaseous state, and the odor air is conveyed to the user’s nose through tubes. Using this system, we also present the spatiality of odor by controlling the odor strength according to the positions of the user and the odor source. With this prototype system, the user can specify the position of the odor source in an outdoor environment. To improve this prototype system, we constructed another wearable olfactory display. Because odor is treated in the gaseous state, the first prototype system has some problems such as the large size of the device and unintentional leakage of the odor into the environment. To solve these issues, we developed and evaluated an advanced wearable olfactory display that uses an inkjet head device to treat odor in the liquid state.

Augmenting or annotating physical objects using steerable video projector and camera combinations works well in the range of the projection system. However, since these systems are usually mounted in fixed locations, users have to stay within a certain radius around that location. By using several of these AR-Projection systems, we can make the application roam these systems and thus follow the user beyond these limits. Using this method, we furthermore remove the need to hold the augmented objects in a suitable direction towards the projector and can reduce the possibility of shadowing the projection by the user. We describe an architecture based on our 'Projected Applications' which coordinates the systems used to project the augmentations and keeps the projected applications' states in sync between the projection systems.

This paper describes a wearable electronic mnemonics, the iFlashBack system that promotes human memorization process for the action the user performed in the real-world. The goal of the system is to exploit unlimited capacity of the human brain for storing the variety of information, by assisting memorization using recorded video and related information captured during interaction process to real-world objects. Memorization reinforcement is achieved by drawing attention, promoting rehearsal by flashing a video of interaction, and organizing memory by providing relevant information. The action of the user is captured by a cap-mounted miniature camera, an RFID reader, and arm posture sensors worn by the user. For video-aided rehearsal, a small HMD presents the captured scenes immediately after the user's interaction behavior and/or at appropriate times in the user's activity. We conducted basic experiments to demonstrate our idea. Two memory tasks - to change cup positions within grids and to memorize random dot images - were performed by subjects. The results suggested that the iFlashBack system could reinforce memorization of the user's action by motion-controlled video aided rehearsal.

In this paper, we propose a new way of augmenting our environment with information without making the user carry any devices. We propose the use of video projection to display the augmentation on the objects directly. We use a projector that can be rotated and in other ways controlled remotely by a computer, to follow objects carrying a marker. The main contribution of this paper is a system that keeps the augmentation displayed in the correct place while the object or the projector moves. We describe the hardware and software design of our system, the way certain functions such as following the marker or keeping it in focus are implemented and how to calibrate the multitude of parameters of all the subsystems.

This paper describes the methodology and prototype for an all-around display system for video avatar presentation in the real world. This system enables us to reconstruct a video avatar which users can look at from all around, by spinning a flat panel display which has a small viewing angle and changing the image on the display panel depending on the display's orientation.

We propose an approach to object manipulation. In the approach, the user's hand or fingers are represented by a set of interface points, the interaction force on each interface point is computed by the collision response computation algorithm, and the motion of the object is simulated based on the interaction force. We employed a fast collision response computation method to accommodate a large number of interaction points. Also, we devised a method for stable computation of object motion. We carried out experiments on interaction with models of varying complexity and demonstrated the feasibility of our approach for dexterous manipulations.

In this study, a remote presentation system that can be used in the shared immersive virtual world was developed This system consists of several elemental technologies, such as the video avatar, a shared database, shared visualization tools and an i-mode interface, and can be used for both meeting-style and demonstration-style presentations. This system was applied to a presentation of collaborative research in the field of fluid dynamics analysis, and the effectiveness was evaluated.

In this paper, we propose a new approach to building a virtual environment by using raw data from a real environment. To manage and analyze different types of datasets from many sensors and databases, we build and integrate different types of environments with different types of datasets, user interactions and rendering methods. To integrate the environments, we merge rendered images of each virtual environment according to the user's viewpoint, depth information, and reliability of rendered images, and display the resultant image to the user. In addition, we propose an integration environment by using an environment manager, which determines the user's viewpoint with respect to the integrated environments according to the user's input, and compose a display image from rendered images of the integrant environments. The reliability of the rendered images is determined by data accuracy and resolution, display resolution and rendering errors from rendering methodologies. By integrating four constructed environments according to the user's viewpoint, we can construct a photorealistic large-scale environment from different types of datasets. Furthermore, we implemented the prototype system in CABIN and demonstrated the scalability of this architecture.

We discuss application possibilities of virtual reality technologies such as immersive projection technology, to the field of genome science, and the guiding principles for the visualization of genome information are proposed. In addition, we develop a visualization methodology by using immersive projection technology for pair-wise comparison between cluster sets generated from different gene expression datasets. Our methodology could display the distribution of overlaps between two hierarchical cluster sets based on hepatocellular carcinomas and hepatoblastomas.

When a 3D object is designed in a virtual world, it is essential that the designer can observe and manipulate the object directly in 3D space. The aim of our study is to develop a virtual workplace in which 3D virtual objects can be observed from various directions and can be touched directly by utilising force feedback. In this paper, we propose a 3D modelling system that consists of an inclined three-screen display and a force feedback device. Using this system, we propose a force feedback grid that extends the concept of 2D CAD manipulation to a grid that can be used in a 3D space, and which integrates with force feedback in order to support modelling in the 3D space. Finally, we report the results of an experiment that we conducted in order to verify the effectiveness of the force feedback grid.

Describes the design and implementation of an artificial force display for immersive projection displays (IPD) such as CAVE or CABIN. In order to give the user maximum freedom of motion in the large virtual space generated by an IPD, it is necessary to use a portable force display that is self-contained on the user's body. Therefore, we developed a wearable force display called HapticGEAR. It is a backpack-type device that transmits applied forces to the wearer by using a wire-tension mechanism. The device is designed such that it minimizes fatigue for the wearer and does not interrupt the user's motion and view. Also, we integrated the device into CABIN and experimentally evaluated both the accuracy it achieved in tracing objects and the displayed haptic sensation.

In this paper we discuss application possibilities of virtual reality technology such as immersive projection technology to the field of genome science. The prototype of the visualization environment is implemented and used in analysis to elucidate the important genes in categorizing liver cancer.

Immersive projection displays such as CABIN and COSMOS have been connected through a broadband network. This kind of network environment is expected to be used as a multimedia virtual laboratory. In particular, video avatar technology has been developed in order to realize high-presence communication in this multimedia virtual laboratory. A video avatar is a computer-synthesized 3D image created using live video. This method has the characteristics of being a natural, accurate and convenient communication tool. In this study, the communication capabilities of the video avatar were experimentally evaluated. In addition, the video avatar technology was applied to several communications applications, such as that of guiding colleagues and undertaking design work in networked immersive projection displays.

A worldwide photorealistic environment includes huge amounts of datasets of different types and scales. It is almost impossible to describe such an environment using a single methodology. To manage different types and scales of these datasets, we build and handle virtual subenvironments of different types and scales separately. In addition, we propose an image-based combination method to switch between these local environments smoothly according to the user's viewpoint. We implemented this prototype system in ImmersivProjection Technology (IPT) and demonstrated the efficiency of the method.

The immersive projection display, CABIN was connected to several similar displays via an ATM network. In order to share the virtual world with a high quality of presence, transmitting the user's figure is a key technology. In this study, a 2.5 D video avatar method was developed. The user's figure is captured by stereo cameras and it is superimposed on the shared virtual world as the 2.5 D video avatar. In addition, the pointing accuracy using the 2.5 D video avatar was experimentally evaluated.

We describe the design and the implementation of a force display for immersive projection displays. To give the user maximum freedom of motion in a large working space, it is necessary to use a portable force display that is grounded on the user's body. Therefore, we developed a portable (wearable) force display called HapticGEAR which makes use of the tension of wires grounded on the user's back. This device is designed to reduce user's fatigue and to have fewer influence on the user's motion and sight.

This paper describes the work which has been conducted to further the development of an effective motion capture system toed on data fusion and the minimal use of sensing devices. Application of the inverse kinematics method is first described along with its shortcoming and limitations. Next, in order to address the problems associated with this method, the Springs method and the concept of Discomfort Potential are introduced. The spring method is based on the idea that the stiffness of a spring located at a articulation may be used to avoid dislocations in the virtual character. The concept of discomfort potential governs the range of motion of specific articulations such as elbows and knees, thereby allowing motion to be effectively monitored and smoothly represented using less sensors or lack of data. Experimental results are also discussed.

Two types of depth cues contributed to the spatial perception of objects in scene: absolute cues, which theoretically provide information about the exact location of a point in space, and relative cues, which give depth information with respect to other points in space. The viewing parameters, such as the viewing distance or frustrum angle, can change the effect of these depth cues. The influence of disparity, object rotation, motion parallax, and the frustrum angle on a simulated telemanipulation task is examined. The results show that disparity is a powerful cue that can improve task performance. Object rotation and head motion parallax neither enhanced nor degraded performance. The frustrum angle affected the completion time but not error performance.>