Users are often seated in the real environment, while their virtual avatars either remain standing stationary or move in virtual reality (VR). This creates posture inconsistencies between the real and virtual embodiment representations. The relationship between posture consistency in locomotion techniques and sense of presence in VR is still unclear. This study investigates how visual and somatosensory integration affects the sense of standing (SoSt) and the sense of self-motion (SoSm) when the sitting posture is varied slightly, including highlighting the importance of sitting posture for locomotion design in VR. The degree and occurrence of SoSt and SoSm were assessed by subjective experiments, and it was found that higher sitting and lower sitting postures present higher SoSt and lower SoSm, respectively. Invocation of SoSt also influences postural perception. Perception of travel distance varied according to the posture condition when identical visual flow was presented. The findings suggest that visual and somatosensory integration related to posture enhances SoSt and SoSm, and a sitting posture with a higher seating position is recommended in seated VR locomotion design.

In underwater environments, the reflected ultrasonic waves from objects generally provide more than just information about their color and shape for object recognition. Previous studies have overlooked the influence of object pose on these wave components. It is crucial to investigate how these poses affect the reflected wave components because object poses can vary widely and are often unpredictable in real-world scenarios. In this work, we introduce a novel dataset comprising reflected wave components collected from objects made of various materials and observed from various angles. We also show the preliminary evaluations on the performance of machine learning-based material classification on object pose. Our results indicate that the accuracy is consistently high (≥ 91%) for known angles but significantly drops (< 60%) when dealing with unknown angles in most cases. Based on these evaluations, we suggest several directions for future research. Our dataset is available at https://github.com/Nyamotaro/U2R.

Entering a small space such as an elevator or a crowded train with a stranger can cause discomfort and suffocation. This is because the stranger is invading the individual’s personal space. However, it is difficult to maintain an appropriate interpersonal distance from others at all times in various situations. Therefore, a noise-canceling HMD [2][3] that uses AR to change the size of the person in the field of vision has been proposed as a means of reducing noise such as discomfort caused by inappropriate interpersonal distance. In this paper, we propose an improvement method using generative AI for background completion in noise-canceling HMDs.

The sense of reality is enhanced by presenting appropriate haptic feedback when the user interacts with a virtual object in virtual reality (VR). To present appropriate feedback, we often use a real object that resembles the virtual one to manipulate. However, such a real object is not always available. The user may feel a visuohaptic inconsistency between real and virtual objects when their shapes are different. To alleviate such an inconsistency, we propose a novel object synthesis method that combines the shape of the real object that the user manipulates in reality and the shape of the virtual object which was to be presented to the user in VR. In other words, this synthesized object, a chimera object, is created by transforming the part of the virtual object that the user would touch into a shape that is similar to the corresponding part of the real object while maintaining the other parts of the virtual object intact. The expected haptic sensation from the appearance of our chimera object is more consistent with the one produced by the real object. Therefore, the visuo-haptic inconsistency is expected to alleviate in VR, compared to the original virtual object. In addition, we propose an interactive system to support the design of a chimera object for users. To investigate the effectiveness of our method, we conducted two user studies. The first experiment confirmed that our proposed chimera object helps enhance visuo-haptic consistency. The second experiment confirmed that our system was effective for chimera object creation by users with acceptable system usability.

Common existing head-mounted displays (HMDs) for virtual reality (VR) provide users with a high presence and embodiment. However, the field of view (FoV) of a typical HMD for VR is about 90 to 110 [deg] in the diagonal direction and about 70 to 90 [deg] in the vertical direction, which is narrower than that of humans. Specifically, the downward FoV of conventional HMDs is too narrow to present the user avatar's body and feet. To address this problem, we have developed a novel HMD with a pair of additional display units to increase the downward FoV by approximately 60 ( 10+50) [deg]. We comprehensively investigated the effects of the increased downward FoV on the sense of immersion that includes presence, sense of self-location (SoSL), sense of agency (SoA), and sense of body ownership (SoBO) during VR experience and on patterns of head movements and cybersickness as its secondary effects. As a result, it was clarified that the HMD with an increased FoV improved presence and SoSL. Also, it was confirmed that the user could see the object below with a head movement pattern close to the real behavior, and did not suffer from cybersickness. Moreover, the effect of the increased downward FoV on SoBO and SoA was limited since it was easier to perceive the misalignment between the real and virtual bodies.

Augmented reality (AR) has been widely used that combines human with the digital world tightly at an unprecedented level, and various types of optical see-through head-mounted displays (OSTHMDs) have been actively developed to meet the requirement of everyday AR use. Correct mutual occlusion between real and virtual objects is often necessary for displaying realistic virtual images for users in the AR application scenarios. Some optical designs have been proposed to realize mutual occlusion by means of an OSTHMD in recent years. However, all of them support a limited field of view (FOV) that is much narrower than that of a natural human. The main limiting factor of the FOV of general OSTHMDs is the limited numerical aperture (NA) of the lenses. To address the problem, we propose an OSTHMD based on the double ellipsoidal mirror structure to avoid a stack of lenses and to achieve a wide FOV close to that of the naked eye with small distortion. A pair of imaging lenses are carefully arranged, then assembled between the two ellipsoidal mirrors with a pinhole mask to improve image quality. An experiment using a monocular prototype shows that a sharp see through view is achieved which remains clear regardless of the focus of the eye-simulating camera. An image undistortion algorithm is also developed to obtain a full-view display, enabling a virtual image to be displayed spanning a super wide FOV of H160°×V74°. Finally, per-pixel mutual occlusion of a wide FOV of H122°×V74° is realized by placing a spatial light modulator (SLM) in front of the entrance pupil.

In recent years, human augmentation has attracted much attention. One type of human augmentation, motion augmentation makes perceived motion larger than in reality, and it can be used for a variety of applications such as rehabilitation of motor functions of stroke patients and a more realistic experience in virtual reality (VR) such as redirected walking (RDW). However, as augmented motion becomes larger than the real motion, a variety of senses that accompany will be more inconsistent with those perceived from somatic sensations, which will cause a severe sense of discomfort. To address the problem, we focus on kinesthetic illusions that are psychological phenomena where a person feels as if his or her own body is moving. Kinesthetic illusions are expected to fill the gap between the intended augmented motion and perceived physical motion. However, it has not been explored if and how large kinesthetic illusions are produced while a user is moving their limbs voluntarily in VR. To expand the knowledge on kinesthetic illusions, we have conducted two user studies on the impact of tendon vibration and visual stimuli on kinesthetic illusions. First experiment confirmed that the perceived elbow angle becomes larger than the actual angle when presented with tendon vibration. Second experiment revealed that the increase of the perceived elbow angle was about 20 degrees when both tendon vibration and visual stimuli were presented whereas it was about 10 degrees when only visual stimuli were presented. Through these experiments, it has been confirmed that combining tendon vibration and visual stimulation enhances kinesthetic illusions.

We have been studying augmented reality (AR)-based gustatory manipulation interfaces and previously proposed a gustatory manipulation interface using generative adversarial network (GAN)-based real time image-to-image translation. Unlike three-dimensional (3D) food model-based systems that only change the color or texture pattern of a particular type of food in an inflexible manner, our GAN-based system changes the appearance of food into multiple types of food in real time flexibly, dynamically, and interactively. In the present paper, we first describe in detail a user study on a vision-induced gustatory manipulation system using a 3D food model and report its successful experimental results. We then summarize identified problems of the 3D model-based system and describe implementation details of the GAN-based system. We finally report in detail the main user study in which we investigated the impact of the GAN-based system on gustatory sensations and food recognition when somen noodles were turned into ramen noodles or fried noodles, and steamed rice into curry and rice or fried rice. The experimental results revealed that our system successfully manipulates gustatory sensations to some extent and that the effectiveness seems to depend on the original and target types of food as well as the experience of each individual with the food.

In virtual reality (VR) systems, a user avatar is typically manipulated based on actual body motion in order to present natural immersive sensations resulted from proprioceptors. It is useful to be able to feel a large body motion in a virtual environment while it is actually smaller in the real environment. As a method to achieve this, we have been working on kinesthetic illusions induced by tendon vibration. In this article, we report on a user study focusing on the effects of a combination of visual stimulus and tendon vibration. In the user study, we measured subjects' perceived elbow angles while they observed the corresponding virtual arm bending to 90 degrees, with different rotation amplification ratios between 1.0 and 1.5, with and without tendon vibration on the wrist joint. The results show that the perceived angle is increased by up to 8 degrees when tendon vibration is applied, roughly in proportion to the rotation amplification ratio. Our study suggests that the vision-induced kinesthetic illusion is further increased by tendon vibration, and that our technique can be applied to VR applications to make users feel larger body motion than that in the real environment.

We propose a concentration improvement technique by using a video see-through head mounted display (HMD). Our technique reduces the visual noise or disturbing visual stimulus, such as moving objects near the central visual field, by lowering its visual saliency in real-time. Earphones and noise cancelling headphones are often used to shutdown auditory noise from surroundings when we need to concentrate on the job. Several studies have proven the effectiveness of such noise reduction on improving the concentration and learning efficiencies. We apply this analogy to the visual noise and propose a visual noise reduction HMD to improve concentration. In this article, we report on two preliminary user studies we conducted to investigate the effectiveness of our method. In the prototype system, we manually specify a part of the visual field as the working area and apply grayscale and blur filters outside it in real-time. We compared two conditions “no effect” and “strong blur” (for Exp 1) or “weak blur” (for Exp 2), using a simple math task. Our preliminary results show that visual noise reduction improves the task completion time by 10% for Exp 1 and 7.9% for Exp 2 on average.

We propose a novel gustatory manipulation interface which utilizes the cross-modal effect of vision on taste elicited with augmented reality (AR)-based real-time food appearance modulation using a generative adversarial network (GAN). Unlike existing systems which only change color or texture pattern of a particular type of food in an inflexible manner, our system changes the appearance of food into multiple types of food in real-time flexibly, dynamically and interactively in accordance with the deformation of the food that the user is actually eating by using GAN-based image-to-image translation. The experimental results reveal that our system successfully manipulates gustatory sensations to some extent and that the effectiveness depends on the original and target types of food as well as each user's food experience.

In this demonstration, we present a novel gustatory manipulation interface which utilizes the cross-modal effect of vision on taste elicited with real-time food appearance modulation using a generative adversarial network (GAN). Unlike existing systems which only change color or texture pattern of a particular type of food in an inflexible manner, our system changes the appearance of food into multiple types of food in real-time flexibly, dynamically and interactively in accordance with the deformation of the food that the user is actually eating by using GAN-based image-to-image translation. Our system can turn somen noodles into ramen noodles or fried noodles, or steamed rice into curry and rice or fried rice. Users of our demonstration system will taste what is visually presented to some extent rather than what they are actually eating.

The goal of this research is developing a system that a rich facial animation can be used in communication is generated from only speech. Generally, a source of the generating facial animation is a camera. Using cameras as an input source, it causes limitations of the angle of view of the camera or problems that cannot be aware of the human face, depending on the orientation of the face. Therefore, it is reasonable for developing a system for generating a facial animation using only voice. In this study, we generate facial expressions from only speech using LSTM-RNN. Comparing 3 patterns of speech analysis data, we showed that the proposed method using A-weighting is effective for facial expression estimation.

Typical Head-Mounted Displays (HMDs) that provide a highly immersive Virtual Reality (VR) experience make any interaction between a user and real space difficult by occluding the user's entire field of view. Video see-through type HMDs can solve this problem by superimposing real-space information on the VR environment. The existing method of supporting interactions with the real space is superimposition of boundary lines of the real space on the virtual space in the HMD. However, overlaying the boundary lines on the entire field of view may reduce the user's immersive feeling. In this paper, we propose two methods to support interactions with the real world while playing immersive VR games without reducing the user's immersive feeling as much as possible, even when the user wanders. The first method is to superimpose a 3D point cloud of real space around the user on the virtual space in the HMD. The second method is to deploy familiar objects (e.g., furniture in his/her room) in the virtual space in the HMD. The user traces the familiar objects as subgoals to reach the goal. We implement the two methods and conduct a user study to compare interaction performance. As a result of the user study, we find that the second method provides better spatial information about the real space without reducing the user's immersive feeling, compared to the existing method.

When people anonymously use social network sites (SNSs) like Twitter, they may interact with not only real world friends but also strangers, who are not acquaintances in the real world. Therefore, both of real friendship (RF) and virtual friendship (VF) coexist in these SNSs. In this research, we investigated the differences in similarity of user pairs in Japan by their types of relationship, i.e. RF or VF. The primary results indicated that RF user pairs have more common follow users on SNSs than VF user pairs, and that contents posted by VF user pairs include more similar words than RF user pairs. It is implied that two users with RF have a similar interest in neighborhood users, and two users with VF are interested in similar topics. After that, we built the models to classify user pairs into RF or VF using the similarity measures. These models showed high performance to distinguish between RF and VF (their F-measures are larger than 0.80).

A picture lifelog is a type of lifelog that consists of pictures, mainly taken by the user. Recently, users have been able to easily create picture lifelogs because many portable devices such as smart phones have a camera. When a user sees a picture in their picture lifelog, it is sometimes difficult to recall the events related to the picture. Therefore, we proposed to combine search queries on a picture lifelog in order to support memory retrieval. Search queries are input into a web search engine to satisfy a user's need for information. Recently, because of the prevalence of smart phones, the opportunity to input search queries has increased to anytime and anywhere. Search queries are stored in a cloud user database such as Google search history. In addition, those search queries imply what the user was thinking at the time. We investigated whether search queries enable a user to recall their thoughts regarding picture lifelogs. Thus, we conducted an experiment to ascertain whether search queries reminded a user of past events. As a result, we reveal that displaying a picture with search queries performed around the time it was taken tends to improve users' memories better than its time, location, or emails sent during that time.

Securing one's personal space is quite important in leading a comfortable social life. However, it is difficult to maintain an appropriate interpersonal distance all the time. Therefore, we propose an interpersonal distance control system with a video see-through system, consisting of a head-mounted display (HMD), depth sensor, and RGB camera. The proposed system controls the interpersonal distance by changing the size of the person in the HMD view. In this paper, we describe the proposed system and conduct an experiment to confirm the capability of the proposed system. Finally, we show and discuss the results of the experiment.

Video conferencing is becoming more widely used in areas other than face-to-face conversation, such as sharing real world experience with remote friends or family. In this paper we explore how adding augmented visual communication cues can improve the experience of sharing remote task space and collaborating together. We developed a prototype system that allows users to share live video view of their task space taken on a Head Mounted Display (HMD) or Handheld Display (HHD), and communicate through not only voice but also using augmented pointer or annotations drawn on the shared view. To explore the effect of having such an interface for remote collaboration, we conducted a user study comparing three video-conferencing conditions with different combination of communication cues: (1) voice only, (2) voice + pointer, and (3) voice + annotation. The participants used our remote collaboration system to share a parallel experience of puzzle solving in the user study, and we found that adding augmented visual cues significantly improved the sense of being together. The pointer was the most preferred additional cue by users for parallel experience, and there were different states of the users' behavior found in remote collaboration.

In this research, we explore using pointing and drawing in a remote collaboration system. Our application allows a local user with a tablet to communicate with a remote expert on a desktop computer. We compared performance in four conditions: (1) Pointers on Still Image, (2) Pointers on Live Video, (3) Annotation on Still Image, and (4) Annotation on Live Video. We found that using drawing annotations would require fewer inputs on an expert side, and would require less cognitive load on the local worker side. In a follow-on study we compared the conditions (2) and (4) using a more complicated task. We found that pointing input requires good verbal communication to be effective and that drawing annotations need to be erased after completing each step of a task.

In this research, we explore how different types of augmented gesture communication cues can be used under different view sharing techniques in a remote collaboration system. In a pilot study, we compared four conditions: (1) Pointers on Still Image, (2) Pointers on Live Video, (3) Annotation on Still Image, and (4) Annotation on Live Video. Through this study, we found three results. First, users collaborate more efficiently using annotation cues than pointer cues for communicating object position and orientation information. Second, live video becomes more important when quick feedback is needed. Third, the type of gesture cue has more influence on performance and user preference than the type of view sharing method.

This paper describes an AR system for remote collaboration using a captured 3D model of the local user's scene. In the system a remote user can manipulate the scene independently of the view of the local user and add AR annotations that appear projected into the real world. Results from a pilot study and the design of a further full study are presented.

With the miniaturization of mobile projectors, we can provide a larger projection surface for information browsing, despite the small size of the projection devices. Moreover, the opportunities for using location-based information services both indoors and outdoors increase. We can obtain the necessary information via the small LCDs of handheld devices, thanks to the prevalence of cell phones and GPS technology. However, these mobile terminal devices restrict the use of one hand and demand that the user keeps a close watch on the small display. It is necessary to take the devices out of a pocket or bag. To solve these problems, many researchers have focused on wearable projection systems. These systems allow for the provision of hands-free viewing via large projected screens and eliminate the need to take out and hold devices. In this paper, we propose a “Toe Input” method, which can realize haptic interaction, direct manipulation, and floor projection in wearable projection systems with a slightly larger projection surface. We evaluate the system in terms of accuracy, required time, comfort, and area suitable for input.

Recently, “discussion” has been adopted in elementary school education. In this activity, the active participation of children is important. It is necessary to be conscious of the interaction with others for active participation in discussion. The goal of this study is to improve the attitude towards participation in discussion of children. The authors of this study will consider a method of making children conscious of their interaction with others by visualizing the entrainment of biological rhythm. A system based on the above-mentioned consideration was developed, and an experiment that uses this system was conducted. The results show that the system has a high potential of improving children's attitude towards listening. Based on these results, developmental experiments were performed.

Under a remote collaboration demanding multiple fieldworkers, using a head-set interface composed from HMD and HMC in a situation of that only one wearable interface is allowed using in a field, there might be discrepancies in the transmission of instructions among the field workers and a remote expert because a worker without the interface of remote collaboration have to receive sidelong instructions from the remote expert via another worker with the interface. We expect that a real-world projection device, such as WACL, absorbs the discrepancies because the projected information can be watched by the people around the projection device. Therefore, we conduct a user study comparing the head-set interface and WACL interface to measure burdens of transmission of instructions under a remote collaboration demanding multiple field workers. The result shows that HMD is a display for single-user imposes more burdens on interface-wearer when forwarding remote instructions to non-interface-worker. On the other hands, a real world projection interface can reduce the burdens. Also, we found that direct instructions with a laser spot of WACL and sidelong instructions by the interface-wearer's hand via HMD has same task performance in a simple pick-up-and-put task. Furthermore, by the effect of view controllability and information-browsability of WACL, non interface-workers can receive instructions from the remote expert directly, and carry out the instructions even when the interface-wearer is working.

We have developed a GUI tool and a mobile MR system for reliving experiences at museums. The GUI tool was developed to provide services based on content made from activity logs and to enable effective analysis of visitors' behaviors using the logs. In addition, by running the GUI tool on a mobile MR system, users can browse visitors' activities with a sensation of realism on the spot. This paper describes the GUI tool and a pilot user study that was conducted to evaluate it.