Cross-modal correspondence, exemplified by the Bouba/Kiki effect, which refers to the association between linguistic sounds and visual shape impressions, is known to influence taste perception. Previous studies have shown that taste perception varies with the visual features of foods, beverages, and tableware. However, it remains unclear how embodying a self-avatar in virtual reality (VR) whose visual features are associated with Bouba/Kiki influences taste perception. We conducted two studies to investigate the effects of cross-modal correspondence between VR self-avatar characteristics and taste perception using lemon-flavored carbonated beverages that elicit both sweetness and sourness. Study 1 directly compared Bouba avatars (round shapes, red colors, slow animations) and Kiki avatars (angular shapes, yellow colors, fast animations) in a within-participants design ($N = 36$N=36). Study 2 examined the effects of each avatar type separately by comparing them with neutral control conditions ($N = 72$N=72). In Study 1, the Bouba avatar led participants to perceive the beverage as mellower compared to the Kiki avatar. However, this effect was not significant when compared with a neutral control avatar in Study 2. In contrast, the Kiki avatar produced robust effects, increasing perceived sourness and sharpness in both studies. These findings suggest that the observed changes in taste are primarily driven by the congruence between avatar characteristics and the intrinsic sensory properties of the beverage. This study demonstrates the potential for avatar-mediated congruence effects to alter perceived taste in VR, highlighting the importance of the alignment between the user's virtual body and the consumed stimulus.

The mirroring effect has been demonstrated to facilitate smooth social interactions. Digital technology can automatically implement mirroring to achieve more natural interactions in contexts where spontaneous mirroring is challenging, such as remote human communication or dialogue with non-human agents. However, many existing automatic mirroring systems replicate a user’s movements with a time delay, and when applied to full-body interactions between human users, such systems often result in excessive and unnatural mimicry. To address this issue, this study examines the effectiveness of Co-embodiment, a technique that blends the movements of a user and their partner in real-time, as a method for achieving both the benefits of mimicry and the naturalness of interaction. We examined how varying the blending ratios (0%, 25%, 50%) affects social impressions in a two-person interaction in the VE. Results showed that moderate blending (25%, 50%) enabled natural and comfortable interactions, comparable to no blending (0%). However, we did not observe significant improvements in perceived closeness or trustworthiness. Semi-structured interviews helped explain these results, revealing that participants differed in their interpretations of subtle synchronized movements. Some viewed them as signs of mutual understanding or cooperation, while others experienced discomfort. In some cases, this discomfort seemed specific to virtual contexts, as subtle similarities made participants doubt the presence of a real human behind the avatar. These findings highlight that impressions depend on whether blended movements feel socially responsive or merely imitative, stressing the need for interaction designs that enhance the sense of social presence.

In the real world, the body typically remains within the downward field of view (FOV) irrespective of whether it is consciously or unconsciously observed. However, with head-mounted displays (HMDs) that have restricted viewing angles, users are unable to see their own avatars directly unless they tilt their heads significantly downward. To overcome this, virtual mirrors are commonly employed in virtual environments (VEs) to provide users with a full-body view of their avatars, which ultimately enhances the viewer's sense of embodiment (SoE). However, positioning users directly in front of a virtual mirror may restrict their movement and reduce their sense of presence. To overcome this limitation, we developed an HMD with an extended downward field of view (FOV) integrated with eye-tracking. We compared the impact of using virtual mirrors alone versus the extended downward FOV in terms of several parameters, including the head pitch angle, perceived SoE, presence, and task difficulty during a reaching task that involves both hand and foot movements. The results demonstrated that the extended downward FOV offered by the proposed HMD improved the users' sense of agency toward their avatars and enhanced their sense of presence compared to when only virtual mirrors are used. In addition, the participants tended to use the virtual mirror for hand movements and relied on the extended downward FOV for foot movements, often maintaining a stable head angle. Both the virtual mirror and extended downward FOV facilitated easier and faster completion of a reaching task while encouraging an upward head angle. However, the combined use of virtual mirrors and extended downward FOV did not have a significant impact on the SoE or presence. Notably, the extended downward FOV alone, without the integration of the virtual mirror, was more effective in increasing both SoE and presence.

Various studies have suggested that altering the appearance of food can impact multisensory flavor perception. The cross-modal effect of such visual changes on gustation may allow for the presentation of food tastes that are difficult to express with simple combinations of taste stimuli. This cross-modal effect of visual changes on gustation holds potential for applications in gustatory displays. However, the current limitation of existing Head-Mounted Displays (HMDs) is their restricted vertical Field of View (FoV), which prohibits the display of images near the mouth while eating. This limitation may impede the cross-modal effect of visual changes on multisensory flavor perception. Additionally, the lack of visibility around the mouth area challenges the ease of eating. To address these issues, we design a Video See-Through (VST)-HMD with an expanded vertical FoV (approx. 100 [deg]). Using the HMD, we investigated how presenting visual information near the mouth affects the cross-modal effects of flavor perception and ease of eating. In our experiment, machine learning techniques were utilized to alter the appearance of food. However, the result showed no significant differences in the amount of cross-modal effects or the ease of eating between the groups with and without visual information near the mouth. As a discussion of this result, the participants may not direct their visual attention to the food when they put the food in their mouths. The experiment also examined whether visual changes alter the taste as well as the smell and texture of the food. The findings demonstrated that visual changes could present the smell and texture of the food following the modifications. This result was confirmed irrespective of the visibility near the mouth.

Common existing head-mounted displays (HMDs) for virtual reality (VR) provide users with a high presence and embodiment. However, the field of view (FoV) of a typical HMD for VR is about 90 to 110 [deg] in the diagonal direction and about 70 to 90 [deg] in the vertical direction, which is narrower than that of humans. Specifically, the downward FoV of conventional HMDs is too narrow to present the user avatar's body and feet. To address this problem, we have developed a novel HMD with a pair of additional display units to increase the downward FoV by approximately 60 ( 10+50) [deg]. We comprehensively investigated the effects of the increased downward FoV on the sense of immersion that includes presence, sense of self-location (SoSL), sense of agency (SoA), and sense of body ownership (SoBO) during VR experience and on patterns of head movements and cybersickness as its secondary effects. As a result, it was clarified that the HMD with an increased FoV improved presence and SoSL. Also, it was confirmed that the user could see the object below with a head movement pattern close to the real behavior, and did not suffer from cybersickness. Moreover, the effect of the increased downward FoV on SoBO and SoA was limited since it was easier to perceive the misalignment between the real and virtual bodies.

We have been studying augmented reality (AR)-based gustatory manipulation interfaces and previously proposed a gustatory manipulation interface using generative adversarial network (GAN)-based real time image-to-image translation. Unlike three-dimensional (3D) food model-based systems that only change the color or texture pattern of a particular type of food in an inflexible manner, our GAN-based system changes the appearance of food into multiple types of food in real time flexibly, dynamically, and interactively. In the present paper, we first describe in detail a user study on a vision-induced gustatory manipulation system using a 3D food model and report its successful experimental results. We then summarize identified problems of the 3D model-based system and describe implementation details of the GAN-based system. We finally report in detail the main user study in which we investigated the impact of the GAN-based system on gustatory sensations and food recognition when somen noodles were turned into ramen noodles or fried noodles, and steamed rice into curry and rice or fried rice. The experimental results revealed that our system successfully manipulates gustatory sensations to some extent and that the effectiveness seems to depend on the original and target types of food as well as the experience of each individual with the food.

We propose a novel gustatory manipulation interface which utilizes the cross-modal effect of vision on taste elicited with augmented reality (AR)-based real-time food appearance modulation using a generative adversarial network (GAN). Unlike existing systems which only change color or texture pattern of a particular type of food in an inflexible manner, our system changes the appearance of food into multiple types of food in real-time flexibly, dynamically and interactively in accordance with the deformation of the food that the user is actually eating by using GAN-based image-to-image translation. The experimental results reveal that our system successfully manipulates gustatory sensations to some extent and that the effectiveness depends on the original and target types of food as well as each user's food experience.

In this demonstration, we present a novel gustatory manipulation interface which utilizes the cross-modal effect of vision on taste elicited with real-time food appearance modulation using a generative adversarial network (GAN). Unlike existing systems which only change color or texture pattern of a particular type of food in an inflexible manner, our system changes the appearance of food into multiple types of food in real-time flexibly, dynamically and interactively in accordance with the deformation of the food that the user is actually eating by using GAN-based image-to-image translation. Our system can turn somen noodles into ramen noodles or fried noodles, or steamed rice into curry and rice or fried rice. Users of our demonstration system will taste what is visually presented to some extent rather than what they are actually eating.