Why do certain interfaces feel effortless while others feel exhausting? This paper proposes Timelines and Topographies as a conceptual framework to answer this question, reinterpreting interface design through the lens of human attention. We distinguish between timelines—a linear mode where the system serializes information to offload the cognitive cost of attentional control (e.g., social media feeds)—and topographies—a spatial mode where the system’s structural relations are directly presented to the user (e.g., folder hierarchies, canvas UIs). We apply this framework to analyze a wide spectrum of interfaces, ranging from classic desktop metaphors to modern Personal Information Management tools like Slack and Notion. Furthermore, we identify Large Language Models as bidirectional translators that bridge these modes by converting complex topographic structures into conversational timelines, and vice versa. Finally, we provide design guidelines for harnessing both modes, enabling interfaces that support fluid transitions between subjective timelines and objective topographies.

Maintaining social connections via informal gathering and communication in physical workspaces is challenging. To maintain connections without causing mental strain, we propose “Async Party,” an activity that reconfigures drinking rituals through asynchronicity by combining physical triggers with digital sharing. In this activity, participants record short videos of individual toasts, ask a nearby colleague to record their “cheers” moment, and upload the short video to the dedicated Slack channel. A three-month field deployment revealed that while the filming rule was designed to spark face-to-face interaction at the moment of recording, the encounters it produced were experienced not as procedural steps but as socially meaningful moments in their own right – a social quality that extended beyond our anticipation. Members derived a sense of community vitality not by watching every video, but by peripherally perceiving the accumulation of ritualistic content. Drawing on Research through Design, this study articulates a novel design space for "asynchronous rituals" – configurations in which synchrony and asynchrony are intentionally composed into a recurring temporal cycle – and derives three transferable design strategies for sustaining low-burden belonging in hybrid environments.

Dexterous freehand manipulation in virtual reality offers rich interaction but is limited by physical reach. Existing indirect remote manipulation techniques often sacrifice this dexterity. We address this by defining “virtually direct” manipulation, a conceptual framework for techniques that break from a purely direct or indirect model by decoupling the virtual hand from the physical one. Within this framework, we present Hitchhiking Hands (HH), a novel implementation designed to preserve the rich dexterity of direct manipulation at a distance. HH allows users to instantly switch control between multiple pre-defined virtual hands using gaze. We evaluated HH in two user studies. A first study showed our approach, which consistently maintains direct-touch properties, surpasses an established baseline in 6DoF manipulation performance and embodiment. A second qualitative study revealed HH excels in structured spaces but is ill-suited for unstructured global tasks, highlighting a trade-off between flexibility and learnability.

Digital flavour modulation represents a significant research challenge within Human-Food Interaction. Previous work has focused on modulating basic tastes, while the modulation of trigeminal sensations such as spiciness and coolness remains underexplored. Spicy and cool substances contribute substantially to the sensory appeal of food and beverages; however, their overconsumption can have adverse health effects. To enhance these trigeminal flavours without chemical additives, this study proposes an integrated multimodal approach combining electrical tongue stimulation with congruent olfactory stimuli. Unlike unimodal methods, our approach leverages the interaction between direct neural stimulation and olfactory integration to selectively modulate distinct spiciness and coolness perception. Psychophysical experiments demonstrated that our method significantly enhanced perceived coolness through combination of electrical tongue stimulation and lemon odor, and significantly enhanced perceived spiciness of a spicy solution by electrical tongue stimulation. These findings suggest that our method expands design space for digital flavour modulation and contributes to healthier and more enriched eating and drinking experiences.

In social virtual reality (VR) platforms, players can embody “non-human” avatars, which are representations whose appearance or skeletal structure diverge from typical human characteristics. This capability fosters the emergence of distinctive cultures of social interaction. This paper reports on interviews with users who employ such avatars, investigating (1) motivations for their adoption, (2) their impact on social interactions, and (3) challenges encountered when employing them in social contexts. Our findings reveal that users adopt “non-human” avatars both to escape the expectations and norms associated with the human body—thereby enabling more relaxed social communication—and to gain access to new forms of embodied experience and creative self-expression. The study also provides empirical evidence and discussion on the cultures of social interaction mediated by alternative embodiments, changes in bodily perception resulting from prolonged use, functional and social challenges related to avatar use, and the design strategies and etiquette practices developed to overcome them.

Force feedback to the hand is essential for enhancing immersion in VR and facilitating skill transfer. However, no existing method allows unconstrained hand and arm positions while keeping the fingertips free. We present a novel fingertip-free technique that induces directional force illusions on the entire hand by applying asymmetric vibrations to antagonistic dorsal and palmar tendons. Through two user studies, we demonstrate that directional accuracy is significantly above chance level under dual-point asymmetric vibration, in contrast single-point or symmetric vibration. By integrating effective haptic feedback with unconstrained hand interaction, our approach offers a promising avenue toward richer VR experiences.

Pseudo-haptics techniques are interesting alternatives for inducing haptic perceptions, achieved by manipulating haptic perception through the appropriate alteration of primarily visual feedback in response to body movements. However, the use of pseudo-haptics techniques with a motion-input system can sometimes be limited. This paper investigates a novel approach for extending the potential of pseudo-haptics techniques in virtual reality (VR), focusing on pseudoweight perception as the target case. The proposed approach utilizes a reaction force from force-input as a substitution of haptic cue for the pseudo-haptic perception. The paper introduced a manipulation method in which the vertical acceleration of the virtual hand is controlled by the extent of push-in of a force sensor. Such a force-input manipulation of a virtual body cannot only present pseudo-haptics with smaller physical spaces and be used by various users including physically handicapped people, but can present the reaction force proportional to the user's input to the user. We hypothesized that such a haptic force cue would contribute to the pseudo-haptic, here, the pseudoweight perception. Therefore, the paper endeavors to investigate the force-input pseudo-haptic perception in comparison with the motion-input pseudo-haptics. The paper compared force-input and motion-input manipulation in a point of achievable range and resolution of pseudo-weight. The experimental results suggest that forceinput manipulation successfully extends the range of perceptible pseudo-weight by 80% in comparison to the motion-input manipulation. On the other hand, it is revealed that motion-input manipulation has 1 step larger number of distinguishable weight levels and is easier to operate.

Cross-modal correspondence, exemplified by the Bouba/Kiki effect, which refers to the association between linguistic sounds and visual shape impressions, is known to influence taste perception. Previous studies have shown that taste perception varies with the visual features of foods, beverages, and tableware. However, it remains unclear how embodying a self-avatar in virtual reality (VR) whose visual features are associated with Bouba/Kiki influences taste perception. We conducted two studies to investigate the effects of cross-modal correspondence between VR self-avatar characteristics and taste perception using lemon-flavored carbonated beverages that elicit both sweetness and sourness. Study 1 directly compared Bouba avatars (round shapes, red colors, slow animations) and Kiki avatars (angular shapes, yellow colors, fast animations) in a within-participants design ($N = 36$N=36). Study 2 examined the effects of each avatar type separately by comparing them with neutral control conditions ($N = 72$N=72). In Study 1, the Bouba avatar led participants to perceive the beverage as mellower compared to the Kiki avatar. However, this effect was not significant when compared with a neutral control avatar in Study 2. In contrast, the Kiki avatar produced robust effects, increasing perceived sourness and sharpness in both studies. These findings suggest that the observed changes in taste are primarily driven by the congruence between avatar characteristics and the intrinsic sensory properties of the beverage. This study demonstrates the potential for avatar-mediated congruence effects to alter perceived taste in VR, highlighting the importance of the alignment between the user's virtual body and the consumed stimulus.

This study investigates how partner avatar design affects learning and memory when an avatar serves as a lecturer. Based on earlier research on the environmental context dependency of memory, we hypothesize that the use of diverse partner avatars results in a slower learning rate but better memory retention than that of a constant partner avatar. Accordingly, participants were tasked with memorizing Tagalog-Japanese word pairs. On the first day of the experiment, they repeatedly learned the pairs over six sessions from a partner avatar in an immersive virtual environment. One week later, on the second day of the experiment, they underwent a recall test in a real environment. We employed a between-participants design to compare the following conditions: the varied avatar condition, in which each repetition used a different avatar, and the constant avatar condition, in which the same avatar was used throughout the experiment. Results showed that participants in the varied avatar condition recalled significantly worse during the learning trials on the first day. However, we found no significant difference between conditions in the delayed recall test on the second day. We discuss these effects in relation to the social presence of the partner avatar. This study opens up a novel approach to optimizing the effectiveness of instructor avatars in immersive virtual environments.

The use of beyond-real interactions (BRI) expands the possibilities of virtual reality (VR) experiences by enabling novel capabilities, such as extending the range of manipulation or employing unusual body configurations. However, such interactions require users to adapt to novel sensorimotor mappings, which can lead to decreased performance and a poorer quality of user experience. Recently, proprioceptive suppression with noisy tendon electrical stimulation (n-TES) has been proposed to promote adaptation to BRI, though its effects have been reported to be limited. This study investigated the effects of proprioceptive suppression via n-TES on the adaptation process to BRI and the consequent changes in body schema, which have remained unexplored in prior research. Using a between-subjects design with 24 participants, we examined the effects of n-TES on a reaching task with Go-Go interaction, which extends virtual hands to enable users to reach farther than physically possible. Body schema changes were measured using a pointing task without visual feedback of the hand after adaptation to the Go-Go interaction. The results revealed that n-TES significantly altered the adaptation process by facilitating early learning and also enhanced body schema changes and aftereffects of adaptation under certain conditions. However, parameter fitting of the learning curves also suggested that n-TES might lead to a deterioration in the final level of performance achieved through adaptation. Our findings provide initial evidence that proprioceptive suppression by n-TES can contribute to facilitating sensorimotor adaptation within VR applications, while its practical utility may require adjusting stimulation to prevent potential side effects in the later stages of adaptation, specifically in trajectory smoothness.

Virtual reality (VR) technology allows users to feel a virtual body as if it was their own (i.e., body ownership illusion). Previous studies have explored how the visual realism of the user's virtual body influences the sense of body ownership in VR, focusing on the aspect of anthropomorphism. However, the effect of photorealism, another element that characterizes the visual realism of a virtual human, has not been systematically examined in the context of body ownership illusion. Therefore, we investigate the effect of the rendering style of virtual hands on the sense of body ownership, hypothesizing that the effect is affected by the rendering style of the virtual environment. In addition, we examine the effect of photorealism on presence (i.e., the sense of being there), as existing studies have offered inconsistent evidence. To this aim, we conducted a 3 × 3 mixed-design remote VR experiment (N = 117) that factored in the rendering styles of both the virtual hand and the environment and analyzed the subjective data from the questionnaire. The results suggest that neither the rendering style of virtual hands nor the consistency of the rendering style between the virtual hand and the environment influenced the sense of body ownership. Nevertheless, the more photorealistically the virtual environment was rendered, the stronger the presence. Our results indicate that different dimensions of virtual body realism affect the sense of body ownership differently, thereby highlighting the importance of applying a suitable classification of realism to understand the body ownership illusion for virtual bodies more in depth.

Pseudo-haptics alters vision to evoke haptic perception without dedicated hardware. We introduce a method that reproduces sticky-surface interactions during attaching-detaching 3D direct manipulation in VR by blending three cues—motion gain, surfacedeformation, and vibration—in various combinations. Accurately sensing how strongly a surface “clings” is vital for realistic grasping, adhesion training and material evaluation, yet it has been largely overlooked in pseudo-haptic research. Three studies evaluated these cues. The first experiment compared individual and combined cues on perceived stickiness, confirming that additional cues reliably strengthened perceived stickiness. The second experiment tested how cue number affects tolerance for visual-physical mismatch, indicating that they lowered the minimum detectable threshold though they did not widen the overall tolerated mismatch. The third experiment measured whether and how much added cues sharpen perceptual resolution, showing multiple cues improved perceptual resolution by reducing just noticeable differences by 44 % ($1.8 \times$finer), doubling discriminable levels from roughly eight with a single cue to sixteen with all cues.

In virtual environments, users can embody diverse avatars beyond physical constraints. The avatar-induced cognitive transformation (i.e. Proteus effect) can contribute to engineer self-perception, promote empathy and augment human capabilities. However, when the physicality of the avatar differs greatly from that of the user, the discrepancy between bodily sensation and prediction will inhibit embodiment. Therefore, it's challenging to evoke a high sense of embodiment in avatars that have different body structures or textures based on visual feedback alone. We propose a method to modulate movement impedance (inertia, viscosity, stiffness) of our own body in response to embodied avatars using a wearable haptic device, enhancing the sense of body ownership and the plausibility of interaction using that new avatar. Using Bayesian optimization, we identified individually optimized haptic parameters that maximize subjective plausibility for each user. Our results revealed that Bayesian optimization significantly enhanced users' perceived plausibility of the avatar via haptic feedback. The optimal parameters exhibited substantial inter-individual variation, highlighting the importance of characterizing user-specific motor sensations. On the other hand, regression analysis found no clear correlation between avatar impression ratings and optimal haptic parameters, suggesting subjective plausibility formation involves personal interpretations beyond measurable impressions.

The mirroring effect has been demonstrated to facilitate smooth social interactions. Digital technology can automatically implement mirroring to achieve more natural interactions in contexts where spontaneous mirroring is challenging, such as remote human communication or dialogue with non-human agents. However, many existing automatic mirroring systems replicate a user’s movements with a time delay, and when applied to full-body interactions between human users, such systems often result in excessive and unnatural mimicry. To address this issue, this study examines the effectiveness of Co-embodiment, a technique that blends the movements of a user and their partner in real-time, as a method for achieving both the benefits of mimicry and the naturalness of interaction. We examined how varying the blending ratios (0%, 25%, 50%) affects social impressions in a two-person interaction in the VE. Results showed that moderate blending (25%, 50%) enabled natural and comfortable interactions, comparable to no blending (0%). However, we did not observe significant improvements in perceived closeness or trustworthiness. Semi-structured interviews helped explain these results, revealing that participants differed in their interpretations of subtle synchronized movements. Some viewed them as signs of mutual understanding or cooperation, while others experienced discomfort. In some cases, this discomfort seemed specific to virtual contexts, as subtle similarities made participants doubt the presence of a real human behind the avatar. These findings highlight that impressions depend on whether blended movements feel socially responsive or merely imitative, stressing the need for interaction designs that enhance the sense of social presence.

Visual and auditory guidance in smart glasses suffers from distraction and noise sensitivity. To address these limitations, haptic feedback has been explored, and nose-pad vibrotactile navigation offers a promising approach, yet the effect of stimulus intervals is unexplored. This study compared cueing intervals of 0.3 s (high frequency) and 1.0 s (low frequency) in a direction-finding task. Results showed that the high-frequency condition reduced task completion time but exhibited overshooting tendencies and induced user discomfort. Although both conditions showed comparable workload and usability, participants reported trade-offs between responsiveness and comfort, suggesting context-adaptive intervals.

Social media platforms widely use the like function to evaluate content and stimulate user engagement. However, while this function was originally intended to express authentic emotional responses, it is now often used for impression management or social signaling. This study proposes an approach that directly links the bodily expression of smiling to evaluation. This approach enhances user engagement with video content by removing the need for manual operations and is also expected to improve the user’s emotional state based on the facial feedback hypothesis, which suggests that deliberately forming a smile induces positive psychological effects. We developed a video-viewing smartphone application that detects smile intensity via the front-facing camera, converts detected smiles into likes , and delivers real-time visual feedback. In a within-subjects study with 18 participants, we compared this smile-based evaluation with a traditional button-based method. Quantitative analysis showed that the smile-based evaluation significantly improved user engagement in terms of attention to content and feeling of involvement. Qualitative analysis revealed Embodied Feedback Loops comprising smile generation, system response, and amplified positive affect, strengthening emotional self-awareness. At the same time, the study highlighted challenges related to variability in users’ expressiveness and biases against content that elicits less visibly expressive content. These findings indicate that smile-based evaluation can curb strategic behavior while enriching affective engagement, offering design insights for perceptually grounded, expression-based interaction techniques in immersive media applications.

Negative psychological effects from losing in online competitive games, such as frustration and reduced self-evaluation, can lead to player disengagement. Previous research using puzzle tasks has shown that providing losers with positive feedback on their performance after the task can enhance their intrinsic motivation. However, such findings were demonstrated in analog settings where feedback was delivered verbally by a human experimenter, and it remains unclear whether similar effects hold in digital environments such as online competitive games. This study investigates a feedback intervention using a virtual agent on a computer, aimed at mitigating such negative impacts and promoting sustained intrinsic motivation. In a controlled experiment with 45 participants, we compared three types of feedback: No Feedback (NF), Feedback for the Participant (FP), which included praise for the player’s good moves, and Feedback for Both (FB), which also praised the opponent and emphasized social connection. The results indicated that providing feedback, whether it was directed at the player alone or at both the player and the opponent, significantly increased perceived competence, autonomy, intrinsic motivation, and the intention to play again compared to not providing any feedback at all. We discuss the potential of personalized, praise-based feedback in sustaining healthy motivation during competitive gameplay.

Previous studies introduced an avatar body control sharing system known as "virtual co-embodiment," where control over bodily movements and external events, or agency, of a single avatar is shared among multiple individuals. However, how this virtual co-embodiment experience influences users' perception of agency, both explicitly and implicitly, and the extent to which they are willing to take responsibility for successful or failed outcomes, remains an imminent problem. In this research, we addressed this issue using: (1) explicit agency questionnaires, (2) implicit intentional binding (IB) effect, (3) responsibility attribution measured through financial gain/loss distribution, and (4) interview to evaluate this experience where agency over the right hand's fingers was fully transferred to a human partner. Given the distinction between two layers of agency (body agency: control over actions, and external agency: action's effect on external events), we also investigated the impact of sharing only the body-level of agency. In a ball-throwing task involving 24 participants, results showed that sharing body agency over the fingers negatively affected the feeling of having control over both the fingers and the entire right upper limb, as measured by the questionnaire. However, sharing external agency did not significantly diminish the participants' perceived control over the ball-throwing, as indicated by IB. Interestingly, while IB demonstrated that participants felt greater causality for failed ball-throwing attempts, they were reluctant to take responsibility and accept financial penalties. Additionally, responsibility attribution was found to be linked to the participants' personal trait-Locus of Control.

Trust in agents within Virtual Reality is becoming increasingly important, as they provide advice and influence people's decision-making. However, previous studies show that encountering speech recognition errors can reduce users' trust in agents. Such errors lead users to ignore the agent's advice and make suboptimal decisions. While agents can offer an apology to repair trust, its effectiveness is often limited because it fails to fully repair the original level of trust. Therefore, we examined the use of social touch, a social interaction involving physical interaction between users and the virtual agent, to enhance the effect of an apology on trust repair and to increase the acceptability of its advice. In a controlled experiment (N=24), participants experienced a robotic arm touching the back of their hands while interacting with the agent before decision-making. The results showed that social touch did not repair participants' trust in agents. However, participants were more likely to accept the agent's advice when they experienced touch with physical feedback, regardless of the level of trust in the agent. We discuss the role of presenting physical haptic feedback and its influence on human-agent interactions in VR.

Recall impairment in a different environmental context from learning is called context-dependent forgetting. Two learning methods have been proposed to prevent context-dependent forgetting: reinstatement and decontextualization. Reinstatement matches the environmental context between learning and retrieval, whereas decontextualization involves repeated learning in various environmental contexts and eliminates the context dependency of memory. Conventionally, these methods have been validated by switching between physical rooms. However, in this study, we use immersive virtual environments (IVEs) as the environmental context assisted by virtual reality (VR), which is known for its low cost and high reproducibility compared to traditional manipulation. Whereas most existing studies using VR have failed to reveal the reinstatement effect, we test its occurrence using a 360-degree video-based IVE with improved familiarity and realism instead of a computer graphics-based IVE. Furthermore, we are the first to address decontextualization using VR. Our experiment showed that repeated learning in the same constant IVE as retrieval did not significantly reduce forgetting compared to repeated learning in different constant IVEs. Conversely, repeated learning in various IVEs significantly reduced forgetting than repeated learning in constant IVEs. These findings contribute to the design of IVEs for VR-based applications, particularly in educational settings.

This study proposes a new learning method that employs multiple embodied self-avatars during learning, to use the potential benefit of virtual reality (VR) for effective learning and training. In this study, by taking advantage of the benefit of virtual reality (VR), we propose a new learning method that employs multiple embodied self-avatars during learning. Based on the multiple-context effect, which posits that learning in diverse situations can prevent forgetting and enhance memory retention, we conducted a between-participants study under two conditions: the varied avatar condition, in which participants learned sign languages with different self-avatars in six iterations, and the constant avatar condition, in which the same self-avatar was used consistently. We employed sign language as a learning material that naturally draws attention to self-avatars and is suitable for investigating the effects of varying self-avatars. Initially, the varied avatar condition performed worse than the constant avatar condition. However, in a test conducted after one week in the real world, the varied avatar condition showed significantly less forgetting and better retention than the constant avatar condition. Furthermore, our results suggested a positive correlation between the degree of embodiment toward the avatars and the effectiveness of the proposed method. This study presents an innovative design approach for the use of self-avatars in VR-based education.

In the real world, the body typically remains within the downward field of view (FOV) irrespective of whether it is consciously or unconsciously observed. However, with head-mounted displays (HMDs) that have restricted viewing angles, users are unable to see their own avatars directly unless they tilt their heads significantly downward. To overcome this, virtual mirrors are commonly employed in virtual environments (VEs) to provide users with a full-body view of their avatars, which ultimately enhances the viewer's sense of embodiment (SoE). However, positioning users directly in front of a virtual mirror may restrict their movement and reduce their sense of presence. To overcome this limitation, we developed an HMD with an extended downward field of view (FOV) integrated with eye-tracking. We compared the impact of using virtual mirrors alone versus the extended downward FOV in terms of several parameters, including the head pitch angle, perceived SoE, presence, and task difficulty during a reaching task that involves both hand and foot movements. The results demonstrated that the extended downward FOV offered by the proposed HMD improved the users' sense of agency toward their avatars and enhanced their sense of presence compared to when only virtual mirrors are used. In addition, the participants tended to use the virtual mirror for hand movements and relied on the extended downward FOV for foot movements, often maintaining a stable head angle. Both the virtual mirror and extended downward FOV facilitated easier and faster completion of a reaching task while encouraging an upward head angle. However, the combined use of virtual mirrors and extended downward FOV did not have a significant impact on the SoE or presence. Notably, the extended downward FOV alone, without the integration of the virtual mirror, was more effective in increasing both SoE and presence.

Virtual reality (VR) enables beyond-real interactions (BRI) that transcend physical constraints, offering effective user experiences like extending a hand to grasp distant objects. However, adapting to novel mappings of BRI often reduces performance and the sense of embodiment. To address this, we propose using noisy tendon electrical stimulation (n-TES) to decrease proprioceptive precision. Previous studies have suggested that attenuating proprioceptive precision is crucial for sensory-motor adaptations. Thus, we hypothesize that n-TES, which has been shown to reduce proprioceptive precision and induce visual-dependent perception in VR, can enhance user adaptation to BRI. We conducted a user study using go-go interaction, a BRI technique for interacting with distant objects, to assess the effects of n-TES. Given the individual variability in n-TES response, participants first underwent a proprioceptive precision test to determine the optimal stimulation intensity to lower the proprioceptive precision from 5 levels $(\sigma=0.25-125\text{mA})$. Reaching tasks using a 2x2 within-participants design evaluated the effects of go-go interaction and n-TES on performance, subjective task load, and embodiment. Results from 24 participants showed that go-go interaction increased reaching time and task load while decreasing the sense of embodiment. Contrary to our hypothesis, n-TES did not significantly mitigate most of these negative effects of go-go interaction, except that perceived agency was higher with n-TES during go-go interaction. The limited effectiveness of n-TES may be due to participants' habituation or sensory adaptation during the tasks. Future research should consider the adaptation process to BRI and investigate different BRI scenarios.

Redirected walking (RDW) is a method for exploring virtual spaces larger than physical spaces while preserving a natural walking sensation. Expanding the range of visual manipulation gains that can be applied without causing discomfort is necessary to apply RDW in practice. Ankle tendon electrical stimulation (TES) can expand the range by inducing body tilt sensation and sway. Therefore, in this study, we proposed a locomotion method that applies ankle TES to RDW. In Experiment 1, we evaluated the effect of TES on the detection threshold (DT), which is the maximal gain at which visual manipulation remains unnoticed. The results indicated that the DT was expanded when TES was applied to induce the body tilt sensation in the same direction as the RDW's visual manipulation. Specifically, the pooled mean of the DT was expanded by more than 18%. In Experiment 2, we evaluated the applicability, a supplementary index for assessing locomotion techniques. The results demonstrated that ankle TES mitigates the reduction of the applicability, especially under a curvature gain of $\pm 0.3 [m^{-1}]$±0.3[m-1].

Virtual reality, which enables users to engage in physical activities in ways distinct from those in the real world, is increasingly recognized for its potential to enhance motor skill acquisition. Research on co-embodiment learning, in which instructors and learners utilize a single avatar that represents a weighted average of their movements, has demonstrated its efficacy in facilitating motor skill development. However, the current implementation of co-embodiment learning necessitates the real-time participation of instructors proficient in both virtual reality and co-embodiment, which poses challenges for its widespread adoption. To address this limitation, this study proposed a method for developing instructors trained on human motor data to effectively support motor skill learning through co-embodiment. The AI model was trained using supervised learning on data obtained from human motor learning sessions that employed co-embodiment. To evaluate the performance of the AI instructor, we compared the learning performance in co-embodiment learning with that of the AI instructor, recorded human instructor data, and a human instructor as well as in solo learning. The results showed that practicing with the AI instructor significantly improved learning efficiency compared with practicing alone or with recorded data and was comparable to that achieved by practicing with a human instructor.

Hand redirection, which subtly adjusts a user's hand movements in a virtual environment, can modify perception and movement by providing real-time corrections to motor feedback. In the context of motor learning and rehabilitation, observing replays of movements has been shown to enhance motor function. The application of hand redirection to these replays by making movements appear larger or smaller than they actually are has the potential to improve motor function. However, the detection threshold for hand redirection, specifically in the context of motion replays, remains unclear, as it has primarily been studied in real-time feedback settings. This study aims to determine the threshold at which hand redirection during post-exercise replay sessions becomes detectable. We conducted two psychophysical experiments to evaluate how much discrepancy between replayed and actual movements can go unnoticed by users, both with hand redirection (N=20) and without (N=18). Our findings reveal a tendency for the amount of movement during replay to be underestimated. Furthermore, compared to conventional real-time hand redirection without replay, replay manipulations involving redirection applied during the preceding reaching task resulted in a significantly larger JND. These insights are crucial for leveraging hand redirection techniques in replay-based motor learning applications.

The methods of displaying virtual agents and avatars as lecturers have been studied in remote lectures and asynchronous education. In this study, we tested the hypothesis that presenting information using multiple virtual avatars facilitates users’ memory performance. In a memory task involving remembering 100 words, we compared a condition where the same avatar narrated all the words to a condition where different avatars each presented 25 words. The results of the online experiment showed no significant differences in the number of words recalled between both conditions. However, we found that switching between avatars significantly aided in recalling the source of information, namely, discerning which avatars read specific words. Furthermore, through an exploratory analysis, we found that using a human avatar resulted in better memory retention than using other avatars (red panda, extraterrestrial being, robot). The participants’ feedback also indicated that the red panda avatar may be perceived favorably and that the quality of lip-syncing was associated with discomfort. These findings are expected to enrich future research on agents and avatars in educational settings.

Advancements in virtual reality technology have facilitated users in freely changing their appearance during communication. Notably, it has become evident that the perceptual and cognitive responses of users vary when their partner uses an animal avatar compared to those when using a human avatar. This study aimed to clarify the effects of partner-avatars on memory through experiments wherein participants were tasked with memorizing the profiles spoken by the partner-avatars. A preliminary experiment validated the assumption that both the animal avatars and the virtual environments could be integrated into processes of profile memory. Subsequently, the primary experiment examined differences in performance in remembering partners’ profiles when the partners were animal avatars versus human avatars. The results showed that the animal avatar condition was marginally inferior to the human avatar condition. Moreover, the perceived intimacy and the presence of others were significantly lower in the animal avatar condition. These findings suggested that interacting with animal avatars may have specific drawbacks in terms of profile memory and engagement in social interactions within virtual environments.

Virtual reality is gaining attention as a tool to facilitate motor skill learning. Numerous studies have been conducted on virtual co-embodiment, in which the movements of the teacher and student are weighted and averaged into a single avatar for motor skill learning. Previous studies have shown that virtual co-embodiment with a human teacher enhances motor skill learning efficiency, and the behavior of the human teacher is important for effective learning. However, this system has some challenges, such as the human playing the teacher’s role must be skilled in teaching and using virtual co-embodiment, and the teacher can be adversely affected by the learner. To solve these problems, we created an AI teacher using long short-term memory, which outputs the behavior of the teacher based on the input of the learner’s behavior data and the state of the experimental environment and trained the AI teacher by supervised learning using behavior training data of a human virtual co-embodiment. We confirmed that this AI teacher can generate behaviors similar to those of the human teacher and investigated the efficiency of motor skill learning using virtual co-embodiment with the AI teacher. Co-embodiment with the AI teacher reduced performance during the learning phase but improved performance during subsequent independent task execution. We further analyzed the assist proportion of the teacher and observed that when the co-embodied with an AI teacher, the assist proportion is lower than when the co-embodied with a human, suggesting that learning efficiency may be enhanced when the co-embodied partner is a mixture of supportive and obstructive.

Robotic avatars and telepresence technology enable people with disabilities to engage in physical work. Despite the recent popularity of the metaverse, few studies have explored the use of virtual avatars and environments by people with disabilities. In this study, seven disabled participants working in a cafe where remote customer service is provided via robotic avatars, were engaged in the development and use of personalized virtual avatars displayed on a large screen in-situ in combination with existing physical robots, creating a hybrid cyber-physical space. We conducted longitudinal semi-structured interviews to investigate the psychological changes experienced by the participants. The results revealed that mass-produced robotic avatars allowed participants to not disclose their disability if they did not want to, but also backgrounded their identities; by contrast, customized virtual avatars shaped without physical constraints, highlighted their personalities. The combined use of robotic and virtual avatars complemented each other and can support pilots in redefining their identity.

This study investigates the impact of both self-and other’s avatars (virtual bodies) with complementary traits on self-image and behavior change, drawing on role theory: individuals adopt and internalize their social roles as expected by others. In our experiment, participants and a non-player character played a cooperative virtual reality action game together, embodying a “warrior” avatar and a “witch” avatar with complementary appearances and in-game abilities. Results revealed that role assignments based on the use of complementary avatars has significant interaction effects with participants’ personal characteristics on influencing individuals’ behavior and self-image. Furthermore, a risk of unpredicted change resulting from failed role assignments was discovered, suggesting the importance of achieving role assignments that match avatar complementarity. By providing a new perspective on the impact of interactions between multiple users with diverse avatars, these findings contribute to our understanding of the mechanisms of cognitive augmentation with avatars and have implications for the design of avatar-related experiences in the metaverse.

Episodic memory is influenced by environmental contexts, such as location and auditory stimuli. The most well-known effect is the reinstatement effect, which refers to the phenomenon where contextual matching between learning and testing enhances memory retrieval. Previous studies have investigated whether the reinstatement effect can be observed within immersive virtual environments. However, only a limited number of studies have reported a significant reinstatement effect using virtual reality, while most have failed to detect it. In this study, we re-examined the reinstatement effect using 360-degree video-based virtual environments. Specifically, we carefully selected virtual environments to elicit different emotional responses, which has been suggested as a key factor in inducing a robust reinstatement effect in the physical world. Surprisingly, we found a significant reversed reinstatement effect with a large effect size. This counter-intuitive result suggests that contextual congruence does not necessarily enhance memory and may even interfere with it. This outcome may be explained by the retrieval-induced forgetting phenomenon, but further exploration is needed. This finding is particularly important for virtual reality-based, educational applications and highlights the need for a deeper understanding of the complex interactions between memory and contextual cues within virtual environments.

As a redirected walking technique, a method has been proposed to enable users to walk in an undulating virtual space even in a flat physical environment by setting the slope of the floor in the virtual environment to be different from that in the physical environment without causing discomfort. However, the slope range in which discrepancies between visual and proprioceptive sensations are not perceived is limited, restricting the slopes that can be presented. In this study, we proposed redirected walking using electrical stimulation of the Achilles and tibialis anterior muscle tendons, extending the applicable slope range of redirected walking without compromising the natural gait sensation. Electrical stimulation of the ankle tendons affects the proprioceptive sensation and gives the illusion of tilting in the standing posture, expanding the applicable slope range. Two experiments showed that the proposed method improved the experience of uphill and downhill walking in terms of the range of the virtual slope where a high naturalness of gait and a high congruency of visual and proprioceptive sensations are maintained. Notably, electrical stimulation of the Achilles tendons significantly improved the naturalness of the walking experience during virtual downhill walking, which has been considered more challenging in previous studies.

The limited nonverbal cues and spatially distributed nature of remote communication make it challenging for unacquainted members to be expressive during social interactions over video conferencing. Though it enables seeing others' facial expressions, the visual feedback can instead lead to unexpected self-focus, resulting in users missing cues for others to engage in the conversation equally. To support expressive communication and equal participation among unacquainted counterparts, we propose SealMates, a behavior-driven avatar in which the avatar infers the engagement level of the group based on collective gaze and speech patterns and then moves across interlocutors' windows in the video conferencing. By conducting a controlled experiment with 15 groups of triads, we found the avatar's movement encouraged people to experience more self-disclosure and made them perceive everyone was equally engaged in the conversation than when there was no behavior-driven avatar. We discuss how a behavior-driven avatar influences distributed members' perceptions and the implications of avatar-mediated communication for future platforms.

During video conferencing, technical issues such as network impairments can unexpectedly hinder remote collaboration for distributed teams. However, it remains unclear how such issues affect the impression formation process between unacquainted interlocutors in situations like job interviews or kick-off meetings. Having first encounters online without prior in-person interaction has become prevalent nowadays. Therefore, examining the impact of technical issues on remote unacquainted interlocutors and exploring design solutions to reduce the negative impact of technical issues on the impression is essential in informing the design of future communication systems. With three controlled experiments simulating an online job interview, we discovered that technical issues made people give low credibility ratings to remote job applicants (Study 1). We further examined two intervention approaches to reduce the negative impact of technical issues in video conferencing, including a virtual agent represented in a human-like avatar to actively take responsibility for the technical issues (Study 2), and forewarning messages to inform people about the technical issues in video conferencing (Study 3). Results demonstrated that introducing an agent that represents a communication system to take responsibility for technical issues actively could reduce the responsibility people assign to remote job applicants, but without increasing their positive credibility rating for remote job applicants. Furthermore, showing forewarning messages, which explicitly made people aware the cause of technical issues was unidentifiable and its potential negative impacts on impression formation, enabled people to rate the credibility of the remote counterpart without being influenced by technical issues. We discussed how interventions can be designed to mitigate negative attribution outcomes when encountering uncontrollable technical issues in computer-mediated communication.

Pseudo-haptic techniques are used to modify haptic perception by appropriately changing visual feedback to body movements. Based on the knowledge that tendon vibration can affect our somatosensory perception, this article proposes a method for leveraging tendon vibration to enhance pseudo-haptics during free arm motion. Three experiments were performed to examine the impact of tendon vibration on the range and resolution of pseudo-haptics. The first experiment investigated the effect of tendon vibration on the detection threshold of the discrepancy between visual and physical motion. The results indicated that vibrations applied to the inner tendons of the wrist and elbow increased the threshold, suggesting that tendon vibration can augment the applicable visual motion gain by approximately 13% without users detecting the visual/physical discrepancy. Furthermore, the results demonstrate that tendon vibration acts as noise on haptic motion cues. The second experiment assessed the impact of tendon vibration on the resolution of pseudo-haptics by determining the just noticeable difference in pseudo-weight perception. The results suggested that the tendon vibration does not largely compromise the resolution of pseudo-haptics. The third experiment evaluated the equivalence between the weight perception triggered by tendon vibration and that by visual motion gain, that is, the point of subjective equality. The results revealed that vibration amplifies the weight perception and its effect was equivalent to that obtained using a gain of 0.64 without vibration, implying that the tendon vibration also functions as an additional haptic cue. Our results provide design guidelines and future work for enhancing pseudo-haptics with tendon vibration.

Walking-in-place (WIP) is a locomotion technique that enables users to "walk infinitely" through vast virtual environments using walking-like gestures within a limited physical space. This article investigates alternative interaction schemes for WIP, addressing successively the control, input, and output of WIP. First, we introduce a novel height-based control to increase advanced speed. Second, we introduce a novel input system for WIP based on elastic and passive strips. Third, we introduce the use of pseudo-haptic feedback as a novel output for WIP meant to alter walking sensations. The results of a series of user studies show that height and frequency based control of WIP can facilitate higher virtual speed with greater efficacy and ease than in frequency-based WIP. Second, using an upward elastic input system can result in a stable virtual speed control, although excessively strong elastic forces may impact the usability and user experience. Finally, using a pseudo-haptic approach can improve the perceived realism of virtual slopes. Taken together, our results suggest that, for future VR applications, there is value in further research into the use of alternative interaction schemes for walking-in-place.

The time elapsed during a virtual reality (VR) experience is estimated to be short. Time estimation, a feeling related to timescales longer than a few seconds, is thought to be related to interoception. The shortening of time estimation may be caused by the VR experience distorting bodily awareness based on interoception; however, the details of this change have not been clarified. The characteristics of avatars are likely to affect bodily awareness. Therefore, based on the hypothesis that bodily awareness during a VR experience is altered by manipulating the avatar’s degree of anthropomorphism, this study investigated the effect of avatar anthropomorphism on bodily awareness and time estimation. In an experiment, participants performed a heartbeat discrimination task and a one-minute time estimation task, in the real world (Real condition), in the virtual world using a human avatar (VR-Human condition), and in the virtual world using a robot avatar (VR-Robot condition). The results showed that the perceived sense of agency was significantly lower in the VR-Human condition than in the VR-Robot condition, and interoceptive accuracy was significantly higher in the VR-Human condition than in the VR-Robot condition. However, there were no differences in time estimation between conditions. These results indicate that it is possible to conduct experiments to manipulate bodily awareness by manipulating the avatar characteristics. These findings can provide insights into understanding the relationship between bodily awareness and time estimation.

This study investigated the learning effect and skill retention when virtual co-embodiment, in which movements of people were weighted and averaged into a single avatar, was used to learn motor skills requiring declarative memory. Previous studies have shown that virtual co-embodiment promotes the efficiency of motor skill learning, which relies on procedural memory such as movement procedures. However, declarative memory, such as the connection between specific instructions and actions, plays an important role in learning motor skills. This study compared the learning efficiency and skill retention after one week of repeatedly performing the task of touching a specified combination of virtual spheres with both hands in accordance with the symbols presented, using virtual co-embodiment and in a condition in which the task was performed alone. The results showed that virtual co-embodiment improves learning efficiency for motor skills related to declarative memory as well as procedural memory and promotes long-term retention of skills.

When receiving text messages from unacquainted colleagues in fully remote workplaces, insufficient mutual understanding and limited social cues can lead people to misinterpret the tone of the message and further influence their impression of remote colleagues. Emojis have been commonly used for supporting expressive communication; however, people seldom use emojis before they become acquainted with each other. Hence, we explored how changing facial expressions in profile pictures could be an alternative channel to communicate socio-emotional cues. By conducting an online controlled experiment with 186 participants, we established that changing facial expressions of profile pictures can influence the impression of the message receivers toward the sender and the message valence when receiving neutral messages. Furthermore, presenting incongruent profile pictures to positive messages negatively affected the interpretation of the message valence, but did not have much effect on negative messages. We discuss the implications of affective profile pictures in supporting text-based communication.

To increase the flexibility of haptic feedback in virtual reality (VR), hand redirection (HR) has been proposed to shift the hand’s virtual position from its actual position. To expand the range of HR applications, a method to broaden the detection threshold (DT), which is the maximum amount of shift that can be applied without the user noticing, is required. Multisensory integration studies have revealed that the reliability of senses affects the weight of integration. To expand the DTs of HR, we propose a method to increase visual dominance in the integration of vision and proprioception by introducing noise to the latter, thereby decreasing its reliability through weak Gaussian white noise electrical stimulation ($\sigma$=0.5mA). The results of a user study comprising 22 participants (11 women and 11 men) confirm that noisy electrical stimulation significantly expands the DTs of HR with the mean range of DTs ($R_{DT}$) was 20.48° (SD =7.90) with electrical stimulation and 19.15° (SD =7.11) without electrical stimulation. Interestingly, this effect was only observed in women. The average $R_{DT}$ for men was 15.36° (SD =6.13) and 15.18°(SD=5.58), whereas that for women was 25. 61°(SD=5.89) and 23.12°(SD=6.21), with and without electrical stimulation, respectively. Electrical stimulation was mostly tolerable for the participants and did not affect embodiment or presence ratings. These results suggest that expansion of the DT without disturbing the user’s VR experience is feasible.

Timbre is high-dimensional and sensuous, making it difficult for musical-instrument learners to improve their timbre. Although some systems exist to improve timbre, they require expert labeling for timbre evaluation; however, solely visualizing the results of unsupervised learning lacks the intuitiveness of feedback because human perception is not considered. Therefore, we employ crossmodal correspondences for intuitive visualization of the timbre. We designed TimToShape, a system that visualizes timbre with 2D shapes based on the user’s input of timbre–shape correspondences. TimToShape generates a shape morphed by linear interpolation according to the timbre’s position in the latent space, which is obtained by unsupervised learning with a variational autoencoder (VAE). We confirmed that people perceived shapes generated by TimToShape to correspond more to timbre than randomly generated shapes. Furthermore, a user study of six violin players revealed that TimToShape was well-received in terms of visual clarity and interpretability.

Most studies on the Proteus Effect, which examines how avatars can influence users’ behavior through evoked stereotypes, have primarily manipulated only participants’ own avatars as the independent variable. However, in reality, there are numerous scenarios where individuals recognize their uniqueness by comparing themselves to others. Therefore, this study aimed to explore the impact of recognizing one’s distinctiveness by comparing one’s own avatar’s appearance with others on behavioral changes. In our experiment, participants and non-player characters engaged in playing the Japanese drum ‘Taiko’ together within a virtual environment. They utilized avatars dressed in suits or ‘Happi,’ which is a traditional Japanese festival costume. The results demonstrated that both the uniformity/distinctiveness and the type of avatar appearance played a joint role in influencing the speed and amplitude of arm swings during the taiko performance. This finding provides valuable insights into comprehending the mechanisms of behavior change in settings where multiple avatars interact, such as social virtual reality, and aids in designing virtual spaces that foster appropriate interactions among individuals.

Episodic memory incorporates environmental contexts, and memory retrieval is aided by matching the retrieval context to the encoding context. This study tested whether similar context-dependency of memory could be confirmed in virtual environments. Participants learned words in a 360-degree video-based virtual environment depicting either natural or urban landscapes. Immediately, they completed a test in the same virtual environment. After two days, half of the participants underwent a final test with the same context as that on the initial day, whereas the other half underwent it with a different context. Surprisingly, participants tested in a different context exhibited significantly lower forgetting than those tested in the same context, which contradicted our hypothesis.

Virtual reality (VR) is a promising tool for motor skill learning. Previous studies have indicated that observing and following a teacher's movements from a first-person perspective using VR facilitates motor skill learning. Conversely, it has also been pointed out that this learning method makes the learner so strongly aware of the need to follow that it weakens their sense of agency (SoA) for motor skills and prevents them from updating the body schema, thereby preventing long-term retention of motor skills. To address this problem, we propose applying "virtual co-embodiment" to motor skill learning. Virtual co-embodiment is a system in which a virtual avatar is controlled based on the weighted average of the movements of multiple entities. Because users in virtual co-embodiment overestimate their SoA, we hypothesized that learning using virtual co-embodiment with a teacher would improve motor skill retention. In this study, we focused on learning a dual task to evaluate the automation of movement, which is considered an essential element of motor skills. As a result, learning in virtual co-embodiment with the teacher improves motor skill learning efficiency compared with sharing the teacher's first-person perspective or learning alone.

Advances in virtual reality technology have enabled the creation of virtual environments (VEs) with significantly high visual fidelity when compared to real environments (REs). In this study, we use a high-fidelity VE to examine two effects caused by alternating VE and RE experiences: "context-dependent forgetting" and "source-monitoring errors." The former effect is that memories learned in VEs are more easily recalled in VEs than in REs, whereas memories learned in REs are more easily recalled in REs than in VEs. The source-monitoring error is that memories learned in VEs are easily confused with those learned in REs, making discriminating the source of the memory difficult. We hypothesized that the visual fidelity of VEs is responsible for these effects and conducted an experiment using two types of VEs: a high-fidelity VE created using photogrammetry techniques and low-fidelity VE created with primitive shapes and materials. The results show that the high-fidelity VE significantly improved the sense of presence. However, the level of the visual fidelity of the VEs did not show any effect on context-dependent forgetting and source-monitoring errors. Notably, the null results of the context-dependent forgetting between the VE and RE were strongly supported by Bayesian analysis. Thus, we indicate that context-dependent forgetting does not necessarily occur, which will be helpful for VR-based education and training.

Can you speak the way you desire without feeling the pressure to conform to standards of speaking? In this study, we investigated the impact of user-controlled voice manipulation and listening to recordings of model speakers on self-perceptions of voice and speech. Quantitative analysis showed that there was a significant improvement in the perceived confidence of tone by listening to model speakers, but there were no significant improvements due to voice manipulation. Qualitative analysis of interviews revealed that participants responded positively to the visual and auditory feedback provided by the voice manipulation software. The participants also evaluated the quality of model speakers to decide whether or not they wanted to refer to them for speech practice. Based on the results of these analyses, we summarized the design implications for a speech practice system that would allow further investigation of the impact of the system on self-perceptions of speech performance.

Presenting haptic feedback in virtual reality (VR) is a long-standing challenge, with passive haptics being one way of presenting haptic feedback inexpensively. However, passive haptics requires props in physical environments that are co-located with their virtual counterparts, which is often not the case in the real world. Although redirected hands and other methods have previously been proposed to solve this problem, significant differences between the displayed and actual hand positions can cause the degradation of presence and sense of embodiment, limiting the range of presentable environments. In this study, we present a new hand displacement method called wormholes, in which the virtual hand is teleported discontinuously as the user inserts their hand into the hole. The experiment showed that the wormhole could maintain the sense of embodiment, presence, and task performance even with large hand displacements. Our method enables to apply passive haptics even when the actual and virtual environments are quite different, contributing to the realization of inexpensive and flexible haptic presentation in VR applications.

In Virtual Reality, self-touch (ST) stimulation is a promising method of sense of body ownership (SoBO) induction that does not require an external effector. However, its applicability to dysmorphic bodies has not been explored yet and remains uncertain due to the requirement to provide incongruent visuomotor sensations. In this, paper, we studied the effect of ST stimulation on dysmorphic hands via haptic retargeting, as compared to a classical external-touch (ET) stimulation, on the SoBO. Our results indicate that ST can induce similar levels of dysmorphic SoBO than ET stimulation, but that some types of dysmorphism might decrease the ST stimulation accuracy due to the nature of the re-targeting that they induce.

Virtual reality helps us learn complex motor skills by providing a situation in which we observe or follow a teacher’s movements from a first-person perspective. However, it has been suggested that if the learners themselves do not behave actively, their body schemes will not be updated and motor skills will be acquired temporarily, but will not be retained in the long term. As a solution to this problem, “co-embodiment” in which two people embody an avatar that reflects the weighted average of their movements was proposed, and it is shown that the user can feel an excessive sense of agency (SoA) even when their control weight is small. From the perspective of motor skill learning, the learner must feel as strong a SoA as possible while performing the exercise as close to the teacher as possible. Therefore, in this study, we propose a method to transitionally change the weights in a situation where co-embodiment is used, such that a strong SoA is felt despite the high weights of control by others. Considering the two-step account of the agency model, which states that the SoA is influenced by context, we tested the hypothesis that an initially strong SoA can maintain the SoA despite a gradually decreased control weight. The experimental results support this hypothesis, and it is expected that the proposed method will enhance the effectiveness of motor skill learning using co-embodiment.

Humans can estimate the properties of wielded objects (e.g., inertia and viscosity) using the force applied to the hand. We focused on this mechanism and aimed to represent the properties of wielded objects by dynamically changing the force applied to the hand. We propose MetamorphX, which uses control moment gyroscopes (CMGs) to generate ungrounded, 3-degrees of freedom moment feedback. The high-response moments obtained CMGs allow the inertia and viscosity of motion to be set to the desired values via impedance control. A technical evaluation indicated that our device can generate a moment with a 60-ms delay. The inertia and viscosity of motion were varied by 0.01 kgm2 and 0.1 Ns, respectively. Additionally, we demonstrated that our device can dynamically change the inertia and viscosity of motion through virtual reality applications.

Emotional contagion is a phenomenon in which one’s emotions are transmitted among individuals unconsciously by observing others’ emotional expressions. In this paper, we propose a method for mediating people’s emotions by triggering emotional contagion through artificial bodily changes such as pseudo tears. We focused on shedding tears because of the link to several emotions besides sadness. In addition, it is expected that shedding tears would induce emotional contagion because it is observable by others. We designed an eyeglasses-style wearable device, Teardrop glasses, that release water drops near the wearer’s eyes. The drops flow down the cheeks and emulate real tears. The study revealed that artificial crying with pseudo tears increased sadness among both wearers and those observing them. Moreover, artificial crying attenuated happiness and positive feelings in observers. Our findings show that actual bodily changes are not necessary for inducing emotional contagion as artificial bodily changes are also sufficient.

Recent embodied cognition studies revealed that our mind can be influenced by the state and recognition of our bodies. Meanwhile, recent developments in virtual reality and augmented human technologies enable us to flexibly modify and extend our body itself and recognition of our bodily states. Considering these together, our mind would be designed by utilizing these technologies which augment our body or body perception. In other words, we can design ourselves through the design of the virtual body and embodied interaction we use in cyber-physical world to become the person we want to be. Here, in reference to the fact that the relationship between the body and the mind has been discussed in many works including M. Shirow's "GHOST IN THE SHELL" since the presentation of G. Ryle's concept of "Ghost in the machine," the function of the human mind that arises from the body is defined as "ghost", and the technology to create the desired change in our mind by changing our virtual body is called as "Ghost Engineering". There are several patterns in the way our body is made possible by technology for the first time, and what kind of mind function changes depends on these patterns. For example, "Transformation", the experience of becoming someone else, or being able to demonstrate abilities beyond one's own, can change one's self-image and influence behavior, abilities, and even thinking. The "Body multiplexing" experience of manipulating multiple bodies not only expands our physical abilities, but also affects our ability to balance our thoughts and changes the way we communicate with multiple people. Merged body, in which multiple people manipulate a single body, not only enables a new kind of cooperative work, but also enables the transfer of motor intentions and physical skills from human to human. I would like to introduce recent achievements and future implications of Ghost Engineering technology, and discuss what kind of possibilities for the self and society that such technology enables.

In this study, considering the characteristics of multisensory integration, we examined a method for improving redirected walking (RDW) by adding noise to the vestibular system to reduce the effects of vestibular inputs on self-motion perception. In RDW, the contradiction between vestibular inputs and visual sensations may make users notice the RDW manipulation, resulting in discomfort throughout the experience. Because humans integrate multisensory information by considering the reliability of each modality, by reducing the effects of vestibular inputs on self-motion perception, it is possible to suppress awareness of and discomfort during RDW manipulation and improve the effectiveness of the manipulation. Therefore, we hypothesized that adding noise to the vestibular inputs would reduce the reliability of the vestibular sensations and enhances the effectiveness of RDW by improving the relative reliability of vision. In this study, we used noisy galvanic vestibular stimulation (GVS) to reduce the reliability of vestibular inputs. GVS is a method of stimulating vestibular organs and nerves by applying small electrical currents to the bilateral mastoid. To reduce the reliability of vestibular inputs, we employed noisy GVS whose current pattern is white noise. We experimented with comparing the threshold of curvature gains between noisy GVS conditions and a control condition.

Cinematic virtual reality, or 360° video, provides viewers with an immersive experience, allowing them to enjoy a video while moving their head to watch in any direction. However, there is an inevitable problem of feeling fear of missing out (FOMO) when viewing a 360° video, as only a part of the video is visible to the viewer at any given time. To solve this problem, we developed a technique to present a panoramic thumbnail of a full 360° video to users through a head-mounted display. With this technique, the user can grasp the overall view of the video as needed. We conducted an experiment to evaluate the FOMO, presence, and quality of viewing experience while using this technique compared to normal viewing without it. The results of the experiment show that the proposed technique relieved FOMO, the quality of viewing experience was improved, and there was no difference in presence. We also investigated how users interacted with this new interface based on eye tracking and head tracking data during viewing, which suggested that users used the panoramic thumbnail to actively explore outside their field of view.

Insufficient social cues between distributed learners in online learning could result in lack of engagement and social bonds. With the development of wearable sensing, sharing physiological data can be used to enhance mutual understanding and connectedness among sharers. Our work aims to explore the potential of sharing heart rate (HR) and heart rate variability (HRV) collected from distributed learners to enhance their online learning experiences. We implemented a physiological streaming system and conducted a field study with 11 learners in online classes. This paper describes the study and discusses our interview findings by contrasting the influence of visualized collective physiological data from viewpoints of data contributors and viewers. Our exploratory results suggest streaming collective HR and HRV from multiple distributed learners could be used in online classes to improve engagement and sense of community.

Several virtual reality (VR) proxies have been developed that can emulate impact sensations by generating actual forces on the hand. Although these proxies contribute to increasing the reality of VR, they still have some limitations, such as high latency, high power consumption, and low frequency to provide impact sensations. To overcome these limitations, we first propose a method to provide an impact sensation without actual force generation by quickly changing the rotational inertia of a handheld proxy while users are swinging it. Then, we developed Unident, a handheld proxy capable of changing its rotational inertia by moving a weight along one axis at a high speed. Two experiments were conducted to evaluate the ability of Unident to provide users with impact sensations. In the first experiment, we demonstrate that Unident can physically provide an impact sensation applied to a handheld object by analyzing the pressure on the user's palm. The second experiment shows that Unident can provide an impact sensation with various magnitudes depending on the amount of rotational inertia to be changed. Finally, we present an application that can be enabled by Unident.

Timbre is a crucial element in playing musical instruments, and it is difficult for beginners to learn it independently. Therefore, external feedback (FB) is required. However, conventional FB methods lack intuitiveness in visualization. In this study, we propose a novel FB method that adopts crossmodal correspondence to enhance the intuitive visualization of timbre with visual shapes. Based on the experiments, it was inferred that the FB based on crossmodal correspondence prevents dependence on FB and promotes learning.

In this article, we introduce a concept called "virtual co-embodiment", which enables a user to share their virtual avatar with another entity (e.g., another user, robot, or autonomous agent). We describe a proof-of-concept in which two users can be immersed from a first-person perspective in a virtual environment and can have complementary levels of control (total, partial, or none) over a shared avatar. In addition, we conducted an experiment to investigate the influence of users' level of control over the shared avatar and prior knowledge of their actions on the users' sense of agency and motor actions. The results showed that participants are good at estimating their real level of control but significantly overestimate their sense of agency when they can anticipate the motion of the avatar. Moreover, participants performed similar body motions regardless of their real control over the avatar. The results also revealed that the internal dimension of the locus of control, which is a personality trait, is negatively correlated with the user's perceived level of control. The combined results unfold a new range of applications in the fields of virtual-reality-based training and collaborative teleoperation, where users would be able to share their virtual body.

Common existing head-mounted displays (HMDs) for virtual reality (VR) provide users with a high presence and embodiment. However, the field of view (FoV) of a typical HMD for VR is about 90 to 110 [deg] in the diagonal direction and about 70 to 90 [deg] in the vertical direction, which is narrower than that of humans. Specifically, the downward FoV of conventional HMDs is too narrow to present the user avatar's body and feet. To address this problem, we have developed a novel HMD with a pair of additional display units to increase the downward FoV by approximately 60 ( 10+50) [deg]. We comprehensively investigated the effects of the increased downward FoV on the sense of immersion that includes presence, sense of self-location (SoSL), sense of agency (SoA), and sense of body ownership (SoBO) during VR experience and on patterns of head movements and cybersickness as its secondary effects. As a result, it was clarified that the HMD with an increased FoV improved presence and SoSL. Also, it was confirmed that the user could see the object below with a head movement pattern close to the real behavior, and did not suffer from cybersickness. Moreover, the effect of the increased downward FoV on SoBO and SoA was limited since it was easier to perceive the misalignment between the real and virtual bodies.

Hand interaction techniques in virtual reality often exploit visual dominance over proprioception to remap physical hand movements onto different virtual movements. However, when the offset between virtual and physical hands increases, the remapped virtual hand movements are hardly self-attributed, and the users become aware of the remapping. Interestingly, the sense of self-attribution of a body is called the sense of body ownership (SoBO) in the field of psychology, and the realistic the avatar, the stronger is the SoBO. Hence, we hypothesized that realistic avatars (i.e., human hands) can foster self-attribution of the remapped movements better than abstract avatars (i.e., spherical pointers), thus making the remapping less noticeable. In this article, we present an experiment in which participants repeatedly executed reaching movements with their right hand while different amounts of horizontal shifts were applied. We measured the remapping detection thresholds for each combination of shift directions (left or right) and avatar appearances (realistic or abstract). The results show that realistic avatars increased the detection threshold (i.e., lowered sensitivity) by 31.3 percent than the abstract avatars when the leftward shift was applied (i.e., when the hand moved in the direction away from the body-midline). In addition, the proprioceptive drift (i.e., the displacement of self-localization toward an avatar) was larger with realistic avatars for leftward shifts, indicating that visual information was given greater preference during visuo-proprioceptive integration in realistic avatars. Our findings quantifiably show that realistic avatars can make remapping less noticeable for larger mismatches between virtual and physical movements and can potentially improve a wide variety of hand-remapping techniques without changing the mapping itself.

In this paper, we propose a novel redirected walking (RDW) technique that applies dynamic bending and curvature gains so that users perceive less discomfort than existing techniques that apply constant gains. Humans are less likely to notice continuous changes than those that are sudden. Therefore, instead of applying constant bending or curvature gains to users, we propose a dynamic method that continuously changes the gains. We conduct experiments to investigate the effect of dynamic gains in bending and curvature manipulation with regards to discomfort. The experimental results show that the proposed method significantly suppresses discomfort by up to 16 and 9% for bending and curvature manipulations, respectively.

Preventing users from walking through virtual boundaries (e.g., walls) is an important issue to be addressed in room-scale virtual environments (VEs), considering the safety and design limitations. Sensory feedback from wall collisions has been shown to be effective; however, it can disrupt the immersion. We assumed that a greater sense of presence would discourage users from walking through walls and conducted a two-factor between-subjects experiment (N = 92) that controls the anthropomorphism (realistic or abstract) and visibility (full-body or hand-only) of self-avatars. We analyzed the participants' behaviors and the moment they first penetrated the wall in game-like VEs that gradually instigated participants to penetrate the walls. The results showed that the realistic full-body self-avatar was the most effective for discouraging the participants from penetrating the walls. Furthermore, the participants with lower presence tended to walk through the walls sooner. This study can contribute to applications that require realistic user responses in VEs.

Here, we present the outcome of the 4th workshop on Multisensory Approaches to Human-Food Interaction (MHFI), developed in collaboration with ICMI 2020 in Utrecht, The Netherlands. Capitalizing on the increasing interest on multisensory aspects of human-food interaction and the unique contribution that our community offers, we developed a space to discuss ideas ranging from mechanisms of multisensory food perception, through multisensory technologies, to new applications of systems in the context of MHFI. All in all, the workshop involved 11 contributions, which will hopefully further help shape the basis of a field of inquiry that grows as we see progress in our understanding of the senses and the development of new technologies in the context of food.

In this paper, we present a study of redirected walking (RDW) that shifts the positional relationship between visual and auditory cues during curvature manipulation. It has been shown that, when presented with incongruent visual and auditory spatial cues during a localization task, human observers integrate that information based on each cue's relative reliability, which determines their final perception of the target object's location. This multi-modal integration model is known as maximum likelihood estimation (MLE). By altering the visual location of objects that users perceive in virtual reality (VR) through auditory cues during redirection manipulation, we expect fewer users to notice the manipulation, which helps increase the usable curvature gain. Most existing studies on MLE in multi-modal integration have used random-dot stereograms as visual cues under stable motion states. In the present study, we first investigated whether this model holds while walking in VR environment. Our results indicate that in a walking state, users' perceptions of the target object's location shift toward auditory cue as the reliability of vision decreases, in keeping with the trend shown in previous studies on MLE. Based on this result, we then investigated the detection threshold of curvature gains during redirection manipulation under a condition with congruent visual-auditory cues as well as a condition in which users' location perceptions of the target object are considered to be affected by the incongruent auditory cue. We found that the detection threshold of curvature gains was higher with incongruent visual-auditory cues than with congruent cues. These results show that incongruent multimodal cues in VR may have a promising application in the area of redirected walking.

We present ThermAirGlove (TAGlove), a pneumatic glove which provides thermal feedback for users, to support the haptic experience of grabbing objects of different temperatures and materials in virtual reality (VR). The system consists of a glove with five inflatable airbags on the fingers and the palm, two temperature chambers (one hot and one cold), and the closed-loop pneumatic thermal control system. Our technical experiments showed that the highest temperature-changing speed of TAGlove system was 2.75°C/s for cooling, and the pneumatic-control mechanism could generate the thermal cues of different materials (e.g., foam, glass, copper, etc.). The user-perception experiments showed that the TAGlove system could provide five distinct levels of thermal sensation (ranging from very cool to very warm). The user-perception experiments also showed that the TAGlove could support users’ material identification among foam, glass, and copper with the average accuracy of 87.2%, with no significant difference compared to perceiving the real physical objects. The user studies on VR experience showed that using TAGlove in immersive VR could significantly improve users’ experience of presence compared to the situations without any temperature or material simulation.

Several redirected walking techniques have been introduced and analyzed in recent years, while the main focus was on manipulations in horizontal directions, in particular, by means of curvature, rotation, and translation gains. However, less research has been conducted on the manipulation of vertical movements and its possible use as a redirection technique. Actually, vertical movements are fundamentally important, e.g., for remotely steering a drone using a virtual reality headset.In this paper, we explored vertical gains, a novel redirection technique, which enables us to purposefully manipulate the mapping of the user’s physical vertical movements to movements in the virtual space and the remote space. This approach allows natural and more active physical control of a real drone. To demonstrate the usability of vertical gains, we implemented a telepresence drone and vertical redirection techniques for stretching and crouching actions using common VR devices. We conducted two user studies to investigate the effective manipulation ranges and its usability: one study using a virtual environment (VE), and one using a camera stream from a telepresence drone. The results revealed that our technique could manipulate a users vertical movement without her/his noticing.

Studies have shown that the sense of body ownership towards virtual humanoid avatars with additional body parts can be successfully elicited when synchronous visuomotor and/or visuotactile feedback is given. In an interactive virtual reality (VR) application, however, it is difficult for users to uninterruptedly observe the added body parts, especially when they are attached to the backs of the avatars. Thus, the embodiment of such body parts needs to be achieved using limited synchronous visuomotor feedback. Commonly, it is specifically done by looking at a virtual mirror reflection of the avatar’s movement. However, the methodology of eliciting the sense of body ownership in such conditions remains to be studied.

Humans can estimate the shape of a wielded object through the illusory feeling of the mass properties of the object obtained using their hands. Even though the shape of hand-held objects influences immersion and realism in virtual reality (VR), it is difficult to design VR controllers for rendering desired shapes according to the perceptions derived from the illusory effects of mass properties and shape perception. We propose Transcalibur, which is a hand-held VR controller that can render a 2D shape by changing its mass properties on a 2D planar area. We built a computational perception model using a data-driven approach from the collected data pairs of mass properties and perceived shapes. This enables Transcalibur to easily and effectively provide convincing shape perception based on complex illusory effects. Our user study showed that the system succeeded in providing the perception of various desired shapes in a virtual environment.

We present PaCaPa, a handheld device that renders haptics on a user's palm when the user interacts with virtual objects using virtual tools such as a stick. PaCaPa is a cuboid device with two wings that open and close. As the user's stick makes contact with a virtual object, the wings open by a specific degree to dynamically change the pressure on the palm and fingers. The open angle of the wings is calculated from the angle between the virtual stick and hand direction. As the stick bites into the target object, a large force is generated. Our device enables three kinds of renderings: size, shape, and stiffness. We conducted user studies to evaluate the performance of our device. We also evaluated our device in two application scenarios. User feedback and qualitative ratings indicated that our device can make indirect interaction with handheld tools more realistic.

We have been studying augmented reality (AR)-based gustatory manipulation interfaces and previously proposed a gustatory manipulation interface using generative adversarial network (GAN)-based real time image-to-image translation. Unlike three-dimensional (3D) food model-based systems that only change the color or texture pattern of a particular type of food in an inflexible manner, our GAN-based system changes the appearance of food into multiple types of food in real time flexibly, dynamically, and interactively. In the present paper, we first describe in detail a user study on a vision-induced gustatory manipulation system using a 3D food model and report its successful experimental results. We then summarize identified problems of the 3D model-based system and describe implementation details of the GAN-based system. We finally report in detail the main user study in which we investigated the impact of the GAN-based system on gustatory sensations and food recognition when somen noodles were turned into ramen noodles or fried noodles, and steamed rice into curry and rice or fried rice. The experimental results revealed that our system successfully manipulates gustatory sensations to some extent and that the effectiveness seems to depend on the original and target types of food as well as the experience of each individual with the food.

Redirection makes it possible to walk around a vast virtual space in a limited real space while providing a natural walking sensation by applying a gain to the amount of movement in a real space. However, manipulating the walking path while keeping it and maintaining the naturalness of walking when turning at a corner cannot be achieved by the existing methods. To realize natural manipulation for turning at a corner, this study proposes novel “turning gains”, which refer to the increase in real and virtual turning degrees. The result of an experiment which aims to estimate the detection thresholds of turning gains indicated that when the turning radius is 0.5 m, discrimination is more difficult compared with the rotation gains (r = 0.0m).

We propose a novel gustatory manipulation interface which utilizes the cross-modal effect of vision on taste elicited with augmented reality (AR)-based real-time food appearance modulation using a generative adversarial network (GAN). Unlike existing systems which only change color or texture pattern of a particular type of food in an inflexible manner, our system changes the appearance of food into multiple types of food in real-time flexibly, dynamically and interactively in accordance with the deformation of the food that the user is actually eating by using GAN-based image-to-image translation. The experimental results reveal that our system successfully manipulates gustatory sensations to some extent and that the effectiveness depends on the original and target types of food as well as each user's food experience.

How does the representation of an embodied avatar influence the way in which one perceives the scale of a virtual environment? In virtual reality, it is common to embody avatars of various appearances, from abstract to realistic. It is known that changes in the realism of virtual hands affect the self-body perception, including body ownership. However, the influence of self-avatar realism on the perception of non-body objects has not been investigated. Considering the theory that the scale of the external environment is perceived relative to the size of one's body (body-based scaling), it can be hypothesized that the realism of an avatar affects not only body ownership but also the fidelity of the avatar with respect to our own body as a metric. Therefore, this study examines how avatar realism affects perceived object sizes as the size of the virtual hand changes. In the experiment, we manipulate the level of realism (realistic, iconic, and abstract) and size (veridical and enlarged) of the virtual hand and measure the perceived size of a cube. The results show that the size of the cube is perceived to be smaller when the virtual hand is enlarged compared to when it is veridical, indicating that the participants perceive the sizes of objects based on the size of the avatar, only in the case of a highly realistic hand. Our findings indicate that the more realistic the avatar, the stronger is the sense of embodiment including body ownership, which fosters scaling the size of objects using the size of the body as a fundamental metric. This provides evidence that self-avatar appearances affect how we perceive not only virtual bodies but also virtual spaces.

The excitement around computing technology in all aspects of life requires that we tackle fundamental issues of healthcare, leisure, labor, education, and food to create the society we want. The aim of this satellite event was to bring together a variety of different stakeholders, ranging from local food producers, chefs, designers, engineers, data scientists, and sensory scientists, to discuss the interwoven future of computing technology and food. This event was co-located with the AVI 2018 conference and supported by the ACM Future of Computing Academy (ACM-FCA). The event followed a co-creation approach that encourages conjoined creative and critical thinking that feeds into the formulation of a manifesto on the future of computing and food. We hope this will inspire future discussions on the transformative role of computing technology on food.

During group decision-making, a group with an imbalanced minority and majority typically suffers normative bias. This bias creates socio-emotional conflict and decreases group member consent and decision power. Thus, we propose a novel video-chat system, "Divided Presence," which aims to reduce this bias by equalizing the apparent number of majority and minority participants using a pseudo-population increase. Members of a discussion use computer graphics avatars on monitors instead of their actual video appearances during communication. The apparent number of discussion members increases when two avatars are assigned to an arbitrary minority member; the avatars then speak on behalf of him or her. We evaluate the system with a three-person consensus game, where a minority member is assigned two avatars. The results show that our system increases the degree of consent among the majority participants when they finally agree to the minority opinion.

PaCaPa is a handheld device that presents haptic stimuli on a user's palm when the user interacts with virtual objects using virtual tools. It is a box-shaped device and has two wings. This device can present the contact force and contact angle by opening and closing the wings based on the angle between the direction of the virtual tool and hand. By changing these haptic sensations on the palm and fingers, it enables the users to perceive different size, shape, and stiffness of virtual objects.

How does the representation of an embodied avatar influence the way in which a human perceives the scale of a virtual environment? It has been shown that the scale of the external environment is perceived relative to the size of one's body. However, the influence of avatar realism on the perceived scale has not been investigated, despite the fact that it is common to embody avatars of various representations, from iconic to realistic. This study examined how avatar realism would affect perceived graspable object sizes as the size of the avatar hand changes. In the experiment, we manipulated the realism (high, medium, and low) and size (veridical and enlarged) of the avatar hand, and measured the perceived size of a cube. The results showed that the size of the cube was perceived to be smaller when the avatar hand was enlarged for all degrees of realism of the hand. However, the enlargement of the avatar hand had a greater influence on the perceived cube size for the highly realistic avatar than for the medium-level and low-level realism conditions. This study shed new light on the importance of the avatar representation in a three-dimensional user interface field, in how it can affect the manner in which we perceive the scale of a virtual environment.

The contribution of this paper is to propose a novel Redirected Walking (RDW) technique that adopts manipulation gain in the roll direction. RDW is a technique that enables users to explore a large Virtual Environment (VE) while walking within a physically limited space by manipulating their virtual vision. Thus far, studies have determined the detection thresholds for translation, rotation, and curvature gains, but the thresholds are limited in the yaw direction. In contrast, in other research areas, movements in the yaw and roll directions have sometimes been addressed at the same time. In the present study, we investigated the detection threshold of inclination gain in the roll direction. We applied an inclination gain that was increased gradually while the participants walked 3 meters straight ahead. The results showed that users can detect an inclination gain of 1.93° on the left hand and 1.39° on the right hand.

A redirected walking (RDW) techniques enable users to walk around infinite virtual environments (VEs) in a finite physical space. In previous studies on RDW, many researchers have discussed manipulations in the yaw direction, but few have tackled with redirection in pitch and roll directions. We propose a novel VR system, which realizes pitch and roll redirections and allows users to experience walking on the 3D model of Mobius Strip in the VE.

This paper presents a novel interactive system that provides users with virtual reality (VR) experiences, wherein users feel as if they are ascending/descending stairs through passive haptic feedback. The passive haptic stimuli are provided by small bumps under the feet of users; these stimuli are provided to represent the edges of the stairs in the virtual environment. The visual stimuli of the stairs and shoes, provided by head-mounted displays, evoke a visuo-haptic interaction that modifies a user's perception of the floor shape. Our system enables users to experience all types of stairs, such as half-turn and spiral stairs, in a VR setting. We conducted a preliminary user study and two experiments to evaluate the proposed technique. The preliminary user study investigated the effectiveness of the basic idea associated with the proposed technique for the case of a user ascending stairs. The results demonstrated that the passive haptic feedback produced by the small bumps enhanced the user's feeling of presence and sense of ascending. We subsequently performed an experiment to investigate an improved viewpoint manipulation method and the interaction of the manipulation and haptics for both the ascending and descending cases. The experimental results demonstrated that the participants had a feeling of presence and felt a steep stair gradient under the condition of haptic feedback and viewpoint manipulation based on the characteristics of actual stair walking data. However, these results also indicated that the proposed system may not be as effective in providing a sense of descending stairs without an optimization of the haptic stimuli. We then redesigned the shape of the small bumps, and evaluated the design in a second experiment. The results indicated that the best shape to present haptic stimuli is a right triangle cross section in both the ascending and descending cases. Although it is necessary to install small protrusions in the determined direction, by using this optimized shape the users feeling of presence of the stairs and the sensation of walking up and down was enhanced.

"Mirroring" refers to the unconscious mimicry of another person's behaviors, such as their facial expressions. Mirroring has many positive effects, such as enhancing closeness and improving the flow of a conversation, which enriches the quality of communication. Our study set out to devise a means of evoking these positive effects in a video chat without any conscious effort of participants. We constructed a videophone system, called FaceShare, which can deform the user's face into a smile in response to their partner's smiling. That is, our system generates mirroring by producing a pseudo-smile through image processing. We conducted an experiment in which pairs of participants had brief conversations via FaceShare. The results implied that mirroring using the pseudo-smile lets the mimicker, whose face is deformed according to the expressions of their partner, feel a closeness, and improves the flow of the conversation for both the mimicker and the mimickee, who sees the mimicker's deformed face.

As intelligent agents learn to behave increasingly autonomously and simulate a high level of intelligence, human interaction with them will be increasingly unpredictable. Would you accept an unexpected and sometimes irrational but actually correct recommendation by an agent you trust? We performed two experiments in which participants played a game. In this game, the participants chose a path by referring to a recommendation from the agent in one of two experimental conditions:the correct or the faulty condition. After interactions with the agent, the participants received an unexpected recommendation by the agent. The results showed that, while the trust measured by a questionnaire in the correct condition was higher than that in the faulty condition, there was no significant difference in the number of people who accepted the recommendation. Furthermore, the trust in the agent made decision time significantly longer when the recommendation was not rational.

To effectively utilize gaze in one-to-many communication, robots must direct and avert their gaze with appropriate timing. However, few studies have focused on this aspect of communication because of the difficulty in detecting the position of many gaze recipients and directing appropriate gazes to each. This paper proposes a system of robotic eyes that can send averted or direct gazes simultaneously to multiple recipients by switching between convex and hollow eye shapes, which we call Transgazer.

Redirected walking allows users to explore a large virtual environment while there is a limitation of the room size. Previous works tried to present users straight path in a virtual environment while they walked on a curved path in reality. We expand a previous technique to present users a various curved path in a virtual environment while they walked on a particular curved path or a straight path with/without haptics. Furthermore, we propose a novel estimation methodology to quantify walking paths which user has thought he walked in reality. The data from our experiment shows that users feel walking a various curved path in VR as same as one-to-one mapping condition.

Humans are capable of haptically perceiving the shape of an object by simply wielding it, even without seeing it. On the other hand, typical hand-held controllers for virtual reality (VR) applications are pre-designed for general applications, and thus not capable of providing appropriate haptic shape perception when wielding specific virtual objects. Contradiction between haptic and visual shape perception causes a lack of immersion and leads to inappropriate object handling in VR. To solve this problem, we propose a novel method for designing hand-held VR controllers which illusorily represent haptic equivalent of visual shape in VR. In ecological psychology, it has been suggested that the perceived shape can be modeled using the limited mass properties of wielded objects. Based on this suggestion, we built a shape perception model using a data-driven approach; we aggregated data of perceived shapes against various hand-held VR controllers with different mass properties, and derived the model using regression techniques. We implemented a design system which enables automatic design of hand-held VR controllers whose actual shapes are smaller than target shapes while maintaining their haptic shape perception. We verified that controllers designed with our system can present aimed shape perception irrespective of their actual shapes.

Agents like human, such as humanoid robots or avatars can be felt as if they have and communicate and communicate due to manipulation of the bodily information. Meanwhile, as in the case of Internet bot, it is still difficult to communiate the emotion described as text, let alone empathizing due to degradation of information online. The current study proposes a method for experiencing emotion on the Internet by reproducing a mechanism of evoking emotion. This method evokes a number of emotions described on the Web, by changing of self-physiological perception with sensory stimuli. To investigate the feasibility of our method, we made a system named "Communious Mouse." This system rewrites the perception of self-skin temperature and pulse in a palm by presenting vibration and thermal stimulation through a mouse device for evoking emotion. The current paper discusses the feasibility of our method based on the obtained feedbacks through an exhibition of the system.

We propose a navigation interface, which enables users to control locomotion and the speed intuitively in virtual environments constructed with spherical images. In these virtual environments, users can only pass through lines where the camera has passed through during recording. Therefore, it is necessary to inform users of the accessible directions. Furthermore, velocity control is needed for close exploration in virtual environments. Therefore, we propose the Motive Compass input interface, which intuitively shows users the accessible directions and enables them to control the velocity of locomotion. We conducted a large-scale demonstration experiment in a real exhibition and showed that our interface more effectively presents accessible directions and enables users to control their velocities.

Lately, mobile augmented reality (AR) has become very popular and is used for many commercial and product promotional activities. However, in almost all mobile AR applications, the user only views annotated information or the preset motion of the virtual object in an AR environment and is unable to interact with the virtual objects as if he/she were interacting with real objects in the real environment. In this paper, in an attempt to realize enhanced intuitive and realistic object manipulation in the mobile AR environment, we propose an integrated view-input AR interaction method, which integrates user device manipulation and virtual object manipulation. The method enables the user to hold a 3D virtual object by touching the displayed object on the 2D touch screen of a mobile device, and to move and rotate the object by moving and rotating the mobile device while viewing the held object by way of the 2D screen of the mobile device. Based on this concept, we implemented three types of integrated methods, namely the Rod, Center, and Touch methods, and conducted a user study to investigate the baseline performance metrics of the proposed method on an AR object manipulation task. The Rod method achieved the highest success rate (91%). Participants' feedback indicated that this is because the Rod method is the most natural, and evoked a fixed mental model that is conceivable in the real environment. These results indicated that visualizing the manipulation point on the screen and restricting the user's interactivity with virtual objects from the user's position of view based on a conceivable mental model would be able to aid the user to achieve precise manipulation.

This paper proposes a high-definition digital display case for manipulating a virtual exhibit that has linking mechanisms. This technique enhances the understanding of dynamic exhibits. It is difficult to construct interactive contents of dynamic virtual exhibits, because measuring the mechanism invokes the risk of an exhibit's deterioration, and it takes tremendous efforts to create a fine spun computer graphics (CG) model for mechanisms. Therefore, we propose an image-based interaction method that uses image-based rendering to construct interactive contents for dynamic virtual exhibits using the interpolation between exhibit pictures with a number of deformational conditions and viewpoints. Using this method, we construct a high-definition digital showcase and exhibit the interactive content at a museum to evaluate the availability of our system.

In this paper we developed the free-viewpoint curved surface shape display, using the effect of visuo-haptic interaction. In our research, we aim to realize the simple mechanic visuo-haptic system with which we can touch various objects with our real hands. We proposed the free-viewpoint shape display system in which users can touch a virtual object with various shape through the tablet device, while walking around the object. The system figured the difference of shape between the real object and the virtual one from the current viewpoint, and calculates the amount of displacement of the touching hand image to fit to the virtual object, in real time. This modification of the movement of a hand image evokes the effect of the visuo-haptic interaction, and enable users to feel touching various shapes from various viewpoints, although actually users touch a physically static object.

In this study, we developed a system for mobile devices designed to provide a virtual experience of past scenery depicted in old photographs by superimposing them on landscapes in video see-through frames. A user is asked to capture a photograph of a landscape to be used as a keyframe and enter correspondence points between the new and old photos. The old photograph is deformed by projective transform with homography computed based on correspondence points. We then superimpose of the old photograph onto video see-through frames of current landscape. To achieve real-time and robust superimposition on mobile devices, both motion-sensor-based pose information and camera-image-keypoint-tracking-based pose information is used for device's camera pose tracking.

The main objective of this study is to create a method that can affect flavor perception, without changing the food itself, by applying thermal sensations to the skin around the nose to simulate skin temperature changes associated with pleasant and unpleasant feelings. Recent physiological research has demonstrated that skin temperature changes affect flavor perception. Based on the James--Lange theory, which claims that changes in bodily responses unconsciously evoke emotions and change our perception related to an emotional state, we hypothesized that flavor perception can be modified by affecting the skin temperature properly. To test this theory, we developed an "Affecting Tumbler," which presents thermal sensations to the skin around the nose for simulating the skin's temperature response during drinking. Our user study suggested that flavor richness and aftertaste strength were significantly improved by heating up the skin in the nasal region. These results indicate that flavor perception can be controlled with the proposed method.

This study develops a method for improving creativity by changing the facial appearance of people during video conferences. We focus on enhancing creativity during human interactions, especially cooperative work situations such as video conferences. Psychological studies have revealed that the facial appearance of people can affect creativity, i.e., "emotion can affect creativity" and "the appearance of others can affect the emotions." Based on the knowledge, we hypothesized that the media technology which modifies the facial appearance of people appropriately can enhance our creativity during cooperative work. Therefore, we develop a method to affect creativity by changing the facial appearance in real-time, such as facial expressions and facial resemblance. We test the effectiveness of this method by building a system for collaborative video conferences. The user study show that the proposed method could enhance creativity.

In this paper, we proposed a visuo-haptic system for displaying various shapes which have curve, edge, and inclined surfaces, using a simple transmutative physical device and the effect of visuo-haptic interaction. We aim to construct a perception-based shape display system to provide users with the sensation of touching virtual objects of varying shapes using only a simple mechanism. We have confirmed that the perception of each primitive shape such as curvature and angle could be modified by displacing a user's hand image on the monitor as if s/he were touching the visual shape while actually touching another shape. In this study, we constructed the method to merge these findings for displaying more various shapes, including angular ones. We built a transmutative device, which the user touches. The device does not undergo significant transformation, but its surface can be slightly bumped in and out, and displayed various shapes with various angles, length and curvature. The results of experimental trials confirmed that our method for displaying each primitive shape can also worked as designed when we combine these findings to display more complex objects using this device which transforms slightly.

The main contribution of this paper is to develop a method for alleviating fatigue during handling medium-weight objects and augmenting our endurance by affecting our weight perception with augmented reality technology. To assist people to lift medium-weight objects without a complex structure or various costs, we focus on the phenomenon that our weight perception during handling objects is affected by visual properties. Our hypothesis is that this illusionary effect in weight perception can be applied to reduce fatigue while handling medium-weight objects without mechatronics-based physical assistance. In this paper, we propose an augmented reality system that changes the brightness value of an object in order to reduce fatigue while handling the object. We conducted two fundamental experiments to investigate the effectiveness of the proposed system. Our results suggested that the system eliminates the need to use excess energy for handling objects and reduces fatigue during the handling task.

This paper reports a reliving video experience system using augmented reality techniques. This system overlays a scene from video material onto the real world and allows users to experience how the camera operator captured the scene by inducing them to move as in the same way as the camera operator did. We propose novel interaction techniques that balance the competing goals of inducing users to move in the same way as the camera operator and allowing them to feel that they are observing freely. The proposed methods implement three induction techniques to induce the user to avoid unintentional movements, and to start moving the device and rotate it appropriately. We implemented this system in an exhibition system that relives the video of old railways and exhibited it at the railway museum for two weeks. The analysis results of users' operational logs and questionnaires suggest that the proposed system and interaction techniques are effective for reliving video experiences in the real world.

The main contribution of this paper is to realize a method for modifying perception of satiety and controlling nutritional intake by changing the apparent size of food with augmented reality. As a possible method for decreasing rates of obesity, we focused on controlling food intake implicitly without any effort.

In our research, we aim to construct the visuo-haptic system which can give users a sense as if they were touching on virtual objects with variety of shapes, using pseudo-haptiec effects. In this paper, we focus on modifying the identification of a position of edges on a object when touching it with a pointing finger, by displacing the visual representation of the user's hand in order to construct a novel visuo-haptic system. We compose a video see-through system, which enables us to change the perception of the shape of an object a user is visually touching, by displacing the visual representation of the user's hand as if s/he was touching the visual shape, when in actuality s/he is touching another shape. We had experiments and showed participants perceived positions of edges that was the same as the one they were visually touching, even though the positions of edges they were actually touching was different. These results prove that the perceived positions of edges could be modified, and even if the ratio of the successive distance between edges is 1 : 1, we can modify the perception of this ratio from 3 : 2 to 2 : 3.

The main contribution of this paper is to realize computer generated augmented flavors and establish a method to integrate gustatory information into computer human interactions. There are several reasons for the scarcity of research on gustatory information. One reason is that taste sensations are affected by a number of factors, such as vision, olfaction and memories. This produces a complex cognition mechanism for a user's gustatory sensation, and makes it difficult to build up a gustatory display which produces a specific taste on demand.

This study tries to increase the effects of odors by controlling the odor emissions. We developed an olfactory feedback system to maintain user's concentration effectively based on biological information. The system presumes the level of user's concentration based on an electrocardiogram analysis. The analysis is carried out using variance of R-R intervals and frequency spectra. According to the presumed concentration level, the system presents an odor with an awakening effect to maintain user's concentration. We performed experiments in which subjects did simple tasks with this system, changing the method of controlling odor emissions. We compared the results of each method and evaluated the effectiveness of our olfactory feedback system and the methods of controlling odor emissions. The comparison of the results showed that presenting an odor when a user loses his/her concentration is effective to decrease errors of addition tasks.

In this paper, we propose a pseudo-gustatory display based on the cross-modal interactions that underlie the perception of flavor. Although several studies on visual, auditory, haptic, and olfactory displays have been conducted, gustatory displays have seldom been studied. This scarcity has been attributed to the fact that synthesizing arbitrary taste from basic tastants is difficult. On the other hand, it has been noted that the perception of taste is influenced by visual cues, auditory cues, smell, the trigeminal system, and touch. In our research, we aim at utilizing this influence between modalities for realizing a "pseudo-gustatory" system that enables the user to experience various tastes without changing the chemical composition of foods. Based on this concept, we built a "Meta Cookie" system to change the perceived taste of a cookie by overlaying visual and olfactory information onto a real cookie which an AR marker pattern. We performed an experiment that investigates how people experience the flavor of a plain cookie by using our system. The result suggests that our system can change the perceived taste based on the effect of the cross-modal interaction of vision, olfaction and gustation.

In our research demonstration, we show a "MetaCookie+" that enables the user to experience various tastes without changing the chemical composition of foods by using the influence between modalities. It is a pseudo-gustatory display by combining the Edible Marker system which can detect the state [number/shape/6-DOF coordinate] of each piece of bitten or divided food in real time, and the "Pseudo-gustation" method to change the perceived taste of food by changing its appearance and scent.

In this research study, we propose a divided planar-object detection method for augmented reality(AR) applications. There are mainly two types of camera-registration methods for AR applications: marker-based methods, and natural-feature-based methods. In addition, the latter methods are classified into visual SLAM and object detection methods. With respect to object detection methods, particularly for planar objects such as paper, methods for dealing with bending, folding, and occlusion are proposed. However, the division of objects has not been studied. Once an object is divided, a conventional object detection method cannot identify each of the pieces because the feature points of only a single piece are recognized as the target object, and the other feature points are regarded as outliers. The proposed system prepares a database of the target object's natural features, and applies progressive sample consensus(PROSAC), which is a robust estimation method, for iterative homography calculation to achieve the multiple planar-object detection. Moreover, the proposed method can detect shapes of pieces by simultaneously using an occlusion detection method. We demonstrate that it is possible to interact with an arbitrarily divided planar object in real time by our method to implement some AR applications.

In this research study, we propose a divided planar-object detection method for augmented reality(AR) applications. The proposed system prepares a database of the target object's natural features, and applies progressive sample consensus (PROSAC), which is a robust estimation method, for iterative homography calculation to achieve the multiple planar-object detection. Moreover, the proposed method can detect shapes of pieces by simultaneously using an occlusion detection method. We demonstrate that it is possible to interact with an arbitrarily divided planar object in real time by our method to implement some AR applications.

In this paper, we proposed the Digital Diorama system to convey background information vividly. The system superimposes computer generated diorama scene reconstructed from related image/video materials on real exhibits. In order to switch and superimpose real exhibits and past photos seamlessly, we implement a matching system for estimating the camera position where photos are taken. By applying this subsystem to 26 past photos about the steam locomotive exhibit, we succeeded in estimating their camera position. Thus, we implement and install a prototype system at estimated position to superimposing virtual scene and real exhibit in the Railway Museum.

Olfactory displays which exist now can only present the set of scents which was prepared beforehand because a set of "primary odors" has not been found. In this paper, we focus on development of an olfactory display using cross modality which can represent more patterns of scents than the patterns of scents prepared. Due to cross modal effect between vision and olfaction, human tends to feel not olfactory but visual stimulation as scents. First, we asked subjects to smell various aroma chemicals and evaluate their similarity. Based on the data of similarity among aromas, we built a map of smell distance. Next, we selected a few aroma chemicals from the smell distance map and implemented a visual and olfactory display. Then we conducted an assessment experiment of the display. We presented various pictures and the selected aroma chemicals to subjects and asked them what kind of scent they smelled like. In this experiment, we succeed in giving subjects feeling more patterns of scents than the number of selected aroma chemicals. Also, we succeed in making not olfactory stimulation by aroma chemicals but visual stimulation by pictures as scents. In particular, we find that the visual effect on olfactory sensation is more strong when the distance between the picture and the aroma chemicals is close than when the distance is far.