Users are more likely to experience visually induced motion sickness (VIMS) during passive motion in virtual reality (VR), particularly in passive virtual driving scenarios. To solve this challenge, we propose a Multi-sensory Simulation of Wind Sensation (MSSWS) method to alleviate VIMS symptoms by enhancing the user’s sense of embodiment (SoE). The method utilizes low-fidelity airflow simulation to align vestibular perception with visual, auditory, and tactile cues. Then, we developed an interactive wind simulation helmet that can generate low-fidelity airflow along four directional axes around the user’s head, and implemented a passive virtual motorcycle riding environment with synchronized visual and auditory feedback matching the airflow patterns. A user experiment was conducted to systematically evaluate the effectiveness of MSSWS in enhancing SoE and reducing VIMS under various conditions, including helmet activation states, speed variations, and movement directions, with additional validation conducted in a Cave Automatic Virtual Environment (CAVE). Experimental results show that MSSWS significantly enhances sense of presence and SoE while substantially reducing VIMS during passive navigation. Notably, MSSWS shows significantly greater effects on VIMS in high-risk conditions characterized by rapid speed changes and multi-directional movement.

For individuals in situations where their hands are occupied, no-handed touch interaction represents a significant yet underexplored area. In this paper, we investigate touch performance on screens using body parts (the nose, tongue, chin, elbow, and toe) rather than the hand. Experiments were conducted with 37 participants to analyze their touch interaction, where they selected targets using various body parts under different conditions. We employed the Dual-Gaussian model and Ternary-Gaussian model to describe the touch endpoints of these body parts. Based on the statistical criterion derived from the models, we proposed a no-handed assistive technology, which achieved up to a 24.4% improvement in tapping speed and a 58.47% increase in tapping accuracy. Additionally, this paper provides HCI developers with design guidelines for optimizing no-handed touch interaction interfaces.

Virtual reality (VR), particularly through multi-user VR systems, enhances collaboration and immersion. However, multi-team VR systems (MTVR), which allow several teams to interact, either by cooperating or competing, are less studied. Drawing inspiration from concepts in game theory, specifically the positive-sum game (PSG) and zero-sum game (ZSG), this study investigated the impact of inter-team competition modes at the team level and haptic feedback at the individual level on team flow and individual flow for the first time. An experimental MTVR that supports inter-team competition in a two-person vs two-person pattern was implemented first. Then, a$2 \times 2$within-subject experiment was conducted to examine the effects of different inter-team competition modes (PSG/ZSG) and haptic feedback (on/off). The results based on 188 participants indicate that the PSG mode leads to significantly higher levels of collective ambition and improved team relationships compared to the ZSG mode. Furthermore, providing haptic feedback can significantly enhance awareness of shared and personal task goals, resulting in more cautious performance during teamwork.

Moving target selection in multimedia interactive systems faces unprecedented challenges as users increasingly interact across diverse, dynamic contexts-from live streaming in moving vehicles to VR gaming in varying environments. Existing approaches rely on probabilistic models that relate endpoint distribution to target properties (size, speed). However, these methods require substantial training data for each new context and lack transferability across scenarios, limiting their practical deployment in diverse multimedia environments where rich multimodal contextual information is readily available. This paper introduces MAGNeT (Multimodal Adaptive Gaussian Networks), which addresses these problems by combining classical statistical modeling with context-aware multimodal method. MAGNeT dynamically fuses pre-fitted Ternary-Gaussian models from various scenarios based on real-time contextual cues, enabling effective adaptation with minimal training data while preserving model interpretability. We take experiments on self-constructed 2D and 3D moving target selection datasets under in-vehicle vibration conditions. Extensive experiments demonstrate that MAGNeT achieves lower error rates with few-shot samples, by applying context-aware fusion of Gaussian experts from multi-factor conditions.

Stereotyped movements play a crucial role in diagnosing Autism Spectrum Disorder (ASD). However, recognizing them poses challenges, due to limited data availability and the movements' specificity and varying duration. To support in-depth analysis of ASD children's movements, we constructed the ACSA653 dataset, comprising 653 videos across six classes of stereotyped movements. This dataset surpasses existing ones in both scale and category. To improve the recognition of stereotyped movements, we propose APMFNet, a model that integrates three modules: Visual Motion Learning (VML), Skeleton Relation Mining (SRM), and Multi-channel Fusion (MF). The VML module focuses on extracting spatial and motion information from RGB and optical-flow sequences. The SRM module effectively mines essential motion patterns associated with stereotyped movements through cross-modal graph. The MF module fuses multi-modal information through cross-modality attention to facilitate decision-making. Tested on ACSA653, APMFNet outperforms current state-of-the-art methods, suggesting its potential to identify stable patterns of stereotyped movements in children with ASD.

To relieve perceived pain in patients undergoing colonoscopy, we developed an endoscopist-patient co-participative VR tool (EPC-VR) based on A Neurocognitive Model of Attention to Pain. It allows the patient to play a VR game actively and supports the endoscopist in triggering a distraction mechanism to divert the patient's attention away from the medical procedure. We performed a comparative clinical study with 40 patients. Patients' perception of pain and affective responses were evaluated, and the results support the effectiveness of EPC-VR: active VR playing with endoscopists' participation can help relieve the perceived pain and scare of patients undergoing colonoscopy. Finally, 87.5% of patients opt to use the VR application in the next colonoscopy.

Human cognition and behavior can be unconsciously affected by personal avatars in the virtual world, a phenomenon known as the Proteus Effect. When using first-person non-human avatars, the characteristics of virtual hands may also induce relevant cognitive and even behavioral patterns in real human hands. Therefore, evaluating human-avatar gesture consistency may be a potentially effective method for objectively assessing the Proteus Effect when using non-human avatars. To explore this question, we first created human and non-human avatars, including three animals. Then, we constructed a dataset of hand gestures and trained a model to dynamically recognize real-hand gestures that were consistent with corresponding avatar hands. Next, we designed a virtual reality experimental task involving grasping objects with intuitive gestures and performed a 2 (avatar type: human/non-human) $* 2$ (virtual hand: presence/absence of spontaneous movement) within-subject experiment to examine the effects of avatar characteristics on self-illusion and human-avatar gesture consistency. The results showed that participants performed a significantly larger percentage of gestures that were consistent with their currently used avatars. Additionally, participants did experience self-illusion when using non-human avatars, although the levels were significantly lower than those when using human avatars. Therefore, self-illusion may serve as a perceptual antecedent of the Proteus Effect, even with non-human avatars, inadvertently altering the behavioral gestures of their real hands. In conclusion, detecting human-avatar gesture consistency can help evaluate the Proteus Effect.

Target selection is a crucial task in augmented reality (AR). Recent evidence suggests that user motion can significantly influence target selection. However, no systematic research has been conducted on target selection within varied intensity user motions and AR settings. This paper was carried out to investigate the effects of four user motions (i.e., standing, walking, running, and jumping) and two viewing modes (i.e., viewpoint-dependent and viewpoint-independent) on user performance of target selection in AR. Two typical selection techniques (i.e., virtual hand and ray-casting) were utilized for short-range and long-range selection tasks, respectively. Our results indicate that the target selection performance decreased as the intensity of user motion increased, and users demonstrated better performance in the viewpoint-independent mode than in the viewpoint-dependent mode. We also observed that users took a longer amount of time to select targets when using the ray-casting technique than the virtual hand technique. We conclude with a set of design guidelines to improve the AR target selection performance of users while in motion.

Abstract Current VR/AR system development studios lack a toolkit that can automatically generate hierarchical tile menu layouts and menu prototypes for VR/AR devices without the need for user programming. This paper proposes a solution in the form of a toolkit that automatically generates a hierarchy tile menu layout using a modified circular treemap algorithm. Users can interactively arrange and resize tiles to create various layouts according to their preferences or needs, utilizing a circle packer method. The toolkit further automatically generates a VR/AR menu prototype based on the resulting layout. Notably, reprogramming is not necessary each time the hierarchy is modified or the menu layout is redesigned. The user test results demonstrate that the proposed toolkit simplifies the creation of hierarchical tile menu layouts, enhances user efficiency in the design process, and enables users to flexibly create hierarchical tile menu prototypes according to their design ideas.

Exploring effective approaches to enhance face-to-face interactions and interpersonal relationships is an important topic in the applications of affective computing. According to the co-actualization model, we propose a face-to-face co-participation serious game for relationship enhancement, with a focus on battling COVID-19. Moreover, a prototype system is developed using an immersive virtual environment and a low-cost brain-computer interface. Through this system, a dynamic flow experience enhancement tool is utilized to involve partners in the cooperative task. To evaluate the system performance, two studies are conducted with schoolmates as participants. Study 1 compares the cooperative and competitive modes, and demonstrates that the former elicited higher level of decision-making challenge and affections, which are beneficial for forming relationships. Study 2 further examines the effect of the dynamic flow enhancement tool in the cooperative task and the results show its effectiveness in promoting flow experience, perceived closeness, and intimacy in relationships. Given this short-term participation, participants felt a greater sense of closeness and intimacy than they had before the test. In conclusion, our proposed system is effective in enhancing schoolmate relationships.

Children diagnosed with Autism Spectrum Disorder (ASD) often exhibit motor disorders. Dance Movement Therapy (DMT) has shown great potential for improving the motor control ability of children with ASD. However, traditional DMT methods often lack vividness and are difficult to implement effectively. To address this issue, we propose a Mixed Reality DMT approach, utilizing interactive virtual agents. This approach offers immersive training content and multi-sensory feedback. To improve the training performance of children with ASD, we introduce a novel training paradigm featuring a self-guided mode. This paradigm enables the rapid creation of a virtual twin agent of the child with ASD using a single photo to embody oneself, which can then guide oneself during training. We conducted an experiment with the participation of 24 children diagnosed with ASD (or ASD propensity), recording their training performance under various experimental conditions. Through expert rating, behavior coding of training sessions, and statistical analysis, our findings revealed that the use of the twin agent for self-guidance resulted in noticeable improvements in the training performance of children with ASD. These improvements were particularly evident in terms of enhancing movement quality and refining overall target-related responses. Our study holds clinical potential in the field of medical treatment and rehabilitation for children with ASD.

The COVID-19 has led to home quarantine in global scale, which increased people’s need for indoor fitness activities in order to relieve stress/anxiety and learn knowledge well during the prevention. In this article, an augmented reality (AR) indoor exergame is developed, mainly a kind of moving target hitting task with Lomachenkos headwear speedball as the prototype. Then, a comparative experiment is performed to test the effect of the exergame in relieving anxiety, as well as the differences between AR and virtual reality modes (using HTC Vive Pro) on user experience (flow experience and motion sickness) and interaction efficiency. Results from 28 participants show that the anxiety level of participants significantly reduced after using the exergame. Although the difference on flow experience is not significant, there is a marginally significant difference on nausea level, a dimension of motion sickness. Moreover, interaction efficiency between the two modes is significantly different. In the task of moving target hitting, AR mode performed better in avoiding motion sickness and improving interaction performance.

The curvature gain of redirected walking method enables users to explore virtual spaces that are larger than real spaces. The estimation of human sensitivity for curvature gain is important for redirected walking. Herein, we conduct two experiments under different path conditions. By adopting the psychophysical “method of limits” to be a new approach, the first experiment re-estimates the sensitivity for a curved path and further proves that people are more sensitive for right-curved paths than left-curved paths. The second experiment investigates the characteristics of preorder paths on the sensitivity to postorder paths, and finds that the existing of preorder paths significantly increases the sensitivity for the postorder path. Moreover, we find an interesting moderation effect on the direction consistency of preorder and postorder paths: the larger curved preorder path can significantly decrease the sensitivity for postorder path when they have the same direction. If their directions diverge, the effect of preorder path is not significant.

Modeling error rates in moving target selection is critical in guiding the design and improving user performance in the user interface with dynamic contents. Despite the high accuracy of existing models in predicting error rates for various inputs, they need to fit a large number of samples for a specific interaction scenario. This paper presents a scenario adaptive model that can quickly learn the characterizes of endpoints from small data in a specific scenario, and then accurately predict the error rates in this scenario. We report on two studies evaluating the adaptability of the model in specific users (children and the older user) and pointing postures (walking and one-handed) with a smartphone. Significant improvements in error rates prediction of our model were observed compared to the previous non-adaptive one. We conclude with the findings from our results for insights into future interface design.

The flow experience-performance link is commonly found weak in virtual environments (VEs). The weak association model (WAM) suggests that distraction caused by disjointed features may be associated with the weak association. People characterized by field independent (FI) or field dependent (FD) cognitive style have different abilities in handling sustained attention, thus they may perform differently in the flow-performance link. To explore the role of the field dependence-independence (FDI) construct on the flow-performance link in virtual reality (VR), we developed a VR experimental environment, based on which two empirical studies were performed. Study 1 revealed FD individuals have higher dispersion degree of fixations and showed a weaker flow-performance link. Next, we provide visual cues that utilize distractors to achieve more task-oriented attention. Study 2 found it helps strengthen the task performance, as well as the flow-performance link of FD individuals without increasing distraction. This paper helps draw conclusions on the effects of human diversity on the flow-performance link in VEs and found ways to design a VR system according to individual characteristics.

Weak association model indicates that distraction caused by the disjunction between the primary task and interactive artifacts may be a key factor directly leading to weak association between flow and task performance in virtual reality (VR) activities. To test the idea, this paper proposed a VR visual search paradigm based on which we constructed a VR oceanic treasure hunting system. Experiment 1 proved that distraction caused by the incongruence was indeed a direct antecedent of weak association. Next, we slightly adjusted the system by providing visual cues to achieve task-oriented selective attention. Experiment 2 found this helped enhance the task performance without damaging flow experience.

In this paper, we developed an agent-based virtual interview training system which can help college students with high shyness level to improve interview skills and reduce their anxiety by themselves before they take a real interview. The system includes three main contents: three virtual agents with different types of personality, three kinds of interview training contents and a multidimensional evaluation method, so that it is able to meet common demands of preparing for interviews. User study indicates the system can help shy college students cope with interview anxiety and improve their interview training performance effectively.

This paper presents a mixed reality storytelling system, which takes handicrafts as tangible interaction tools. Via the system, users can learn handicraft and then use the handicraft pieces to design, create and tell stories with HoloLens iteratively. In order to overcome the limitations of HoloLens gesture interaction, the system uses hand tracking with Kinect to implement a touch-like effect on the desktop. User study shows that our system has good usability, and it is welcomed by users. In addition, it can stimulate users interest in handicraft and storytelling, and even promote parent-child interaction effectively.

This paper presents a toolkit called Rotbav for easily constructing mixed reality (MR) apps that can be experienced in real time in large indoor physical space via HoloLens. It resolves the problem that existing MR devices, e.g. HoloLens, are unable to scan and model an entire large scene with several rooms at once. We introduce a custom data structure called VorPa, based on the Voronoi diagram, to implement path editing, accelerated rendering and location effectively. Our experiments and applications show that the toolkit is convenient and easy to use for constructing MR apps targeting large indoor physical spaces, in which users can roam in real time.

Many studies conducted in non-virtual activities have shown that flow significantly influences performance, yet studies in virtual activities often reveal only a weak association. This paper begins by building a theoretical explanatory model, and then conducts 3 empirical studies to explore this question. Study 1 exams the mechanism of weak association in two virtual activities. Study 2 tests the effectiveness of a potential approach to strengthen this association. In Study 3 we applied our proposed model and design approach to optimize a VR tennis game. Results show that the influence of flow on performance was not significant in those virtual activities where the primary task and the operation of interactive artifacts were less congruent such that the artifacts can lead to flow experience that is independently of the primary task. Our research offers a theoretical and empirical basis on how to optimize virtual environment design and maximize positive effect of the flow experience.

Stereoscopic display is a standard display mode for virtual reality environments. Typical 3D projection provides only a single stereoscopic video stream; thus co-located users cannot correctly perceive the virtual scene based on their own position and view. Several works devoted to developing multi-user stereoscopic display, but the number of users is very limited or the technical implementation is complicated. In this paper we put forward two flexible and simple projection-based multi-user stereoscopic display systems. The first one, named TPA, is based on a triple-projector array and provides a 120Hz active stereo for three users. Two TPAs can be combined to form a six-user system. The second one, named DPA, is a dual-projector and easy-implemented system providing individual stereoscopic video stream for two to six users. Finally, a co-located multi-user virtual fireman simulation training system and a virtual tennis simulation system were created to verify the effectiveness of our systems.

In this paper, we present a method of view-dependent stereoscopic projection on curved screen. It allows the user to walk around with the correct perspective view of the virtual scene consistent with his/her location. To solve the problem of distortion and drift of virtual objects when projecting the view-dependent scene images on curved screen, we operate a dynamic parallax adjustment of the stereoscopic images according to viewpoints. User evaluation shows that our proposed approaches are effective on improving visual experience.

With the fast development of virtual reality games, one of the key research questions is how players may express their creativity and participate in the process of game design. In this paper, we present a new game genre which combines user-controlled game design in physical space with game play in virtual space on a mobile device. The new system supports authoring by anyone, creating virtual reality games that can be easily modified or developed for physical space, and be used anywhere by novice end-users without any knowledge of tracking technology. We present the design and implementation of the system, as well as a user experiment. Findings illustrate that the proposed system promotes participation and provides a richer, more interactive and engaging experience.