The development of virtual reality (VR) technology has allowed virtual museums to enrich experiences of personal memories. In this work, we first conducted a survey study to understand the way people use to record, store, and share their digital memories. Informed by the results, we present MemoryVR, a personal virtual museum system designed to preserve and share digital memories. It allows individuals to organize and share their own memories in a virtual museum environment and to visit the memory spaces of others in an immersive way. We invited participants to experience MemoryVR and analyzed their behavior and expectations. The evaluation results showed that users perceived excellent pragmatic and hedonic qualities of MemoryVR. Participants found their experiences of memories in personal virtual museums to be fulfilling. Additionally, we gathered suggestions from participants about MemoryVR, providing design recommendations for customized personal virtual museums.

Cross Reality (CR) is a new emerging field based on the current developments in Mixed Reality hardware, especially supported by the broad market penetration of video-based see-through Head-Mounted Displays. It refers to applications that span across different stages (real, Augmented Reality, Augmented Virtuality, Virtual Reality) of the reality-virtuality continuum, where users are interconnected between different stages and/or are able to transition between these stages. This publication follows the concept of other grand challenges publications and reflects the discussion of various researchers invested in CR. After an initial discussion at the 1stJoint Workshop on Cross Reality at IEEE ISMAR 2023, six topic groups have been identified, leading to 22 challenges, which were discussed in groups over the period of multiple months. The discussion of these challenges should act as a road map for future research in the area of CR.

Teleportation is widely used in Virtual Reality (VR) applications, especially for enabling efficient navigation in large-scale virtual environments. Leveraging foot movements for teleportation frees the hands for parallel tasks and helps maintain spatial orientation, making it particularly suitable for seated and multitasking VR scenarios. This work explores feasible foot actions for teleportation in confined seating conditions, such as traveling on an airplane or train, where users' leg motion is feasible, but its range is constrained. Such scenarios are common but are underexplored. We propose seven metaphor-inspired teleportation techniques based on foot interactions and evaluate their effectiveness through a user study. The findings provide valuable insights for designing effective teleportation techniques in VR scenarios where users are seated in constrained spaces.

The advent of web-based interactive technologies has opened up new possibilities for virtual gatherings in 3D environments. Live-streaming, in particular, has gained increasing attention due to its effectiveness in engaging a large number of users in collective online activities. With an emphasis on audience participation, live-streaming shares common characteristics of the outlook of the metaverse and is driving new waves of interaction in virtual gatherings, such as engaging users through crowdsourcing control. However, this type of social interaction has not been examined in the Asian context, and it lacks systematic investigation of user experience with different crowdsourcing control methods. In this paper, we present a novel crowdsourcing control method based on DanMu, the subtitle system of Bilibili, one of the most successful and prevalent live-streaming platforms in Asia. We organized virtual gatherings by live-streaming a Minecraft virtual campus and examined the use of DanMu for crowdsourcing control. Our first study investigated the influence of three crowdsourcing control methods (First Come First Served, Vote, and Super Command) on collective navigation task efficiency and user experience. These influences were further discussed with user activeness and group sizes in a follow-up study. The results showed that Super Command, a representative mode on top of the democratic voting mechanism, offers better user experiences and social richness in large groups. Participants also rated its usability higher in small groups. Besides, virtual gathering in small groups allows greater pragmatic quality, usability, and a sense of agency than in large groups. Our work provides design guidelines for developers and HCI practitioners to develop crowdsourcing control methods and improve novel virtual gathering experiences in virtual worlds and the future metaverse.

Cultural heritage, a testament to human history and civilization, has gained increasing recognition for its significance in preservation and dissemination. The integration of immersive technologies has transformed how cultural heritage is presented, enabling audiences to engage with it in more vivid, intuitive, and interactive ways. However, the adoption of these technologies also brings a range of challenges and potential risks. This paper presents a systematic review, with an in-depth analysis of 177 selected papers. We comprehensively examine and categorize current applications, technological approaches, and user devices in immersive cultural heritage presentations, while also highlighting the associated risks and challenges. Furthermore, we identify areas for future research in the immersive presentation of cultural heritage. Our goal is to provide a comprehensive reference for researchers and practitioners, enhancing understanding of the technological applications, risks, and challenges in this field, and encouraging further innovation and development.

As see-through Mixed Reality Head-Mounted Displays (MRHMDs) proliferate, their usage is gradually shifting from controlled, private settings to spontaneous, public contexts. While location-based augmented reality mobile games such as Pokémon GO have been successful, the embodied interaction afforded by MRHMDs moves play beyond phone-based screen-tapping toward co-located, bodily, movement-based play. In anticipation of widespread MRHMD adoption, major technology companies have teased concept videos envisioning urban streets as vast mixed reality playgrounds-imagine Harry Potter-style wizard duels in city streets-which we term Immersive Mixed Reality Street Play (IMRSP). However, few real-world studies examine such scenarios. Through empirical, in-the-wild studies of our research-through-design game probe, Multiplayer Omnipresent Fighting Arena (MOFA), deployed across diverse public venues, we offer initial insights into the social implications, challenges, opportunities, and design recommendations of IMRSP. The MOFA framework, which includes three gameplay modes-''The Training'', ''The Duel'', and ''The Dragon''-is open-sourced at https://github.com/realitydeslab/mofa.

Recent advances in eye-tracking technology have positioned gaze as an efficient and intuitive input method for Virtual Reality (VR), offering a natural and immersive user experience. As a result, gaze input is now leveraged for fundamental interaction tasks such as selection, manipulation, crossing, and steering. Although several studies have modeled user steering performance across various path characteristics and input methods, our understanding of gaze-based steering in VR remains limited. This gap persists because the unique qualities of eye movements-involving rapid, continuous motions-and the variability in eye-tracking make findings from other input modalities nontransferable to a gaze-based context, underscoring the need for a dedicated investigation into gaze-based steering behaviors and performance. To bridge this gap, we present two user studies to explore and model gaze-based steering. In the first one, user behavior data are collected across various path characteristics and eye-tracking conditions. Based on this data, we propose four refined models that extend the classic Steering Law to predict users' movement time in gaze-based steering tasks, explicitly incorporating the impact of tracking quality. The best-performing model achieves an adjusted R2 of 0.956, corresponding to a 16% improvement in movement time prediction. This model also yields a substantial reduction in AIC (from 1550 to 1132) and BIC (from 1555 to 1142), highlighting improved model quality and better balance between goodness of fit and model complexity. Finally, data from a second study with varied settings, such as a different eye-tracking sampling rate, illustrate the strong robustness and predictability of our models. Finally, we present scenarios and applications that demonstrate how our models can be used to design enhanced gaze-based interactions in VR systems.

The viewing experience of 3D artifacts in Virtual Reality (VR) museums is constrained and affected by various factors, such as pedestal height, viewing distance, and object scale. User experiences regarding these factors can vary subjectively, making it difficult to identify a universal optimal solution. In this paper, we collect empirical data on user-determined parameters for the optimal viewing experience in VR museums. By modeling users' viewing behaviors in VR museums, we derive predictive functions that configure the pedestal height, calculate the optimal viewing distance, and adjust the appropriate handheld scale for the optimal viewing experience. This led to our novel 3D responsive design, ResponsiveView. Similar to the responsive web design that automatically adjusts for different screen sizes, ResponsiveView automatically adjusts the parameters in the VR environment to facilitate users' viewing experience. The design has been validated with two popular inputs available in current commercial VR devices: controller-based interactions and hand tracking, demonstrating enhanced viewing experience in VR museums.

Prior work has shown that frame rate significantly influences user behavior in fast-response tasks in 2D and 3D contexts. However, its impact on a steering task, which involves navigating an object along a path from the start to the end, remains relatively unexplored, especially in the context of virtual reality (VR). This task is considered a typical non-fast-response activity, as it does not demand rapid reactions within a limited time frame. Our work aims to understand and model users' steering behavior and predict movement time with different task complexities and frame rates in VR environments. We first conducted a user study to collect user behavior in a steering task with four factors: frame rate, path length, width, and radius of curvature. Based on the results, we then quantified the effects of frame rate and built two predictive models. Our models exhibited the best fit ($r^{2}> 0.957$r2>0.957) and over 17% improvement in prediction accuracy for movement time compared to existing models. Our models' robustness was further validated by applying them to predict steering performance with different VR tasks and frame rates. The two models keep the best predictability for both movement time and speed.

Tangible interfaces in mixed reality (MR) environments allow for intuitive data interactions. Tangible cubes, with their rich interaction affordances, high maneuverability, and stable structure, are particularly well-suited for exploring multi-dimensional data types. However, the design potential of these cubes is underexplored. This study introduces a design space for tangible cubes in MR, focusing on interaction space, visualization space, sizes, and multiplicity. Using spatio-temporal data, we explored the interaction affordances of these cubes in a workshop (N=24). We identified unique interactions like rotating, tapping, and stacking, which are linked to augmented reality (AR) visualization commands. Integrating user-identified interactions, we created a design space for tangible-cube interactions and visualization. A prototype visualizing global health spending with small cubes was developed and evaluated, supporting both individual and combined cube manipulation. This research enhances our grasp of tangible interaction in MR, offering insights for future design and application in diverse data contexts.

Piloting requires users to control and navigate the aircraft within a designated pathway, with a controller that utilizes two joysticks to control the aircraft. This task is representative of various daily and gaming scenarios, such as controlling the aircraft to capture the photo or navigating an object in a game from the start position to the end via a trajectory. In this work, we explore a model (based on the Steering Law) that predicts the piloting time required in spatial-constrained environments. Thus, two user studies are conducted to help us understand the relationship between path complexity (curvature) and spatial constraints (width and height). According to the results, we propose a model that can achieve $52.6 \%$ and $60.6 \%$ improvement in R-square and the Akaike Information Criterion (AIC), respectively. Next, an additional study was conducted to further verify the performance and efficiency of our proposed model with the change of movement direction and orientation. Our model and experimental results can benefit both game and interface designers of applications that require controlling moving objects along specific trajectories in virtual reality environments.

Recognizing the potential of freehand gestures for interacting with virtual objects in virtual environments, our research introduces a user-defined freehand gesture set of typical referents in virtual reality (VR), focusing on a specific scenario: VR museums. We conducted a comprehensive elicitation study with two experiments to define and refine the gesture set. Meanwhile, we demonstrated an enhanced real-time Wizard of Oz approach that facilitated users’ understanding of referents in VR and their gesture design. Our findings revealed significant improvements in gesture consistency and user agreement through two experiments, with an average agreement score of firstchoice and second-choice advancing from 0.211 and 0.160 to 0.412 and 0.284, respectively. By offering a consistent user-centered gesture set, this work contributes to guiding museum curators toward creating more immersive user experiences in VR museums. The gestures can also be extended to other VR applications that necessitate travel, selection and manipulation, and system control tasks.

In a physical museum, text descriptions are typically displayed on placards or signage next to exhibits. Within a virtual museum environment, these text descriptions can be presented in various ways, such as fixed in the virtual environment, attached to the exhibits, or held in users’ hands. By seamlessly integrating text descriptions into the virtual environment and allowing users to engage with the content, the information presentation can be highly interactive. However, the design space of artifact information presentation in virtual museums was under-explored. In this paper, we investigated appropriate ways to present text descriptions of artifacts in immersive virtual museums. Specifically, we studied (1) users’ perceived importance of various information dimensions (observable, non-observable, and interpretation), (2) users’ expected display of text panels (shown or hidden), and (3) the relationship between the artifact information dimensions and layout types (environment-, object-, and user-based). Our results showed that participants rated significantly higher importance for non-observable information than observable and interpretation information. In addition, we summarize a design space for artifact information presentation using different layout types with prioritized options. Our work provides insights for the interaction design of artifact information in virtual museums and the presentation of text in virtual reality.

Text selection is a common task in interactive systems. Often, it can be difficult because the letters and words are too small and clustered together to allow precise selection. Compared to traditional 2D interfaces, text selection is more challenging in virtual reality (VR) head-mounted displays (HMDs) because users interact with the immersive 3D space via mid-air interaction, which has higher degrees of freedom but becomes more imprecise and involves a higher workload due to the lack of support from a fixed structure like a desk. There has been limited exploration of techniques that support precise and rapid text selection at the character, word, sentence, or paragraph levels in VR HMDs. To fill this gap, we propose three controller-based text selection methods: Joystick Movement, Depth Movement, and Wrist Orientation. They are evaluated against a baseline selection method via a user study with 24 participants. Results show that the three proposed techniques significantly improved the performance and user experience over the baseline, especially for the selection beyond the character level.

Museum visitors are typically advised to follow trajectories planned by curators. Nevertheless, the diverse locomotion techniques available in Virtual Reality (VR) offer various navigation methods that are unattainable within physical museum spaces. Interestingly, these techniques have rarely been explored within museum settings. Our study aims to investigate appropriate navigation methods in VR museums. We first conducted a study in a virtual reconstruction of a local museum with the following navigation methods: a 2D minimap, a World-in-Miniature (WiM) system, and a WiM map. Our results showed that the WiM map with a point-and-select interaction technique outperformed the other two regarding ease of learning, reduced workload, lessened motion sickness, and greater user preferences. Based on the findings, we improved the WiM map and introduced MagicMap. It builds upon the WiM map and translates the curatorial principles of museum visiting into a hierarchical menu layout. Our further evaluation showed that MagicMap supported prolonged engagement in VR museums, enhanced system usability and overall user experience, and reduced users’ perceived workload. Our findings have implications for the future design of navigation systems in VR museums and complex indoor environments.

Museum artifacts are the main way for visitors to experience and learn about cultural heritage. Augmented reality (AR) allows for high interactivity and is increasingly applied in museums to improve tourists’ experience and learning. It also supports the extension of museum experience to outside of the physical museum space, contributing to the visiting trajectory and takeaway experience. In this paper, we present our design of two tangible AR interfaces for cultural artifacts: Postcard AR and CubeMuseum AR, followed by three user studies that evaluate and optimize the design. In Study 1, we conducted a within-subjects study (N = 24) that compares the two AR interfaces with a baseline condition (Leaflet). Our results demonstrate the positive effects of tangible AR interfaces on users’ motivation and engagement in learning cultural heritage. In Study 2, we further explored how to optimize CubeMuseum AR by adopting a user-centered design approach. Through the analysis of expert interviews (N = 7) and an online survey (N = 207), the results specify a series of requirements and design guidelines for tangible AR interfaces to be used as a learning tool and a hybrid gift. Based on the findings, the design of the CubeMuseum AR was optimized and evaluated in Study 3. A between-subjects user study was conducted (N = 32) to compare the optimized design with the initial design. The results verified the positive effects of gamified tangible AR interfaces on users’ motivation, engagement, and performance in learning cultural heritage. We present our design and evaluation results, and discuss the implications of designing tangible AR interfaces for cultural heritage learning and museum gifting.

Steering is a fundamental task in interactive Virtual Reality (VR) systems. Prior work has demonstrated that movement direction can significantly influence user behavior in the steering task, and different interactive environments (VEs) can lead to various behavioral patterns, such as tablets and PCs. However, its impact on VR environments remains unexplored. Given the widespread use of steering tasks in VEs, including menu adjustment and object manipulation, this work seeks to understand and model the directional effect with a focus on barehand interaction, which is typical in VEs. This paper presents the results of two studies. The first study was conducted to collect behavioral data with four categories: movement time, average movement speed, success rate, and reenter times. According to the results, we examined the effect of movement direction and built the SθModel. We then empirically evaluated the model through the data collected from the first study. The results proved that our proposed model achieved the best performance across all the metrics (r2 > 0.95), with more than 15% improvement over the original Steering Law in terms of prediction accuracy. Next, we further validated the SθModel by another study with the change of device and steering direction. Consistent with previous assessments, the model continues to exhibit optimal performance in both predicting movement time and speed. Finally, based on the results, we formulated design recommendations for steering tasks in VEs to enhance user experience and interaction efficiency.

Target selection is a fundamental task in interactive Augmented Reality (AR) systems. Predicting the intended target of selection in such systems can provide users with a smooth, low-friction interaction experience. Our work aims to predict gaze-based target selection in AR headsets with eye and head endpoint distributions, which describe the probability distribution of eye and head 3D orientation when a user triggers a selection input. We first conducted a user study to collect users’ eye and head behavior in a gaze-based pointing selection task with two confirmation mechanisms (air tap and blinking). Based on the study results, we then built two models: a unimodal model using only eye endpoints and a multimodal model using both eye and head endpoints. Results from a second user study showed that the pointing accuracy is improved by approximately 32% after integrating our models into gaze-based selection techniques.

For museums in Virtual Reality (VR), various interaction and manipulation techniques could be employed for users to engage with artefact interactions. This study examined four combinations of interaction (controller-based and hand-tracking) and manipulation (direct and indirect) techniques, assessing user performance and experience with these interaction techniques in a virtual museum environment. We conducted a within-subjects experiment and asked participants to perform a series of transform manipulation tasks using the four techniques. Participants’ task completion time was measured. They also provided feedback on acceptance, learnability, presence, sickness, and fatigue, and gave an overall ranking through post-experiment questionnaires and interviews. The results revealed that controller-based direct manipulation outperformed the other techniques in terms of task performance and user experience, with hand-tracking indirect manipulation being the least efficient and the least preferred option. The study offers insights for future research and development in refining interaction and manipulation techniques and designing more user-friendly VR museum experiences.

Locomotion plays a critical role in user experience in Virtual Reality (VR). This work presents a novel locomotion device, LeanOn, which aims to enhance immersion and feedback experience in VR. Inspired by balance vehicles, LeanOn is a leaning-based locomotion device that allows users to control their location by tilting a board on two balance wheels, with rotation enabled by two buttons near users’ feet. To create a more realistic riding experience, LeanOn is equipped with a terrain vibration system that generates varying levels of vibration based on the roughness of the terrain. We conducted a within-subjects experiment $(\mathrm{N}=24)$ and compared the use of LeanOn and joystick steering in four aspects: cybersickness, spatial presence, feedback experience, and task performance. Participants used LeanOn with and without the vibration system to investigate the necessity of tactile feedback. The results showed that LeanOn significantly improved users’ feedback experience, including autotelic, expressivity, harmony, and immersion, and maintained similar levels of cybersickness and spatial presence, compared to joystick steering. Our work contributes to the field of VR locomotion by validating a leaning-based steering prototype and showing its positive effect on improving users’ feedback experience in VR. We also showed that tactile feedback in locomotion is necessary to further enhance immersion in VR.

The support for multi-focus data exploration is vital in collaborative visualization. In these scenarios, which often involve multiple devices and large displays, users may focus on specific information on their individual screens while also sharing contextual views with others. While many visualization techniques developed for single-user applications can be adapted for use in collaborative settings, little research has been done on how to design adaptive versions of these techniques or how they may impact collaborative tasks involving large datasets. In this work, we perform a comparative study of three collaborative visualization techniques (Zooming, Interactive lenses and Overview+Detail) on large displays in three map-based visualization tasks (Exploration, Comparison and Spatial Memorizing). These three collaborative techniques draw on three different classical visualization techniques in a single-user setting. Our results show that these techniques have different impacts on users’ task performance and preferences. The collaborative Overview+Detail technique benefits users most in supporting Spatial Memorizing. Closely coupled groups prefer collaborative Zooming in Target Exploration. Based on these results, we further discuss the design of collaborative visualization techniques and propose suggestions for adapting classical single-user visualization techniques to a collaborative setting.

Hybridity in immersive technologies has not been studied for factors that are likely to influence engagement. A noticeable factor is the spatial enclosure that defines where users meet. This involves a mutual object of interest, contents that the users may generate around the object, and the proximity between users. This study examines these factors, namely how object interactivity, user-generated contents (UGC) and avatar proximity influence engagement. We designed a Hybrid Virtual and Augmented Reality (HVAR) environment that supports paired users to experience cultural heritage in both Virtual Reality (VR) and Augmented Reality (AR) . A user study was conducted with 60 participants, providing assessments of engagement and presence via questionnaires, together with mobile electroencephalogram (mEEG) and user activity data that measures VR user engagement in real-time. Our findings provide insights into how engagement between users can occur in HVAR environments for the future hybrid reality with multi-device connectivity.

An increasing number of reconstructed digital assets are being created worldwide to preserve cultural heritage. These assets can be used in interactive systems such as augmented reality (AR) and virtual reality (VR) to provide effective ways to access and learn about cultural heritage. One of the widely adopted reconstruction techniques is close-range photogram-metry. However, scanned models need to be processed and optimized before they can be used in interactive systems, which requires a series of retopology and baking work to reduce the size of models while maintaining visual fidelity. Nevertheless, manual retopology and baking are complex processes. An efficient optimization workflow is essential for the use of cultural heritage assets in interactive systems. This paper presents an optimization workflow for retopology and texture baking using free and open-source software. Evaluations show that the workflow demon-strates its strengths in its high efficiency, versatility, learnability, and low cost. This work contributes insights to researchers and practitioners in the field of cultural heritage.