Cross Reality (CR) is a new emerging field based on the current developments in Mixed Reality hardware, especially supported by the broad market penetration of video-based see-through Head-Mounted Displays. It refers to applications that span across different stages (real, Augmented Reality, Augmented Virtuality, Virtual Reality) of the reality-virtuality continuum, where users are interconnected between different stages and/or are able to transition between these stages. This publication follows the concept of other grand challenges publications and reflects the discussion of various researchers invested in CR. After an initial discussion at the 1stJoint Workshop on Cross Reality at IEEE ISMAR 2023, six topic groups have been identified, leading to 22 challenges, which were discussed in groups over the period of multiple months. The discussion of these challenges should act as a road map for future research in the area of CR.

We investigate hybrid user interfaces (HUIs), aiming to establish a cohesive understanding and to adopt consistent terminology for this nascent research area. HUIs combine heterogeneous devices in complementary roles, leveraging the distinct benefits of each. Our work focuses on cross-device interaction between 2D devices and mixed reality environments, which are particularly compelling, leveraging the familiarity of traditional 2D platforms while providing spatial awareness and immersion. Although prior work has prominently explored such HUIs in the context of mixed reality, we still lack a cohesive understanding of the unique design possibilities and challenges of such combinations, resulting in a fragmented research landscape. We conducted a systematic survey and present a taxonomy of HUIs that combine conventional display technology and mixed reality environments. Based on this, we discuss past and current challenges, the evolution of definitions, and prospective opportunities to tie together the past 30 years of research with our vision of future HUIs.

We introduce the SpatialMouse, a hybrid pointing device that combines the capabilities of a desktop mouse with the spatial input of a virtual reality (VR) controller, enabling seamless transitions between 2D and 3D interaction spaces in immersive mixed reality environments. Holistic usage scenarios in mixed reality involve tasks suited alternately to 2D or 3D information spaces. Yet, existing input devices excel in either 2D or 3D, but not both, making it necessary to switch between multiple input devices (e.g., mouse and VR controller). Our SpatialMouse addresses this issue, offering the affordances of a desktop mouse for indirect 2D pointing and the spatial capabilities of VR controllers with six degrees of freedom. In a user study with 12 participants, our prototype significantly reduced perceived task load and improved user experience compared to switching between separate devices. We extract design recommendations to further support such hybrid input approaches.

Smartphones provide convenient access to vast data collections (e.g., online shops, social media) within a compact, portable form factor. While the prevalent infinite scroll lists address the inherently restricted screen space, they also introduce navigation and orientation challenges. Users often lose track of their position within these lists and find it difficult to efficiently access, compare, and filter items of interest. To address this challenge, we introduce Push2AR, a novel interaction concept that extends the phone’s high-resolution display and familiar touch interaction with the virtual display space offered by Augmented Reality (AR) headsets. Push2AR enables users to transfer individual list items from their phone to its surrounding AR space, facilitating bookmarking, filtering, and side-by-side comparisons while maintaining orientation through visual links to the original scroll position. Our evaluation shows that our approach enhances user experience and reduces subjective workload involved in locating and comparing list items in contrast to conventional phone-only lists.

Smartphones conveniently place large information spaces in the palms of our hands. While research has shown that larger screens positively affect spatial memory, workload, and user experience, smartphones remain fairly compact for the sake of device ergonomics and portability. Thus, we investigate the use of hybrid user interfaces to virtually increase the available display size by complementing the smartphone with an augmented reality head-worn display. We thereby combine the benefits of familiar touch interaction with the near-infinite visual display space afforded by augmented reality. To better understand the potential of virtually-extended displays and the possible issues of splitting the user’s visual attention between two screens (real and virtual), we conducted a within-subjects experiment with 24 participants completing navigation tasks using different virtually-augmented display sizes. Our findings reveal that a desktop monitor size represents a “sweet spot” for extending smartphones with augmented reality, informing the design of hybrid user interfaces.

We present RagRug, an open-source toolkit for situated analytics. The abilities of RagRug go beyond previous immersive analytics toolkits by focusing on specific requirements emerging when using augmented reality (AR) rather than virtual reality. RagRug combines state of the art visual encoding capabilities with a comprehensive physical-virtual model, which lets application developers systematically describe the physical objects in the real world and their role in AR. We connect AR visualizations with data streams from the Internet of Things using distributed dataflow. To this end, we use reactive programming patterns so that visualizations become context-aware, i.e., they adapt to events coming in from the environment. The resulting authoring system is low-code; it emphasises describing the physical and the virtual world and the dataflow between the elements contained therein. We describe the technical design and implementation of RagRug, and report on five example applications illustrating the toolkit's abilities.

The nascent field of mixed reality is seeing an ever-increasing need for user studies and field evaluation, which are particularly challenging given device heterogeneity, diversity of use, and mobile deployment. Immersive analytics tools have recently emerged to support such analysis in situ, yet the complexity of the data also warrants an ex-situ analysis using more traditional non-immersive visual analytics setups. To bridge the gap between both approaches, we introduce ReLive: a mixed-immersion visual analytics framework for exploring and analyzing mixed reality user studies. ReLive combines an in-situ virtual reality view with a complementary ex-situ desktop view. While the virtual reality view allows users to relive interactive spatial recordings replicating the original study, the synchronized desktop view provides a familiar interface for analyzing aggregated data. We validated our concepts in a two-step evaluation consisting of a design walkthrough and an empirical expert user study.

Recent research in the area of immersive analytics demonstrated the utility of head-mounted augmented reality devices for visual data analysis. However, it can be challenging to use the by default supported mid-air gestures to interact with visualizations in augmented reality (e.g. due to limited precision). Touch-based interaction (e.g. via mobile devices) can compensate for these drawbacks, but is limited to two-dimensional input. In this work we present STREAM: Spatially-aware Tablets combined with Augmented Reality Head-Mounted Displays for the multimodal interaction with 3D visualizations. We developed a novel eyes-free interaction concept for the seamless transition between the tablet and the augmented reality environment. A user study reveals that participants appreciated the novel interaction concept, indicating the potential for spatially-aware tablets in augmented reality. Based on our findings, we provide design insights to foster the application of spatially-aware touch devices in augmented reality and research implications indicating areas that need further investigation.

Immersive technologies such as augmented reality devices are opening up a new design space for the visual analysis of data. This paper studies the potential of an augmented reality environment for the purpose of collaborative analysis of multidimensional, abstract data. We present ART, a collaborative analysis tool to visualize multidimensional data in augmented reality using an interactive, 3D parallel coordinates visualization. The visualization is anchored to a touch-sensitive tabletop, benefiting from well-established interaction techniques. The results of group-based, expert walkthroughs show that ART can facilitate immersion in the data, a fluid analysis process, and collaboration. Based on the results, we provide a set of guidelines and discuss future research areas to foster the development of immersive technologies as tools for the collaborative analysis of multidimensional data.