Cross Reality (CR) is a new emerging field based on the current developments in Mixed Reality hardware, especially supported by the broad market penetration of video-based see-through Head-Mounted Displays. It refers to applications that span across different stages (real, Augmented Reality, Augmented Virtuality, Virtual Reality) of the reality-virtuality continuum, where users are interconnected between different stages and/or are able to transition between these stages. This publication follows the concept of other grand challenges publications and reflects the discussion of various researchers invested in CR. After an initial discussion at the 1stJoint Workshop on Cross Reality at IEEE ISMAR 2023, six topic groups have been identified, leading to 22 challenges, which were discussed in groups over the period of multiple months. The discussion of these challenges should act as a road map for future research in the area of CR.

Recent Head-Mounted Displays enable users to perceive the real environment using a video-based see-through mode and the fully virtual environment within a single display. Leveraging these advancements, we present a generic concept to seamlessly transition between the real and virtual environment, with the goal of supporting users in engaging with and disengaging from any real environment into Virtual Reality. This transition process uses a digital replica of the real environment and incorporates various stages of Milgram’s Reality-Virtuality Continuum, along with visual transitions that facilitate gradual navigation between them. We implemented the overall transition concept and four object-based transition techniques. The overall transition concept and four techniques were evaluated in a qualitative user study, focusing on user experience, the use of the replica and visual coherence.

Transitional Interfaces are a yet underexplored, emerging class of cross-reality user interfaces that enable users to freely move along the reality-virtuality continuum during collaboration. To analyze and understand how such collaboration unfolds, we propose four analytical lenses derived from an exploratory study of transitional collaboration with 15 dyads. While solving a complex spatial optimization task, participants could freely switch between three contexts, each with different displays (desktop screens, tablet-based augmented reality, head-mounted virtual reality), input techniques (mouse, touch, handheld controllers), and visual representations (monoscopic and allocentric 2D/3D maps, stereoscopic egocentric views). Using the rich qualitative and quantitative data from our study, we evaluated participants’ perceptions of transitional collaboration and identified commonalities and differences between dyads. We then derived four lenses including metrics and visualizations to analyze key aspects of transitional collaboration: (1) place and distance, (2) temporal patterns, (3) group use of contexts, (4) individual use of contexts.

Safety-critical interactive spaces for supervision and time-critical control tasks are usually characterized by many small displays and physical controls, typically found in control rooms or automotive, railway, and aviation cockpits. Using Virtual Reality (VR) simulations instead of a physical system can significantly reduce the training costs of these interactive spaces without risking real-world accidents or occupying expensive physical simulators. However, the user's physical interactions and feedback methods must be technologically mediated. Therefore, we conducted a within-subjects study with 24 participants and compared performance, task load, and simulator sickness during training of authentic aircraft cockpit manipulation tasks. The participants were asked to perform these tasks inside a VR flight simulator (VRFS) for three feedback methods (acoustic, haptic, and acoustic+haptic) and inside a physical flight simulator (PFS) of a commercial airplane cockpit. The study revealed a partial equivalence of VRFS and PFS, control-specific differences input elements, irrelevance of rudimentary vibrotactile feedback, slower movements in VR, as well as a preference for PFS.

Cross-Virtuality applications enabling users to move between different stages of Milgram’s reality-virtuality continuum are a rapidly growing field of research. Modern video see-through head-mounted displays allow users to switch between augmented and virtual reality without removing the headset. This enables for the first time a fluent transition between augmented and virtual reality. Based on insights from literature and preliminary experiments we designed and implemented four transitions: Fade, SimpleCut, TeleportBeam and Portal. These techniques were expected to represent the best suitable concepts for transitioning seamlessly between augmented and virtual reality. After incorporating results from a pre-study, the transition techniques were evaluated in a qualitative user study regarding user experience, simulator sickness, continuity and applicability. Participants were able to freely move between both realities during the study in an immersive analytics scenario for logistics data. In the user study, users preferred Fade in a workplace setting due to its efficiency and simplicity when transitioning frequently between realities. The Portal technique was deemed visually exciting and suitable for infrequent transitions between realities that differ greatly.

Abstract Cross‐virtuality analytics (XVA) is a novel field of research within immersive analytics and visual analytics. A broad range of heterogeneous devices across the reality–virtuality continuum, along with respective visual metaphors and analysis techniques, are currently becoming available. The goal of XVA is to enable visual analytics that use transitional and collaborative interfaces to seamlessly integrate different devices and support multiple users. In this work, we take a closer look at XVA and analyse the existing body of work for an overview of its current state. We classify the related literature regarding ways of establishing cross‐virtuality by interconnecting different stages in the reality–virtuality continuum, as well as techniques for transitioning and collaborating between the different stages. We provide insights into visualization and interaction techniques employed in current XVA systems. We report on ways of evaluating such systems, and analyse the domains where such systems are becoming available. Finally, we discuss open challenges in XVA, giving directions for future research.

Augmented reality (AR) can create the illusion of being virtually co-located during remote collaboration, e.g., by visualizing remote co-workers as avatars. However, spatial awareness of each other's activities is limited as physical spaces, including the position of physical devices, are often incongruent. Therefore, alignment methods are needed to support activities on physical devices. In this paper, we present the concept of Re-locations, a method for enabling remote collaboration with augmented reality in incongruent spaces. The idea of the concept is to enrich remote collaboration activities on multiple physical devices with attributes of co-located collaboration such as spatial awareness and spatial referencing by locally relocating remote user representations to user-defined workspaces. We evaluated the Re-locations concept in an explorative user study with dyads using an authentic, collaborative task. Our findings indicate that Re-locations introduce attributes of co-located collaboration like spatial awareness and social presence. Based on our findings, we provide implications for future research and design of remote collaboration systems using AR.

This pictorial explores the design space for communicating surface gestures to users of textile interfaces by experimenting with the interfaces’ physical design and affordances. First, we created a collection of functional and non-functional textile samples. Their development was based on three aspects: design, fabrication, and sensing. The design aspect covered different visual (shape, color) and haptic (details, textures) designs, fabrication explored three textile-specific fabrication methods, and electronic sensing offered options for adding touch-sensing capabilities. Second, we reflected on created samples and their characteristics contrasting different designs and speculating on why some work better than others. Our main findings and insights are presented in five clusters: ergonomics, visual affordances, perception of textures, the direction of movement, and the economic usage of design elements. This intermediate-level knowledge can provide a starting point for each professional or novice designer to take inspiration from, when creating their own textile user interfaces.

It is our great pleasure to welcome you to this issue of the Proceedings of the ACM on Human-Computer Interaction, the second to focus on the contributions from the research community Interactive Surfaces and Spaces (ISS). Interactive Surfaces and Spaces increasingly pervade our everyday life, appearing in various sizes, shapes, and application contexts, offering a rich variety of ways to interact. This diverse research community explores the design, development, and use of new and emerging interactive surface technologies and interactive spaces. The call for articles for this issue on ISS attracted 77 submissions, from all over the world. This issue has 23 papers, 4 submitted in February 2021 and 19 submitted in July 2021. After the winter round, 4 (total of 19 articles, 21.1%) articles were accepted and 5 (26.3%) articles required major revisions. After the summer round, 19 (total of 58 articles, 32.8%) articles were accepted, and 18 (31,0%) articles required major revisions. The editorial committee worked hard over the two iterations of the review process, winter and summer rounds, to arrive at final decisions. In total, counting both the winter and the summer rounds, 23 articles (total of 77 articles, 29.9%) were accepted. All authors of the accepted articles are invited to present at the ISS conference from November 14--17, 2021. This issue exists because of the dedicated volunteer effort of 31 senior editors who served as Associate Chairs (ACs), 105 expert reviewers in the winter round, and 206 expert reviewers in the summer round to ensure high quality and insightful reviews for all articles. Reviewers and committee members were kept constant for papers that submitted to both rounds. The Editorial Board is presented here: https://iss.acm.org/2021/organization/editorial_board

We propose using virtual reality (VR) as a design tool for sketching and simulating spatially-aware interactive spaces. Using VR, designers can quickly experience their envisioned spaces and interactions by simulating technologies such as motion tracking, multiple networked devices, or unusual form factors such as spherical touchscreens or bezel-less display tiles. Design ideas can be rapidly iterated without restrictions by the number, size, or shape and availability of devices or sensors in the lab. To understand the potentials and challenges of designing in VR, we conducted a user study with 12 interaction designers. As their tool, they used a custom-built virtual design environment with finger tracking and physics simulations for natural interactions with virtual devices and objects. Our study identified the designers' experience of space in relation to their own bodies and playful design explorations as key opportunities. Key challenges were the complexities of building a usable yet versatile VR-based "World Editor".

PolarTrack is a novel camera-based approach to detecting and tracking mobile devices inside the capture volume. In PolarTrack, a polarization filter continuously rotates in front of an off-the-shelf color camera, which causes the displays of observed devices to periodically blink in the camera feed. The periodic blinking results from the physical characteristics of current displays, which shine polarized light either through an LC overlay to produce images or through a polarizer to reduce light reflections on OLED displays. PolarTrack runs a simple detection algorithm on the camera feed to segment displays and track their locations and orientations, which makes PolarTrack particularly suitable as a tracking system for cross-device interaction with mobile devices. Our evaluation of PolarTrack's tracking quality and comparison with state-of-the-art camera-based multi-device tracking showed a better tracking accuracy and precision with similar tracking reliability. PolarTrack works as standalone multi-device tracking but is also compatible with existing camera-based tracking systems and can complement them to compensate for their limitations.

Using predictive models for data-driven optimization of production lines is a trending topic in manufacturing. A challenge is to support the visualization and management of data and arising uncertainties in various phases. Modern production lines can contain thousands of sensors to collect readings every few milliseconds. When creating predictive models for optimization from this data, its sheer amount not only poses challenges to data storage and management, but also during the different phases of data preparation, modeling, decision making, and executing optimizations. Our research aims at a better support of all these phases with data visualization, with new strategies for the management of uncertainty. Predictive models help operators to identify problems and the degree of wear of tools or parts in production lines. However, data analysts, supervisors, and service technicians must be aware that these models contain uncertainties and the recommendations made by these models should be critically reflected before acceptance. One way to mitigate the negative impact of uncertainties is to use visualization as tool for analyzing the potentials and limits of prediction models. Therefore, we propose a process model for the different activities of optimization in a smart production environment and discuss challenges for the visualization of data and uncertainty in each step.

We present Domino, a descriptive framework for hybrid collaboration and hybrid coupling styles in partially distributed teams. Domino enables researchers to describe, analyze, and understand real-world hybrid collaboration practices, i.e., collaborative practices that involve simultaneous co-located and remote collaboration with phases of both synchronous and asynchronous work that spans multiple groupware applications and devices. It also helps to categorize collaborative activities based on yet undocumented hybrid coupling styles between the members of multiple partially distributed or co-located subgroups. Our Domino framework was derived from initial observations of real-world practice and refined by the detailed analysis of participants' behavior and working styles during a simulation of a complex hybrid collaboration task with six partially distributed teams of four users in our lab. The resulting framework allows researchers to view collaboration through a new analytical lens, use new analytical tools, and also derive implications for the design of collaborative tools.

A sizable part of HCI research on cross-device interaction is driven by the vision of users conducting complex knowledge work seamlessly across multiple mobile devices. This is based on the Weiserian assumption that people will be inclined to distribute their work across multiple ``pads' if such are available. We observed that this is not the reality today, even when devices were in abundance. We present a study with 24 participants in 12 dyads completing a collaborative visualization task with up to six tablets. They could choose between three different visualization types to answer questions about economic data. Tasks were designed to afford simultaneous use of tablets, either with linked or independent views. We found that users typically utilized only one tablet per user. A quantitative and qualitative analysis revealed a ``legacy bias' that introduced barriers for using more tablets and reduced the overall benefit of multi-device visualization.

The urge to collaborate is deeply anchored in human nature. This is substantiated by evidence for different forms of group work already performed by early humans (e.g., when defending against threats). Over time, the behavior of working together, the motivation behind and the technological support have changed which also broadened the spectrum of collaborative activities. Especially today, on the brink of the ubiquity era that envisions a person to use thousands of computers everyday, new pre-conditions for co-located and remote, planned and spontaneous collaboration are manifested. Thus, previous models that distinguished between phases of collaboration such as starting, planning, executing and reporting might become less applicable. The relevance of the emergence of collaboration phase, which has been considered only marginally so far, is likely to increase drastically. This position paper discusses considerations on the nature of collaboration throughout the ubiquity era and suggests an accordingly extended phases of collaboration model.

Constructive assemblies are tangible user interfaces (TUIs) that involve the interconnection of modular parts. As such, they offer a unique means to support us in our various and diverse activities. However, little information is available for understanding the intricacies of taking a modular, constructive assembly approach to TUI design. Based on an analysis of extensive data collected from interviews with eight world-class TUI experts, we propose a descriptive, conceptual framework to facilitate systematic investigation and critical consideration of constructive assemblies. The paper presents a lifecycle model for constructive assemblies and discusses their main design qualities and associated parameters. We demonstrate how this can be used to structure critical discussions by applying the principles to existing works and the design of our own constructive assembly.

Cross-device collaboration with tablets is an increasingly popular topic in HCI. Previous work has shown that tablet-only collaboration can be improved by an additional shared workspace on an interactive tabletop. However, large tabletops are costly and need space, raising the question to what extent the physical size of shared horizontal surfaces really pays off. In order to analyse the suitability of smaller-than-tabletop devices (e.g. tablets) as a low-cost alternative, we studied the effect of the size of a shared horizontal interactive workspace on users' attention, awareness, and efficiency during cross-device collaboration. In our study, 15 groups of two users executed a sensemaking task with two personal tablets (9.7") and a horizontal shared display of varying sizes (10.6", 27", and 55"). Our findings show that different sizes lead to differences in participants' interaction with the tabletop and in the groups' communication styles. To our own surprise we found that larger tabletops do not necessarily improve collaboration or sensemaking results, because they can divert users' attention away from their collaborators and towards the shared display.

Dynamic peephole navigation represents a technique for navigating large information spaces in an egocentric way. Studies have shown cognitive benefits for a vertical peephole orientation, when compared to non-egocentric interaction styles. To see how the aspect of canvas orientation effects user performance, we conducted a study (N=16) which revealed that canvas orientation has no significant effect on either navigation performance or spatial memory. We also found a significantly lower physical demand and a higher mental demand in the horizontal orientation. For short-term activities we therefore propose a vertical orientation, while for long-term activities horizontal dynamic peephole navigation is more suitable.

Cross-device interaction between multiple mobile devices is a popular field of research in HCI. However, the appropriate design of this interaction is still an open question, with competing approaches such as spatially-aware vs. spatially-agnostic techniques. In this paper, we present the results of a two-phase user study that explores this design space: In phase 1, we elicited gestures for typical mobile cross-device tasks from 4 focus groups (N=17). The results show that 71% of the elicited gestures were spatially-aware and that participants strongly associated cross-device tasks with interacting and thinking in space. In phase 2, we implemented one spatially-agnostic and two spatially-aware techniques from phase 1 and compared them in a controlled experiment (N=12). The results indicate that spatially-aware techniques are preferred by users and can decrease mental demand, effort, and frustration, but only when they are designed with great care. We conclude with a summary of findings to inform the design of future cross-device interactions.

Dynamic peephole navigation is an increasingly popular technique for navigating large information spaces such as maps. Users can view the map through handheld, spatially aware displays that serve as peepholes and navigate the map by moving these displays in physical space. We conducted a controlled experiment of peephole map navigation with 16 participants to better understand the effect of a peephole's size on users' map navigation behavior, navigation performance, and task load. Simulating different peephole sizes from 4' (smartphone) up to 120' (control condition), we confirmed that larger peepholes significantly improve learning speed, navigation speed, and reduce task load; however, this added benefit diminishes with growing sizes. Our data shows that a relatively small, tablet-sized peephole can serve as a 'sweet spot' between peephole size and both user navigation performance and user task load.

Our research is concerned with designing support for ubiquitous access, organization, and interpretation of digital assets that users produce and store across multiple devices and computing platforms. Through a co-design study with scientists we identified specific aspects of conceptual maps they wish to create for their projects in order to make sense of diverse collections of digital media related to their individual and collective work. By the use of a technology probe called DeskPiles we observed the mechanisms involved in such a process. We identified the practice of using sub-document elements to compose the views of digital collections and diverse linking strategies to express semantic relations among concepts and ideas and to enable access to supporting digital assets. Our findings reveal the need to expand traditional information architectures and include (1) an extended range of content reference options and (2) transformation services to enable extraction, conversion, and linking of digital content. We advocate a granular and multi-format representation of content as the basis for reuse and ubiquitous access of digital media in diverse multi-device environments.

Interactive spaces are ubiquitous computing environments for computer-supported collaboration that exploit and enhance the existing cognitive, physical and social skills of users or groups of users. The workshop aims at documenting and advancing the current state-of-the-art of co-located collaboration in interactive spaces and identifying research challenges and formulating a research agenda by inviting high-quality position and research papers from HCI, Information Visualization, CSCW and CSCL.

Recent findings from Embodied Cognition reveal strong effects of arm and hand movement on spatial memory. This suggests that input devices may have a far greater influence on users' cognition and users' ability to master a system than we typically believe -- especially for spatial panning or zooming & panning user interfaces. We conducted two experiments to observe whether multi-touch instead of mouse input improves users' spatial memory and navigation performance for such UIs. We observed increased performances for panning UIs but not for zooming & panning UIs. We present our results, provide initial explanations and discuss opportunities and pitfalls for interaction designers.

“Interactive spaces” are physical environments or rooms for collaborative work that are augmented with ubiquitous computing technology. Their purpose is to enable a computer-supported collaboration between multiple users that is based on a seamless use of different devices for natural “post-WIMP” interaction (e.g., multitouch walls, interactive tabletops, tablet PCs, or digital pen and paper). However, to this day, there are no well-established guidelines or toolkits for designing and implementing such distributed user interfaces (DUIs). Therefore, this article introduces the Zoomable Object-Oriented Information Landscape (ZOIL), a novel design approach and software framework for post-WIMP DUIs in interactive spaces. In the following, the ZOIL design principles are first introduced and illustrated. They provide recommendations and examples of DUI interaction design for interactive spaces. Then the different software patterns and architectures that have been employed for implementing the open-source ZOIL software framework are described. This framework facilitates the implementation of ZOIL's design principles in practice. Lessons learned from ZOIL's implementation are shared, and the implementation is discussed and compared with related work and approaches. The results of an evaluation of ZOIL with designers and developers conclude the article.

Application programming interfaces (APIs) are the interfaces to existing code structures, such as widgets, frameworks, or toolkits. Therefore, they very much do have an impact on the quality of the resulting system. So, ensuring that developers can make the most out of them is an important challenge. However standard usability evaluation methods as known from HCI have limitations in grasping the interaction between developer and API as most IDEs (essentially the GUI) capture only part of it. In this paper we present the Concept Map method to study the usability of an API over time. This allows us to elicit the mental model of a programmer when using an API and thereby identify usability issues and learning barriers and their development over time.

We introduce "Facet-Streams", a hybrid interactive surface for co-located collaborative product search on a tabletop. Facet-Streams combines techniques of information visualization with tangible and multi-touch interaction to materialize collaborative search on a tabletop. It harnesses the expressive power of facets and Boolean logic without exposing users to complex formal notations. Two user studies reveal how Facet-Streams unifies visual and tangible expressivity with simplicity in interaction, supports different strategies and collaboration styles, and turns product search into a fun and social experience.

Pen and paper support the rapid production of sketches. However, the paper interface is not always optimal for collaborative sketching as seen in brainstorming sessions where multiple parties would often like to communicate and participate in the sketching synchronously. Novel interactive paper solutions may provide the answer by bridging the paper-digital divide and allowing users to sketch on paper simultaneously while capturing the actions digitally. We present an analysis of collaborative sketching activities in working environments with remote participation. After highlighting the importance of paper for natural interaction in these settings, we introduce PaperSketch, an interactive paper-digital tool for collaborative remote sketching. We discuss the collaborative development of ideas based on the prototype and outline how important feedback issues have been addressed by utilising spatial constraints and multimodal features.

MedioVis is a visual information seeking system that aims to support users' natural seeking behavior, particularly in complex information spaces. To achieve this goal we introduce multiple complementary visualization techniques together with an easy-to-use and consistent interaction concept. Over the last four years, MedioVis was developed in the context of digital libraries following a user-centered design process. The focus of this paper is the presentation of our interaction model and further to give an overview of the applied visualization techniques.