We characterize 16 challenges faced by those investigating and developing remote and synchronous collaborative experiences around visualization. Our work reflects the perspectives and prior research efforts of an international group of 29 experts from across human-computer interaction and visualization sub-communities. The challenges are anchored around five collaborative activities that exhibit a centrality of visualization and multimodal communication. These activities include exploratory data analysis, creative ideation, visualization-rich presentations, joint decision making grounded in data, and real-time data monitoring. The challenges also reflect the changing dynamics of these activities in the face of recent advances in extended reality (XR) and artificial intelligence (AI). As an organizing scheme for future research at the intersection of visualization and computer-supported cooperative work, we align the challenges with a sequence of four sets of research and development activities: technological choices, social factors, AI assistance, and evaluation.

Current Extended Reality (XR) devices are increasingly being used as productivity tools. Compared to conventional setups, they allow for more flexibility in dynamic work locations beyond the desktop while providing a large virtual workspace. Recent research has explored how users organize digital documents and how virtual interfaces could be adapted to different locations and scenarios. However, there has been limited research on how location changes affect productivity tasks in XR environments and how users manually adapt virtual content layouts after such task interruptions. To address this, we conducted an exploratory user study (N=17) in which participants worked on a document-centered organization and planning task while changing locations every five minutes. We examined how these spatial transitions interfered with the task and identified layout strategies and patterns. From our observations and participant responses, we derived a set of design guidelines to inform the development of future XR knowledge work systems in mobile contexts.

This paper investigates the challenges of designing mixed-presence environments for Mixed Reality and suggests future research directions derived from an expert workshop. Developing mixed-presence systems is a complex undertaking that combines the intricacies of both co-located and distributed mixed-reality spaces. Current literature in this field describes various promising design and development approaches but lacks a systematic overview, resulting in fragmented solutions to re-occurring challenges. Therefore, we conducted a comprehensive review of mixed-presence and multi-user remote mixed-reality systems, categorizing the prevalent challenges faced during the development of such systems, but also current trends, common use cases, study tasks and methodologies. Supported by these results, we then conducted an expert ideation workshop to collect and structure promising future research directions. As a result, we provide a detailed resource to orient and prepare developers for probable challenges and support researchers in making informed design decisions for future mixed-presence studies in Mixed Reality.

In traditional visual analysis, brushing and linking is commonly used to visually connect multiple views using highlighting techniques. However, brushing and linking has rarely been used in situated analytics, which uses visualizations to analyze data in the context of physical referents. In situated analytics, data representations must be visually linked to real-world objects. Previous work has assessed situated brushing and linking in a virtual reality simulation of a supermarket scenario. Here, we replicate and extend the previous approach by studying brushing and linking in an actual physical space with augmented reality, while further improving the highlighting techniques. Using a video see-through display, we compare augmented reality with virtual reality. Results suggest that AR performs better in time and accuracy, but the effectiveness of the techniques varies by condition. These results provide a new framing of how the real-world stimuli matter in situated analytics.

This paper investigates avatar self-views in virtual reality (VR) and their impact on multi-user communication scenarios. Self-views in video conferencing have been shown to impact our self-perception and mental load, so we explore whether similar effects occur in VR, as personal and professional gatherings progressively move to virtual spaces with 3D avatars. We identify the following key design dimensions for self-representations in VR: spatiality, anchoring and size, and self-visibility. Based on these, we designed three variants (Remote Perspective View, Personal Mirror, and Miniature Avatar), which we compare to a baseline (No Additional Self-View) in a user study. Our analysis of sixteen dyads playing a word-guessing game requiring verbal and non-verbal communication (i.e., explaining and charades) in VR confirms that self-views are beneficial for communication scenarios that require expressive body language or facial expressions, as this allows monitoring the own avatar's animations. Further, the Miniature Avatar was overall preferred due to its spatiality, world-anchoring, and full self-visibility. Based on our results, we offer design recommendations for self-views covering the key design dimensions in VR.

In recent years, data-driven stories have found a firm footing in journalism, business, and education. They leverage visualization and storytelling to convey information to broader audiences. Likewise, immersive technologies, like augmented and virtual reality devices, provide excellent potential for exploring and explaining data, thus inviting research on how data-driven storytelling transfers to immersive environments. To gain a better understanding of this exciting novel research area, we conducted a scoping review on the emerging notion of immersive data-driven storytelling, extended by surveying immersive data journalism and by analyzing immersive games, selected based on community reviews and tags. We present our methodology for the survey and discuss prominent themes that coalesce the knowledge we extracted from the literature, journalism, and games. These themes include, among others, the spatial embodiment of narration, the incorporation of the users and their context into narratives, and the balance between guiding the user versus promoting serendipity. Our discussion of these themes reveals research opportunities and challenges that will inform the design of immersive data-driven stories in the future.

In this paper, we examine the potential of incorporating transparent, handheld devices into head-mounted Augmented Reality (AR). Additional mobile devices have long been successfully used in head-mounted AR, but they obscure the visual context and real world objects during interaction. Transparent tangible displays can address this problem, using either transparent OLED screens or rendering by the head-mounted display itself. However, so far, there is no systematic analysis of the use of such transparent tablets in combination with AR head-mounted displays (HMDs), with respect to their benefits and arising challenges. We address this gap by introducing a research platform based on a touch-enabled, transparent interaction panel, for which we present our custom hardware design and software stack in detail. Furthermore, we developed a series of interaction concepts for this platform and demonstrate them in the context of three use case scenarios: the exploration of 3D volumetric data, collaborative visual data analysis, and the control of smart home appliances. We validate the feasibility of our concepts with interactive prototypes that we used to elicit feedback from HCI experts. As a result, we contribute to a better understanding of how transparent tablets can be integrated into future AR environments.

In this paper, we present MIRIA, a Mixed Reality Interaction Analysis toolkit designed to support the in-situ visual analysis of user interaction in mixed reality and multi-display environments. So far, there are few options to effectively explore and analyze interaction patterns in such novel computing systems. With MIRIA, we address this gap by supporting the analysis of user movement, spatial interaction, and event data by multiple, co-located users directly in the original environment. Based on our own experiences and an analysis of the typical data, tasks, and visualizations used in existing approaches, we identify requirements for our system. We report on the design and prototypical implementation of MIRIA, which is informed by these requirements and offers various visualizations such as 3D movement trajectories, position heatmaps, and scatterplots. To demonstrate the value of MIRIA for real-world analysis tasks, we conducted expert feedback sessions using several use cases with authentic study data.

Immersive Analytics is a quickly evolving field that unites several areas such as visualisation, immersive environments, and human-computer interaction to support human data analysis with emerging technologies. This research has thrived over the past years with multiple workshops, seminars, and a growing body of publications, spanning several conferences. Given the rapid advancement of interaction technologies and novel application domains, this paper aims toward a broader research agenda to enable widespread adoption. We present 17 key research challenges developed over multiple sessions by a diverse group of 24 international experts, initiated from a virtual scientific workshop at ACM CHI 2020. These challenges aim to coordinate future work by providing a systematic roadmap of current directions and impending hurdles to facilitate productive and effective applications for Immersive Analytics.

We present Marvis, a conceptual framework that combines mobile devices and head-mounted Augmented Reality (AR) for visual data analysis. We propose novel concepts and techniques addressing visualization-specific challenges. By showing additional 2D and 3D information around and above displays, we extend their limited screen space. AR views between displays as well as linking and brushing are also supported, making relationships between separated visualizations plausible. We introduce the design process and rationale for our techniques. To validate Marvis’ concepts and show their versatility and widespread applicability, we describe six implemented example use cases. Finally, we discuss insights from expert hands-on reviews. As a result, we contribute to a better understanding of how the combination of one or more mobile devices with AR can benefit visual data analysis. By exploring this new type of visualization environment, we hope to provide a foundation and inspiration for future mobile data visualizations.

In this paper, we investigate mobile devices as interactive controllers to support the exploration of 3D data spaces in head-mounted Augmented Reality (AR). In future mobile contexts, applications such as immersive analysis or ubiquitous information retrieval will involve large 3D data sets, which must be visualized in limited physical space. This necessitates efficient interaction techniques for 3D panning and zooming. Smartphones as additional input devices are promising because they are familiar and widely available in mobile usage contexts. They also allow more casual and discreet interaction compared to free-hand gestures or voice input. We introduce smartphone-based pan & zoom techniques for 3D data spaces and present a user study comparing five techniques. Our results show that spatial device gestures can outperform both touch-based techniques and hand gestures in terms of task completion times and user preference. We discuss our findings in detail and suggest suitable techniques for specific AR navigation tasks.

Today, the widespread use of mobile devices allows users to search information "on the go»», whenever and wherever they want, no longer confining Information Retrieval to classic desktop interfaces. We believe that technical advances in Augmented Reality will allow Information Retrieval to go even further, making use of both the users» surroundings and their abilities to interact with the physical world. In this paper, we present the fundamental concept of Reality-Based Information Retrieval, which combines the classic Information Retrieval process with Augmented Reality technologies to provide context-dependent search cues and situated visualizations of the query and the results. With information needs often stemming from real-world experiences, this novel combination has the potential to better support both Just-in-time Information Retrieval and serendipity. Based on extensive literature research, we propose a conceptual framework for Reality-Based Information Retrieval. We illustrate and discuss this framework and present two prototypical implementations, which we tested in small user studies. They demonstrate the feasibility of our concepts and inspired our discussion of notable challenges for further research in this novel and promising area.

Three-dimensional visualizations employing traditional input and output technologies have well-known limitations. Immersive technologies, natural interaction techniques, and recent developments in data physicalization may help to overcome these issues. In this context, we are specifically interested in the usage of spatial interaction with mobile devices for improved 3D visualizations. To contribute to a better understanding of this interaction style, we implemented example visualizations on a spatially-tracked tablet and investigated their usage and potential. In this paper, we report on a qualitative study comparing spatial interaction with inplace 3D visualizations to classic touch interaction regarding typical visualization tasks: navigation of unknown datasets, comparison of individual data objects, and the understanding and memorization of structures in the data. We identify several distinct usage patterns and derive recommendations for using spatial interaction in 3D data visualization.

In this paper, we demonstrate IllumiPaper: a system that provides new forms of paper-integrated visual feedback and enables multiple input channels to enhance digital paper applications. We aim to take advantage of traditional form sheets, including their haptic qualities, simplicity, and archivability, and simultaneously integrate rich digital functionalities such as dynamic status queries, real-time notifications, and visual feedback for widget controls. Our approach builds on emerging, novel paper-based technologies. We describe a fabrication process that allow us to directly integrate segment-based displays, touch and flex sensors, as well as digital pen input on the paper itself. With our fully functional research platform we demonstrate an interactive prototype for an industrial form-filling maintenance application to service computer networks that covers a wide range of typical paper-related tasks.

By now, mobile 3D interaction is often limited to simple multi-touch input on standard devices and less expressive or hard to use. We present the concept of mobile dual-display devices that can be folded for the exploration of 3D content. We examine different display modes and introduce new presentation and 3D interaction techniques that make use of the special form factor and the added input modality of folding two displays. In particular, we also consider the advantages of our proposed device for head-coupled perspective rendering -- virtually extending the view and providing independent perspectives for two users.

At present, photorealistic augmentation is not yet possible since the computational power of mobile devices is insufficient. Even streaming solutions from stationary PCs cause a latency that affects user interactions considerably. Therefore, we introduce a differential rendering method that allows for a consistent illumination of the inserted virtual objects on mobile devices, avoiding delays. The computation effort is shared between a stationary PC and the mobile devices to make use of the capacities available on both sides. The method is designed such that only a minimum amount of data has to be transferred asynchronously between the participants. This allows for an interactive illumination of virtual objects with a consistent appearance under both temporally and spatially varying real illumination conditions. To describe the complex near-field illumination in an indoor scenario, HDR video cameras are used to capture the illumination from multiple directions. In this way, sources of illumination can be considered that are not directly visible to the mobile device because of occlusions and the limited field of view. While our method focuses on Lambertian materials, we also provide some initial approaches to approximate non-diffuse virtual objects and thereby allow for a wider field of application at nearly the same cost.

As a novel class of mobile devices with rich interaction capabilities we introduce tPads -- transparent display tablets. tPads are the result of a systematic design investigation into the ways and benefits of interacting with transparent mobiles which goes beyond traditional mobile interactions and augmented reality (AR) applications. Through a user-centered design process we explored interaction techniques for transparent-display mobiles and classified them into four categories: overlay, dual display & input, surface capture and model-based interactions. We investigated the technical feasibility of such interactions by designing and building two touch-enabled semi-transparent tablets called tPads and a range of tPad applications. Further, a user study shows that tPad interactions applied to everyday mobile tasks (application switching and image capture) outperform current mobile interactions and were preferred by users. Our hands-on design process and experimental evaluation demonstrate that transparent displays provide valuable interaction opportunities for mobile devices.

In many cases, Tangible User Interfaces allow the manipulation of digital content with physical objects recognized by an interactive tabletop. Usually, such tangible objects are made of opaque wood or synthetic materials, thereby occluding the display. In this paper, we systematically investigate the promising potential of tangibles entirely made of transparent or translucent materials. Besides visualizing content directly below a manipulable tangible, transparent objects also facilitate direct touch interaction with the content below, dynamic illumination and glowing effects. We propose a comprehensive design space for transparent tangibles on tabletops based on a thorough review of existing work. By reporting on our own experiments and prototypes, we address several gaps in this design space, regarding aspects of both interaction and visualization. These include the illumination of tangibles as well as the precise input with transparent tangibles for which we also present the promising results of an initial user study. Finally, benefits and shortcomings of transparent tangibles are discussed and resulting design considerations are presented.

Mobile devices become more and more important today, especially for augmented reality (AR) applications in which the camera of the mobile device acts like a window into the mixed reality world. Up to now, no photorealistic augmentation is possible since the computational power of the mobile devices is still too weak. Even a streaming solution from a stationary PC would cause a latency that affects user interactions considerably. Therefore, we introduce a differential illumination method that allows for a consistent illumination of the inserted virtual objects on mobile devices, avoiding a delay. The necessary computation effort is shared between a stationary PC and the mobile devices to make use of the capacities available on both sides. The method is designed such that only a minimum amount of data has to be transferred asynchronously between the stationary PC and one or multiple mobile devices. This allows for an interactive illumination of virtual objects with a consistent appearance under both temporally and spatially varying real illumination conditions. To describe the complex near-field illumination in an indoor scenario, multiple HDR video cameras are used to capture the illumination from multiple directions. In this way, sources of illumination can be considered that are not directly visible to the mobile device because of occlusions and the limited field of view of built-in cameras.