A truly anytime, anywhere approach to immersive analytics is predicated on the ability for users to author and explore visualizations in an immersive fashion and without a desktop computer. In this paper, we present a generalizable visual programming technique for immersive authoring, exploration, and presentation of data visualizations through direct manipulation, and implement it in a web-based augmented reality (AR) system. In a preregistered user study, we use this implementation to explore how analysts author and use data visualizations in immersive AR accessed using a head-mounted display (HMD). The study focuses on the authoring technique, spatial organization strategies, and sensemaking and presentation practices in such immersive analytics environments for HMD-based AR. Our findings suggest that participants particularly liked the direct manipulation of our approach, and that participants' spatial organization is not based on physical landmarks in the room.

We introduce Datamancer, a wearable device enabling bimanual gesture interaction across multi-display ubiquitous analytics environments. Datamancer addresses the gap in gesture-based interaction within data visualization settings, where current methods are often constrained by limited interaction spaces or the need for installing bulky tracking setups. Datamancer integrates a finger-mounted pinhole camera and a chest-mounted gesture sensor, allowing seamless selection and manipulation of visualizations on distributed displays. By pointing to a display, users can acquire the display and engage in various interactions, such as panning, zooming, and selection, using both hands. Our contributions include (1) an investigation of the design space of gestural interaction for physical ubiquitous analytics environments; (2) a prototype implementation of the Datamancer system that realizes this model; and (3) an evaluation of the prototype through demonstration of application scenarios, an expert review, and a user study.

Extended Reality (XR) head-mounted displays (HMDs) coupled with standard mouse and keyboard hardware present a unique opportunity to rethink traditional desktop productivity. In this paper, we propose WindowSpace, an XR-based window manager for placing 2D windows in immersive 3D space while using a traditional mouse and keyboard. WindowSpace is a web-based system supporting consumer-level HMD-AR that can display native 2D windows from standard operating systems using a native desktop streaming server based on a standards-compliant optimized WebRTC protocol. To facilitate interacting with these native applications, we investigate the design space for effectively controlling the cursor between windows arranged in immersive 3D space. We propose three pointing techniques in our design space: an off-screen cursor (EtherXR), a head-tracking based pointer (Face&Jump), and a novel technique based on switching screens by extending the mouse trajectory (BallisticXR). Our empirical study revealed that EtherXR performs slightly better overall; however, BallisticXR showed better accuracy. Notably, Face&Jump yielded slower and less accurate performance, suggesting that full gaze assistance may be necessary to support this approach effectively.

We introduce DashSpace, a live collaborative immersive and ubiquitous analytics (IA/UA) platform designed for handheld and head-mounted Augmented/Extended Reality (AR/XR) implemented using WebXR and open standards. To bridge the gap between existing web-based visualizations and the immersive analytics setting, DashSpace supports visualizing both legacy D3 and Vega-Lite visualizations on 2D planes, and extruding Vega-Lite specifications into 2.5D. It also supports fully 3D visual representations using the Optomancy grammar. To facilitate authoring new visualizations in immersive XR, the platform provides a visual authoring mechanism where the user groups specification snippets to construct visualizations dynamically. The approach is fully persistent and collaborative, allowing multiple participants-whose presence is shown using 3D avatars and webcam feeds-to interact with the shared space synchronously, both co-located and remotely. We present three examples of DashSpace in action: immersive data analysis in 3D space, synchronous collaboration, and immersive data presentations.

Backchannel communication, like whispering or instant messaging, is common in meetings and holds significant value. However, it remains largely unexplored in the context of hybrid meetings with co-located and remote participants. To address this gap, we derived unique challenges of backchannel communication in hybrid meetings through an interview study. These challenges inspired a new voice-based backchannel communication system, WhisperChannel, aimed to be inclusive and low effort. WhisperChannel enables users to whisper remotely to anyone in the meeting — similar to whispering to co-located seat neighbors. In a user study with three groups, each having two sessions of hybrid meetings, we investigated the inclusiveness and effort of WhisperChannel for both co-located and remote participants. We provide insights into the benefits and limitations of using remote whispering. We show that WhisperChannel helped remote and co-located participants feel more included while requiring low effort to use, however, also introduced new challenges in backchannel communication.

How can we let users adapt video-based meetings as easily as they rearrange furniture in a physical meeting room? We describe a design space for video conferencing systems that includes a five-step “ladder of tailorability,” from minor adjustments to live reprogramming of the interface. We then present Mirrorverse and show how it applies the principles of computational media to support live tailoring of video conferencing interfaces to accommodate highly diverse meeting situations. We present multiple use scenarios, including a virtual workshop, an online yoga class, and a stand-up team meeting to evaluate the approach and demonstrate its potential for new, remote meetings with fluid transitions across activities.

Computational media describes a vision of software, which, in contrast to application-centric software, is (1) malleable, so users can modify existing functionality, (2) computable, so users can run custom code, (3) distributable, so users can open documents across different devices, and (4) shareable, so users can easily share and collaborate on documents. Over the last ten years, the Webstrates and Codestrates projects aimed at realizing this vision of computational media. Webstrates is a server application that synchronizes the DOM of websites. Codestrates builds on top of Webstrates and adds an authoring environment, which blurs the use and development of applications. Grounded in a chronology of the development of Webstrates and Codestrates, we present eight tensions that we needed to balance during their development. We use these tensions as an analytical lens in three case studies and a game challenge in which participants created games using Codestrates. We discuss the results of the game challenge based on these tensions and present key takeaways for six of them. Finally, we present six lessons learned from our endeavor to realize the vision of computational media, demonstrating the balancing act of weighing the vision against the pragmatics of implementing a working system.

Most modern applications are immutable and turn-key despite the acknowledged benefits of empowering users to modify their software. Writing extensible software remains challenging, even for expert programmers. Reprogramming or extending existing software is often laborious or wholly blocked, requiring sophisticated knowledge of application architecture or setting up a development environment. We present Varv, a programming model representing reprogrammable interactive software as a declarative data structure. Varv defines interactive applications as a set of concepts that consist of a schema and actions. Applications in Varv support incremental modification, allowing users to reprogram through addition and selectively suppress, modify, or add behavior. Users can define high-level concepts, creating an abstraction layer and effectively a domain-specific language for their application domain, emphasizing reuse and modification. We demonstrate the reprogramming and collaboration capabilities of Varv in two case studies and illustrate how the event engine allows for extensive tooling support.

Nurses frequently transfer patients as part of their daily work. However, manual patient transfers pose a major risk to nurses’ health. Although the Kinaesthetics care conception can help address this issue, existing support to learn the concept is low. We present KiTT, a tablet-based system, to promote the learning of ergonomic patient transfers based on the Kinaesthetics care conception. KiTT supports the training of Kinaesthetics-based patient transfers by two nurses. The nurses are guided by the phases (i) interactive instructions, (ii) training of transfer conduct, and (iii) feedback and reflection. We evaluated KiTT with 26 nursing-care students in a nursing-care school. Our results indicate that KiTT provides a good subjective support for the learning of Kinaesthetics. Our results also suggest that KiTT can promote the ergonomically correct conduct of patient transfers while providing a good user experience adequate to the nursing-school context, and reveal how KiTT can extend existing practices.

The popularity of computational notebooks heralds a return of software as computational media rather than turn-key applications. We believe this software model has potential beyond supporting just the computationally literate. We studied a biomolecular nano-design lab that works on a current frontier of science - RNA origami - whose researchers depend on computational tools to do their work, yet are not trained as programmers. Using a participatory design process, we developed a computational labbook to concretise what computational media could look like, using the principles of computability, malleability, shareability, and distributability suggested by previous work. We used this prototype to co-reflect with the nanoscientists about how it could transform their practice. We report on the computational culture specific to this research area; the scientists' struggles managing their computational environments; and their subsequent disempowerment yet dependence. Lastly, we discuss the generative potential and limitations of the four design principles for the future of computational media.

Programming assignments in computer science courses are often processed in pairs or groups of students. While working together, students face several shortcomings in today's software: The lack of real-time collaboration capabilities, the setup time of the development environment, and the use of different devices or operating systems can hamper students when working together on assignments. Text processing platforms like Google Docs solve these problems for the writing process of prose text, and computational notebooks like Google Colaboratory for data analysis tasks. However, none of these platforms allows users to implement interactive applications. We deployed a web-based literate programming system for three months during an introductory course on application development to explore how collaborative programming practices unfold and how the structure of computational notebooks affect the development. During the course, pairs of students solved weekly programming assignments. We analyzed data from weekly questionnaires, three focus groups with students and teaching assistants, and keystroke-level log data to facilitate the understanding of the subtleties of collaborative programming with computational notebooks. Findings reveal that there are distinct collaboration patterns; the preferred collaboration pattern varied between pairs and even varied within pairs over the course of three months. Recognizing these distinct collaboration patterns can help to design future computational notebooks for collaborative programming assignments.

"Rebuild Palmyra?" is a multimedia exhibition about the ancient city of Palmyra, its destruction by Daesh, and the question of whether it should be rebuilt. As such, it tackles today's pressing question of how humanity should deal with the destruction of cultural heritage. In the design of the exhibition, we pursued the Blended Museum approach, in which we strive to seamlessly integrate interactive media into exhibition design to increase the overall visitor experience. In this work, we present the exhibition, which consists of four rooms. We focus on three interactive installations in which the topic of reconstructing Palmyra is mediated using new technologies such as 3D Printing, Augmented Reality, and Virtual Reality. The installations helped visitors in developing their own point of view on the question of rebuilding Palmyra. Lastly, we provide insights into the technical implementation of the installations and discuss the results of quantitative and qualitative evaluations.