Speculative and systemic design are both used by HCI researchers to engage in complex sociotechnical change. However, they are rarely integrated in ways that make their complementary strengths explicit. This paper introduces Systemic Futures Dialogue, a design approach that interleaves speculative and systemic design methods across micro-macro and present-future dimensions. We report on an 18-month case study with the Architecture, Engineering, and Construction (AEC) industry, that applied a mixture of methods used in both design disciplines. These included semi-structured interviews, systems mapping, future-based scenarios, speculative probes, and participatory reflection. The resulting design approach generates grounded futures by connecting macro-level system dynamics with micro-level speculative critique, identifying tensions between present-day solutions and desired futures. The final Systemic Futures Dialogue contributes methodological guidance for conducting critical, participatory design work within a sociotechnical system.

Visions for the future of computing, such as those on Ubiquitous Computing or Tangible Interfaces, are highly cited and frequently used in teaching. Yet, we know little about the practical value of these visions for research on Human-Computer Interaction (HCI) or how HCI researchers engage with them individually and collectively. To address this gap, we conducted a survey with 172 HCI researchers. We identified key benefits and pitfalls as well as specific uses of visions by researchers. Researchers appreciate how visions guide us, drive us, and initiate new fields. Simultaneously, researchers acknowledge how visions create hype, restrict our creativity, and make us disregard real-world problems. Based on these insights, we derive tensions related to the pursuit of visions and discuss critical reading practices. Our paper offers a metascientific account of visions in the HCI field along with tools for critical reflection when engaging with them.

We investigate hybrid user interfaces (HUIs), aiming to establish a cohesive understanding and to adopt consistent terminology for this nascent research area. HUIs combine heterogeneous devices in complementary roles, leveraging the distinct benefits of each. Our work focuses on cross-device interaction between 2D devices and mixed reality environments, which are particularly compelling, leveraging the familiarity of traditional 2D platforms while providing spatial awareness and immersion. Although prior work has prominently explored such HUIs in the context of mixed reality, we still lack a cohesive understanding of the unique design possibilities and challenges of such combinations, resulting in a fragmented research landscape. We conducted a systematic survey and present a taxonomy of HUIs that combine conventional display technology and mixed reality environments. Based on this, we discuss past and current challenges, the evolution of definitions, and prospective opportunities to tie together the past 30 years of research with our vision of future HUIs.

Hybrid meetings have become common practice in collaborative work environments. However, they are constrained by the fixed spatial configurations of videoconferencing technology. This limits opportunities for mobile and spontaneous interactions; qualities that are critical to successful collaboration. In this paper, we explore the concept of radically mobile hybrid meetings. Our work investigates the design space of multimodal devices as mobile alternatives to traditional videoconferencing. We conducted three group co-design sessions, where participants prototyped mobile hybrid meeting technologies to explore how such meetings could be supported. From these workshops, we derive design fictions envisioning future uses of these technologies, which we evaluate with a questionnaire to spark reflections on future mobile hybrid collaboration tools and practices. We contribute an initial exploration of the design space for radically mobile hybrid meetings, laying the groundwork for developing tools that enable spontaneous, effective, and inclusive collaboration in hybrid mobile settings.

Collaborative Mixed Reality (MR) enables embodied meetings for distributed collaborators working across a variety of locations. However, providing a coherent experience for all users regardless of the spatial configurations of their respective physical environments is a central challenge. We present the Spatial Heterogeneity Framework, which breaks the problem into four core components: the activity zones, heterogeneity ladder, blended proxemics, and MR solutions matrix. We explain the interplay between these components, demonstrating their interconnectivity via a case study. Our framework enables researchers to navigate differences and trade-offs between solutions for distributed MR collaboration. It also supports designers to think about the role of space, technology, and social behaviours in MR collaboration. Ultimately, our contributions advance the field by conceptualising the challenges of spatial heterogeneity and strategies to overcome them.

Collaboration in distributed mixed reality (MR) creates co-presence illusions by bringing a digital representation of the remote collaborator into the local user’s physical space. However, constraints in each physical environment, such as different spatial layouts and furniture, can hinder collaboration and momentarily break the illusion of co-location. Since the remote user’s physical space is invisible to the local user, the source of the interaction problem is often imperceptible, leading to familiar real-world interactions suddenly feeling unfamiliar—a concept we term the Jamais Vu Effect. In this paper, we conceptualise and demonstrate this effect through a five-part user study that elicits the user experience of the Jamais Vu Effect in MR. We contribute a shared vocabulary and understanding of spatial interface design challenges to help designers and researchers discuss and improve distributed collaboration in MR.

The whiteboard is essential for collaborative work. To preserve its physicality in remote collaboration, Mixed Reality (MR) can blend real whiteboards across distributed spaces. Going beyond reality, MR can further enable interactions like panning and zooming in a virtually reconfigurable infinite whiteboard. However, this reconfigurability conflicts with the sense of physicality. To address this tension, we introduce Blended Whiteboard, a remote collaborative MR system enabling reconfigurable surface blending across distributed physical whiteboards. Blended Whiteboard supports a unique collaboration style, where users can sketch on their local whiteboards but also reconfigure the blended space to facilitate transitions between loosely and tightly coupled work. We describe design principles inspired by proxemics; supporting users in changing between facing each other and being side-by-side, and switching between navigating the whiteboard synchronously and independently. Our work shows exciting benefits and challenges of combining physicality and reconfigurability in the design of distributed MR whiteboards.

Task switching can occur frequently in daily routines with physical activity. In this paper, we introduce Spatial Gaze Markers, an augmented reality tool to support users in immediately returning to the last point of interest after an attention shift. The tool is task-agnostic, using only eye-tracking information to infer distinct points of visual attention and to mark the corresponding area in the physical environment. We present a user study that evaluates the effectiveness of Spatial Gaze Markers in simulated physical repair and inspection tasks against a no-marker baseline. The results give insights into how Spatial Gaze Markers affect user performance, task load, and experience of users with varying levels of task type and distractions. Our work is relevant to assist physical workers with simple AR techniques and render task switching faster with less effort.

Backchannel communication, like whispering or instant messaging, is common in meetings and holds significant value. However, it remains largely unexplored in the context of hybrid meetings with co-located and remote participants. To address this gap, we derived unique challenges of backchannel communication in hybrid meetings through an interview study. These challenges inspired a new voice-based backchannel communication system, WhisperChannel, aimed to be inclusive and low effort. WhisperChannel enables users to whisper remotely to anyone in the meeting — similar to whispering to co-located seat neighbors. In a user study with three groups, each having two sessions of hybrid meetings, we investigated the inclusiveness and effort of WhisperChannel for both co-located and remote participants. We provide insights into the benefits and limitations of using remote whispering. We show that WhisperChannel helped remote and co-located participants feel more included while requiring low effort to use, however, also introduced new challenges in backchannel communication.

Collaborating across dissimilar, distributed spaces presents numerous challenges for computer-aided spatial communication. Mixed reality (MR) can blend selected surfaces, allowing collaborators to work in blended f-formations (facing formations), even when their workstations are physically misaligned. Since collaboration often involves more than just participant pairs, this research examines how we might scale MR experiences for large-group collaboration. To do so, this study recruited collaboration designers (CDs) to evaluate and reimagine MR for large-scale collaboration. These CDs were engaged in a four-part user study that involved a technology probe, a semi-structured interview, a speculative low-fidelity prototyping activity and a validation session. The outcomes of this paper contribute (1) a set of collaboration design principles to inspire future computer-supported collaborative work, (2) eight collaboration patterns for blended f-formations and collaboration at scale and (3) theoretical implications for f-formations and space-place relationships. As a result, this work creates a blueprint for scaling collaboration across distributed spaces.

Mixed Reality allows for distributed meetings where people’s local physical spaces are virtually aligned into blended interaction spaces. In many cases, people’s physical rooms are dissimilar, making it challenging to design a coherent blended space. We introduce the concept of Partially Blended Realities (PBR) — using Mixed Reality to support remote collaborators in partially aligning their physical spaces. As physical surfaces are central in collaborative work, PBR supports users in transitioning between different configurations of tables and whiteboard surfaces. In this paper, we 1) describe the design space of PBR, 2) present RealityBlender to explore interaction techniques for how users may configure and transition between blended spaces, and 3) provide insights from a study on how users experience transitions in a remote collaboration task. With this work, we demonstrate new potential for using partial solutions to tackle the alignment problem of dissimilar spaces in distributed Mixed Reality meetings.

Turn-taking is one of the biggest interactivity challenges in multiparty remote meetings. One contributing factor is that current videoconferencing tools lack support for proxemic cues; i.e., spatial cues that humans use to enact their social relations and intentions. While more recent tools provide support for proxemic metaphors, they often focus on approach and leave-taking rather than turn-taking. In this paper, we present OpenMic, a videoconferencing system that utilizes proxemic metaphors for conversational floor management by providing 1) a Virtual Floor that serves as a fixed-feature space for users to be aware of others’ intention to talk, and 2) Malleable Mirrors, which are video and screen feeds that can be continuously moved and resized for conversational floor transitions. Our exploratory user study found that these system features can aid the conversational flow in multiparty video meetings. With this work, we show potential for embedding proxemic metaphors to support conversational floor management in videoconferencing systems.

In video meetings, individuals may wish to share various physical objects with remote participants, such as physical documents, design prototypes, and personal belongings. However, our formative study discovered that this poses several challenges, including difficulties in referencing a remote user’s physical objects, the limited visibility of the object, and the friction of properly framing and orienting an object to the camera. To address these challenges, we propose ThingShare, a video-conferencing system designed to facilitate the sharing of physical objects during remote meetings. With ThingShare, users can quickly create digital copies of physical objects in the video feeds, which can then be magnified on a separate panel for focused viewing, overlaid on the user’s video feed for sharing in context, and stored in the object drawer for reviews. Our user study demonstrated that ThingShare made initiating object-centric conversations more efficient and provided a more stable and comprehensive view of shared objects.

Gaze-Hand Alignment has recently been proposed for multimodal selection in 3D. The technique takes advantage of gaze for target pre-selection, as it naturally precedes manual input. Selection is then completed when manual input aligns with gaze on the target, without need for an additional click method. In this work we evaluate two alignment techniques, Gaze&Finger and Gaze&Handray, combining gaze with image plane pointing versus raycasting, in comparison with hands-only baselines and Gaze&Pinch as established multimodal technique. We used Fitts’ Law study design with targets presented at different depths in the visual scene, to assess effect of parallax on performance. The alignment techniques outperformed their respective hands-only baselines. Gaze&Finger is efficient when targets are close to the image plane but less performant with increasing target depth due to parallax.

How can we let users adapt video-based meetings as easily as they rearrange furniture in a physical meeting room? We describe a design space for video conferencing systems that includes a five-step “ladder of tailorability,” from minor adjustments to live reprogramming of the interface. We then present Mirrorverse and show how it applies the principles of computational media to support live tailoring of video conferencing interfaces to accommodate highly diverse meeting situations. We present multiple use scenarios, including a virtual workshop, an online yoga class, and a stand-up team meeting to evaluate the approach and demonstrate its potential for new, remote meetings with fluid transitions across activities.

Augmented Reality (AR) has shown great potential for supporting co-located collaboration. Yet, it is rarely articulated in the design rationales of AR systems that they promote a certain socio-spatial configuration of the users. Learning from proxemics, we argue that such configurations enable and constrain different co-located spatial behaviors with consequences for collaborative activities. We focus specifically on enabling different collaboration styles via the design of Handheld Collaborative Augmented Reality (HCAR) systems. Drawing upon notions of proxemics, we show how different HCAR designs enable different socio-spatial configurations. Through a design exploration, we demonstrate interaction techniques to expand on the notion of collaborative coupling styles by either deliberately designing for aligning with physical reality or going beyond. The main contributions are a proxemics-based conceptual lens and vocabulary for supporting interaction designers in being mindful of the proxemic consequences when developing handheld multi-user AR systems.

There is a growing focus on computational thinking (CT) in terms of supporting children’s understanding of everyday technologies. But unlike other technologies, Augmented Reality (AR) has received limited attention. In this paper, we present ExposAR – a collaborative cross-device AR system enabling children to create, use, and reflect on AR technologies through the co-creation of a simple AR application. With ExposAR, we explore three core design principles: 1) reify computational concepts, 2) support collaborative cross-device authoring, and 3) incorporate the user’s own world. These principles were developed through a co-design process with teachers and evaluated with 46 primary school students. We found that the collaborative authoring process with ExposAR supported students in understanding AR concepts and challenged their perspectives on AR. With these results, we bring AR to the CT agenda and contribute novel design principles for exposing the underlying mechanisms and implications of AR.

With eye-tracking increasingly available in Augmented Reality, we explore how gaze can be used to assist freehand gestural text entry. Here the eyes are often coordinated with manual input across the spatial positions of the keys. Inspired by this, we investigate gaze-assisted selection-based text entry through the concept of spatial alignment of both modalities. Users can enter text by aligning both gaze and manual pointer at each key, as a novel alternative to existing dwell-time or explicit manual triggers. We present a text entry user study comparing two of such alignment techniques to a gaze-only and a manual-only baseline. The results show that one alignment technique reduces physical finger movement by more than half compared to standard in-air finger typing, and is faster and exhibits less perceived eye fatigue than an eyes-only dwell-time technique. We discuss trade-offs between uni and multimodal text entry techniques, pointing to novel ways to integrate eye movements to facilitate virtual text entry.

Gaze and freehand gestures suit Augmented Reality as users can interact with objects at a distance without need for a separate input device. We propose Gaze-Hand Alignment as a novel multimodal selection principle, defined by concurrent use of both gaze and hand for pointing and alignment of their input on an object as selection trigger. Gaze naturally precedes manual action and is leveraged for pre-selection, and manual crossing of a pre-selected target completes the selection. We demonstrate the principle in two novel techniques, Gaze&Finger for input by direct alignment of hand and finger raised into the line of sight, and Gaze&Hand for input by indirect alignment of a cursor with relative hand movement. In a menu selection experiment, we evaluate the techniques in comparison with Gaze&Pinch and a hands-only baseline. The study showed the gaze-assisted techniques to outperform hands-only input, and gives insight into trade-offs in combining gaze with direct or indirect, and spatial or semantic freehand gestures.

In hybrid meetings, multiple co-located participants communicate with remote participants through video. But video communication inhibits non-verbal cues, and this often causes remote participants to feel excluded. To address this issue, we built MirrorBlender: a What-You-See-Is-What-I-See video-conferencing system for blending, repositioning, and resizing mirrors. Mirrors here denote shared video feeds of people and screens. In a qualitative study of MirrorBlender with three hybrid meeting sessions, we found that the shared control of mirrors supported users in negotiating a blended interpersonal space. Moreover, it enabled diverse acts of inclusion of remote participants. In particular, remote participants brought attention to themselves by manipulating the position, scale, and translucency of their camera and screen feeds. Participants also embodied and leveraged their mirror images for deictic gestures and playful interactions. Based on these findings, we discuss new opportunities for supporting video-mediated collaboration.

Programming by Demonstration (PbD) is a well-known technique that allows non-programmers to describe interactivity by performing examples of the expected behavior, but it has not been extensively explored for AR. We present Rapido, a novel early-stage prototyping tool to create fully interactive mobile AR prototypes from non-interactive video prototypes using PbD. In Rapido, designers use a mobile AR device to record a video prototype to capture context, sketch assets, and demonstrate interactions. They can demonstrate touch inputs, animation paths, and rules to, e.g., have a sketch follow the focus area of the device or the user’s world-space touches. Simultaneously, a live website visualizes an editable overview of all the demonstrated examples and infers a state machine of the user flow. Our key contribution is a method that enables designers to turn a video prototype into an executable state machine through PbD. The designer switches between these representations to interactively refine the final interactive prototype. We illustrate the power of Rapido’s approach by prototyping the main interactions of three popular AR mobile applications.

Cross-device interactions enable ad hoc sharing of content and control in co-located collaboration. Cross-device research often draws from proxemics theory for designing interactions based on detection of spatial relations such as distance and orientation between people and devices. However, detection of human-human or human-device proximity also constrains flexibility in co-located social interaction. We suggest a proxemics-based approach to designing flexible cross-device interactions. From observations in a field study, we articulate how co-located sharing practices are shaped by the interplay between everyday mobile devices and the physical environment. Based on these insights, we present three cross-device prototypes as proofs-of-concept, demonstrating three design sensitivities for considering proxemics beyond proximity; incorporating features in the environment, enabling flexibility in interpersonal distance and orientation, and providing multiple alternative action possibilities. Drawing from characteristics of our prototypes, we discuss concrete proposals for designing cross-device interactions to enable flexible social interaction.

A core challenge in tabletop research is to support transitions between individual activities and team work. Shape-changing tabletops open up new opportunities for addressing this challenge. However, interaction design for shape-changing furniture is in its early stages - so far, research has mainly focused on triggering shape-changes, and less on the actual interface transitions. We present KirigamiTable - a novel actuated shape-changing tabletop for supporting transitions in collaborative work. Our work builds on the concept of Proxemic Transitions, considering the dynamic interplay between social interactions, interactive technologies and furniture. With KirigamiTable, we demonstrate the potential of interactions for proxemic transitions that combine transformation of shape and digital contents. We highlight challenges for shape-changing tabletops: initiating shape and content transformations, cooperative control, and anticipating shape-change. To address these challenges, we propose a set of novel interaction techniques, including shape-first and content-first interaction, cooperative gestures, and physical and digital preview of shape-changes.

Previous work has shown that large high resolution displays (LHRDs) can enhance collaboration between users. As LHRDs allow free movement in front of the screen, an understanding of movement behavior is required to build successful interfaces for these devices. This paper presents Pac-Many; a multiplayer version of the classical computer game Pac-Man to study group dynamics when using LHRDs. We utilized smartphones as game controllers to enable free movement while playing the game. In a lab study, using a 4m × 1m LHRD, 24 participants (12 pairs) played Pac-Many in collaborative and competitive conditions. The results show that players in the collaborative condition divided screen space evenly. In contrast, competing players stood closer together to avoid benefits for the other player. We discuss how the nature of the task is important when designing and analyzing collaborative interfaces for LHRDs. Our work shows how to account for the spatial aspects of interaction with LHRDs to build immersive experiences.

The field of Shape-Changing Interfaces explores the qualities of physically dynamic artifacts. At furniture-scale, such artifacts have the potential of changing the ways we collaborate and engage with interiors and physical spaces. Informed by theories of proxemics, empirical studies of informal meetings and design work with shape-changing furniture, we develop the notion of proxemic transitions. We present three design aspects of proxemic transitions: transition speed, stepwise reconfiguration, and radical shifts. The design aspects focus on how to balance between physical and digital transformations in designing for proxemic transitions. Our contribution is three-fold: 1) the notion of proxemic transitions, 2) three design aspects to consider in designing for proxemic transitions, and 3) initial exploration of how these design aspects might generate designs of dynamic furniture. These contributions outline important aspects to consider when designing shape-changing furniture for informal workplace meetings.

People inherently share spaces with other people. Congenitally, interactive technologies and ubiquitous environments shape our opportunities for enacting social relations. Proxemics and Spatial Sharing have been suggested as foundations for our understanding of the socio-spatial aspects of computing. By tandeming these theoretical perspectives in a set of cases in the office domain, we develop a contribution comprised of 3 key sensitizing concepts: Proxemic Malleability, Proxemic Threshold and Proxemic Gravity articulating socio-spatial qualities at the interplay between interactive systems, spaces, interior elements and co-located people. The sensitizing concepts qualify interaction designers in considering proxemic consequences of technology design; they serve both as analytic lenses and as generative instruments in a design process. The proposed sensitizing concepts and the theoretical work of the paper contribute to enhanced Socio-spatial literacy in HCI.

We propose combining shape-changing interfaces and spatial augmented reality for extending the space of appearances and interactions of actuated interfaces. While shape-changing interfaces can dynamically alter the physical appearance of objects, the integration of spatial augmented reality additionally allows for dynamically changing objects' optical appearance with high detail. This way, devices can render currently challenging features such as high frequency texture or fast motion. We frame this combination in the context of computer graphics with analogies to established techniques for increasing the realism of 3D objects such as bump mapping. This extensible framework helps us identify challenges of the two techniques and benefits of their combination. We utilize our prototype shape-changing device enriched with spatial augmented reality through projection mapping to demonstrate the concept. We present a novel mechanical distance-fields algorithm for real-time fitting of mechanically constrained shape-changing devices to arbitrary 3D graphics. Furthermore, we present a technique for increasing effective screen real estate for spatial augmented reality through view-dependent shape change.