Recent advancements in the conversational and social capabilities of generative AI (GenAI) have sparked interest in its role as an agent capable of actively participating in human-AI group discussions. Despite this momentum, we don’t fully understand how GenAI shapes conversational dynamics or how the interface design impacts its influence on the group. In this paper, we introduce interface-driven social prominence as a design lens for collaborative GenAI systems. We then present a GenAI-based conversational agent that can actively engage in spoken dialogue during video calls and design three distinct collaboration modes that vary the social prominence of the agent by manipulating its presence in the shared space and the degree of control users have over its participation. A mixed-methods within-subjects study, in which 18 dyads engaged in realistic discussions with a GenAI agent, offers empirical insights into how communication patterns and the collective negotiation of GenAI’s influence shift based on how it is embedded into the collaborative experience. Based on these findings, we outline design implications for supporting the coordination and critical engagement required in human-AI groups.

Gaze-based selection in XR requires visual confirmation due to eye-tracking limitations and target ambiguity in 3D contexts. Current designs for wide-FOV displays use world-locked, central overlays, which is not conducive to always-on AR glasses. This paper introduces PeriphAR (/peh-ree-faar/), a visualization technique that leverages peripheral vision for feedback during gaze-based selection on a monocular AR display. In a first user study, we isolated text, color, and shape properties of target objects to compare peripheral selection cues. Peripheral vision was more sensitive to color than shape, but this sensitivity rapidly declined at lower contrast. To preserve preattentive processing of color, we developed two strategies to enhance color in users’ peripheral vision. In a second user study, our strategy that maximized contrast of the target to the neighboring object with the most similar color was subjectively preferred. As proof of concept, we implemented PeriphAR in an end-to-end system to test performance with real-world object detection.

VR is often utilized for organizing virtual events such as meetings, conferences, and concerts; however, support for live production is lacking in most existing VR tools.We present XCam, a toolkit enabling mixed-initiative control over virtual camera systems-from fully manual control by users to increasingly automated, systemdriven control with minimal user intervention.XCam's architectural design separates the concerns of object tracking, camera motion, and scene transition, giving more degrees of freedom to operators who can adjust the level of automation along all three dimensions.We used XCam to conduct two studies: (1) interviews with six VR content creators probe into what aspects should and shouldn't be automated based on six applications developed with XCam; (2) three workshops with experts explore XCam's utility in live production of an interactive VR flm sequence, a lecture on cinematography, and an alumni meeting in social VR.Expert feedback from our studies suggests how to balance automation and control, and the opportunities and limits of future AI-driven tools.

Peer Reviewed

Although the point-and-select interaction method has been shown to lead to user and system-initiated errors, it is still prevalent in VR scenarios.Current solutions to facilitate selection interactions exist, however they do not address the challenges caused by targeting inaccuracy.To reduce the effort required to target objects, we developed a model that quickly detected targeting errors after they occurred.The model used implicit multimodal user behavioral data to identify possible targeting outcomes.Using a dataset composed of 23 participants engaged in VR targeting tasks, we then trained a deep learning model to differentiate between correct and incorrect targeting events within 0.5 seconds of a selection, resulting in an AUC-ROC of 0.9.The utility of this model was then evaluated in a user study with 25 participants that identified that participants recovered from more errors and faster when assisted by the model.These results advance our understanding of targeting errors in VR and facilitate the design of future intelligent error-aware systems.

iGYM is an augmented reality exercise game for inclusive play that allows people with and without wheelchairs to participate equally in a soccer game with a projected virtual field and ball. However, currently iGYM requires all players to be co-located and lacks capabilities for remote play. In this work, we describe a networked iGYM implementation that allows teams of players to play with each other remotely. The key networking challenge is meeting tight end-to-end latency requirements for interactive play over the Internet. We demonstrate a portable tabletop version of iGYM, implemented in Unity and ROS2, using a distributed authority model to keep track of ownership and propagate state updates about players, game objects, and scores. Each client receives local player tracking updates and spawns player peripersonal circles under its own authority; a shared ball object is synchronized under server ownership with clientside interpolation. The demo GUI will let attendees inject artificial network delay to explore its impact on interactivity.

Evaluation lkthrough with Privacy Experts manner?How can we optimize negotiations of AR sensing capabilities to balance multiple usersʼ UX and privacy needs in a fine-grained manner?How could we mediate negotiations of AR sensing capabilities to balance multiple individualsʼ UX and privacy needs in a fine-grained manner?Figure 1: Augmented reality glasses pose privacy risks for co-located individuals, but today, their use in public spaces is governed solely by the wearer.Our work explores how to facilitate multi-user negotiations of AR sensing capabilities, formulating this process as an optimization approach to maintain core AR functionality while achieving a balance, or Equilibrium, with privacy.

While generative artificial intelligence (GenAI) is finding increased adoption in workplaces, current tools are primarily designed for individual use. Prior work established the potential for these tools to enhance personal creativity and productivity towards shared goals; however, we don't know yet how to best take into account the nuances of group work and team dynamics when deploying GenAI in work settings. In this paper, we investigate the potential of collaborative GenAI agents to augment teamwork in synchronous group settings through an exploratory study that engaged 25 professionals across 6 teams in speculative design workshops and individual follow-up interviews. Our workshops included a mixed reality provotype to simulate embodied collaborative GenAI agents capable of actively participating in group discussions. Our findings suggest that, if designed well, collaborative GenAI agents offer valuable opportunities to enhance team problem-solving by challenging groupthink, bridging communication gaps, and reducing social friction. However, teams' willingness to integrate GenAI agents depended on its perceived fit across a number of individual, team, and organizational factors. We outline the key design tensions around agent representation, social prominence, and engagement and highlight the opportunities spatial and immersive technologies could offer to modulate GenAI influence on team outcomes and strike a balance between augmentation and agency.

We introduce SonoHaptics, an audio-haptic cursor for gaze-based 3D object selection. SonoHaptics addresses challenges around providing accurate visual feedback during gaze-based selection in Extended Reality (XR), e. g., lack of world-locked displays in no- or limited-display smart glasses and visual inconsistencies. To enable users to distinguish objects without visual feedback, SonoHaptics employs the concept of cross-modal correspondence in human perception to map visual features of objects (color, size, position, material) to audio-haptic properties (pitch, amplitude, direction, timbre). We contribute data-driven models for determining cross-modal mappings of visual features to audio and haptic features, and a computational approach to automatically generate audio-haptic feedback for objects in the user’s environment. SonoHaptics provides global feedback that is unique to each object in the scene, and local feedback to amplify differences between nearby objects. Our comparative evaluation shows that SonoHaptics enables accurate object identification and selection in a cluttered scene without visual feedback.

Dark patterns are deceptive designs that influence a user’s interactions with an interface to benefit someone other than the user. Prior work has identified dark patterns in windows, icons, menus, and pointer (WIMP) interfaces and ubicomp environments, but how dark patterns can manifest in Augmented and Virtual Reality (collectively XR) requires more attention. We therefore conducted 10 co-design workshops with 20 experts in XR and deceptive design. Our participants co-designed 42 scenarios containing dark patterns, based on application archetypes presented in recent HCI/XR literature. In the co-designed scenarios, we identified 10 novel dark patterns in addition to 39 existing ones, as well as 10 examples in which specific characteristics associated with XR potentially amplified the effect dark patterns could have on users. Based on our findings and prior work, we present a classification of XR-specific properties that facilitate dark patterns: perception, spatiality, physical/virtual barriers, and XR device sensing. We also present the experts’ assessments of the likelihood and severity of the co-designed scenarios and highlight key aspects they considered for this evaluation, for example, technological feasibility, ease of upscaling and distributing malicious implementations, and the application’s context of use. Finally, we discuss means to mitigate XR dark patterns and support regulatory bodies to reduce potential harms.

The HCI community has explored new interaction designs for collaborative AR interfaces in terms of usability and feasibility; however, security & privacy (S&P) are often not considered in the design process and left to S&P professionals. To produce interaction proposals with S&P in mind, we extend the user-driven elicitation method with a scenario-based approach that incorporates a threat model involving access control in multi-user AR. We conducted an elicitation study in two conditions, pairing AR/AR experts in one condition and AR/S&P experts in the other, to investigate the impact of each pairing. We contribute a set of expert-elicited interactions for sharing AR content enhanced with access control provisions, analyze the benefits and tradeoffs of pairing AR and S&P experts, and present recommendations for designing future multi-user AR interactions that better balance competing design goals of usability, feasibility, and S&P in collaborative AR.

While current augmented reality (AR) authoring tools lower the technical barrier for novice AR designers, they lack explicit guidance to consider potentially harmful aspects of AR with respect to security & privacy (S&P). To address potential threats in the earliest stages of AR design, we developed Reframe, a digital storyboarding tool for designers with no formal training to analyze S&P threats. We accomplish this through a frame-based authoring approach, which captures and enhances storyboard elements that are relevant for threat modeling, and character-driven analysis tools, which personify S&P threats from an underlying threat model to provide simple abstractions for novice AR designers. Based on evaluations with novice AR designers and S&P experts, we find that Reframe enables designers to analyze threats and propose mitigation techniques that experts consider good quality. We discuss how Reframe can facilitate collaboration between designers and S&P professionals and propose extensions to Reframe to incorporate additional threat models.

Research on enabling novice AR/VR developers has emphasized the need to lower the technical barriers to entry. This is often achieved by providing new authoring tools that provide simpler means to implement XR interactions through abstraction. However, novices are then bound by the ceiling of each tool and may not form the correct mental model of how interactions are implemented. We present XRSpotlight, a system that supports novices by curating a list of the XR interactions defined in a Unity scene and presenting them as rules in natural language. Our approach is based on a model abstraction that unifies existing XR toolkit implementations. Using our model, XRSpotlight can find incomplete specifications of interactions, suggest similar interactions, and copy-paste interactions from examples using different toolkits. We assess the validity of our model with professional VR developers and demonstrate that XRSpotlight helps novices understand how XR interactions are implemented in examples and apply this knowledge in their projects.

Augmented Reality (AR) in human rights museums and memorialization efforts can empower these initiatives to create stronger connections between audiences and victims; however, there is little research on the risks of depicting sensitive narratives through immersive technologies. We examined the opportunities and challenges of applying AR to memorialization by designing and deploying an AR application with a human rights museum in Colombia. We report lessons from our collaboration about navigating the risk of re-victimizing testimonial authors while creating engaging AR interactions. Furthermore, we report on a user study where participants interacted with our museum exhibition. Based on observations of our co-design process and the user study results, we discuss implications for immersive application design with strategies for selecting immersive content, balancing audience engagement, and identifying technology gaps. Finally, we reflect on the implications for collaborations between HCI researchers, human rights professionals, and organizations to inform designs involving sensitive narratives.

Current research in augmented, virtual, and mixed reality (XR) reveals a lack of tool support for designing and, in particular, prototyping XR applications. While recent tools research is often motivated by studying the requirements of non-technical designers and end-user developers, the perspective of industry practitioners is less well understood. In an interview study with 17 practitioners from different industry sectors working on professional XR projects, we establish the design practices in industry, from early project stages to the final product. To better understand XR design challenges, we characterize the different methods and tools used for prototyping and describe the role and use of key prototypes in the different projects. We extract common elements of XR prototyping, elaborating on the tools and materials used for prototyping and establishing different views on the notion of fidelity. Finally, we highlight key issues for future XR tools research.

Prior work has demonstrated augmented reality’s benefits to education, but current tools are difficult to integrate with traditional instructional methods. We present Paper Trail, an immersive authoring system designed to explore how to enable instructors to create AR educational experiences, leaving paper at the core of the interaction and enhancing it with various forms of digital media, animations for dynamic illustrations, and clipping masks to guide learning. To inform the system design, we developed five scenarios exploring the benefits that hand-held and head-worn AR can bring to STEM instruction and developed a design space of AR interactions enhancing paper based on these scenarios and prior work. Using the example of an AR physics handout, we assessed the system’s potential with PhD-level instructors and its usability with XR design experts. In an elicitation study with high-school teachers, we study how Paper Trail could be used and extended to enable flexible use cases across various domains. We discuss benefits of immersive paper for supporting diverse student needs and challenges for making effective use of AR for learning.

Despite significant improvements to Virtual Reality (VR) technologies, most VR displays are fixed focus and depth perception is still a key issue that limits the user experience and the interaction performance. To supplement humans’ inherent depth cues (e.g., retinal blur, motion parallax), we investigate users’ perceptual mappings of distance to virtual objects’ appearance to generate visual cues aimed to enhance depth perception. As a first step, we explore color-to-depth mappings for virtual objects so that their appearance differs in saturation and value to reflect their distance. Through a series of controlled experiments, we elicit and analyze users’ strategies of mapping a virtual object’s hue, saturation, value and a combination of saturation and value to its depth. Based on the collected data, we implement a computational model that generates color-to-depth mappings fulfilling adjustable requirements on confusion probability, number of depth levels, and consistent saturation/value changing tendency. We demonstrate the effectiveness of color-to-depth mappings in a 3D sketching task, showing that compared to single-colored targets and strokes, with our mappings, the users were more confident in the accuracy without extra cognitive load and reduced the perceived depth error by 60.8%. We also implement four VR applications and demonstrate how our color cues can benefit the user experience and interaction performance in VR.

Augmented Reality (AR) has the potential to leverage environmental information to better facilitate distributed collaboration, however, such applications are difficult to develop. We present XSpace, a toolkit for creating spatially-aware AR applications for distributed collaboration. Based on a review of existing applications and developer tools, we design XSpace to support three methods for creating shared virtual spaces, each emphasizing a different aspect: shared objects, user perspectives, and environmental meshes. XSpace implements these methods in a developer toolkit, and also provides a set of complimentary visual authoring tools to allow developers to preview a variety of configurations for a shared virtual space. We present five example applications to illustrate that XSpace can support the development of a rich set of collaborative AR experiences that are difficult to produce with current solutions. Through XSpace, we discuss implications for future application design, including user space customization and privacy and safety concerns when sharing users' environments.

There is increased interest in using virtual reality in education, but it often remains an isolated experience that is difficult to integrate into current instructional experiences. In this work, we adapt virtual production techniques from filmmaking to enable mixed reality capture of instructors so that they appear to be standing directly in the virtual scene. We also capitalize on the growing popularity of live streaming software for video conferencing and live production. With XRStudio, we develop a pipeline for giving lectures in VR, enabling live compositing using a variety of presets and real-time output to traditional video and more immersive formats. We present interviews with media designers experienced in film and MOOC production that informed our design. Through walkthrough demonstrations of XRStudio with instructors experienced with VR, we learn how it could be used in a variety of domains. In end-to-end evaluations with students, we analyze and compare differences of traditional video vs. more immersive lectures with XRStudio.

While augmented and virtual reality technologies are becoming mainstream, it is still technically challenging and time-consuming to create new applications. Many designers draw from traditional low-fidelity prototyping methods that do not lend themselves well to designing in 3D. Developers use high-end programming frameworks such as Unity and Unreal which require significant hardware/software setups and coding skills. We see a gap in the medium-fidelity range where there is an opportunity for new tools to leverage the advantages of 360° content for AR/VR prototyping. Existing tools, however, have only limited support for 3D geometry, spatial and proxemic interactions, puppeteering, and storytelling. We present 360theater, a new method and a tool for rapid prototyping of AR/VR experiences, which takes dioramas into the virtual realm by enhancing 360° video capture with 3D geometry and simulating spatial interactions via Wizard of Oz. Our comparative evaluation of techniques with novice and experienced AR/VR designers shows that 360theater can close the gap and achieve a higher fidelity and more realistic AR/VR prototypes than comparable methods.

Augmented Reality (AR) and Virtual Reality (VR) devices are becoming easier to access and use, but the barrier to entry for creating AR/VR applications remains high. Although the recent spike in HCI research on novel AR/VR tools is promising, we lack insights into how AR/VR creators use today's state-of-the-art authoring tools as well as the types of challenges that they face. We interviewed 21 AR/VR creators, which we grouped into hobbyists, domain experts, and professional designers. Despite having a variety of motivations and skillsets, they described similar challenges in designing and building AR/VR applications. We synthesize 8 key barriers that AR/VR creators face nowadays, starting from prototyping the initial experiences to dealing with "the many unknowns" during implementation, to facing difficulties in testing applications. Based on our analysis, we discuss the importance of considering end-user developers as a growing population of AR/VR creators, how we can build learning opportunities into AR/VR tools, and the need for building AR/VR toolchains that integrate debugging and testing.

Immersive authoring is an increasingly popular technique to design AR/VR scenes because design and testing can be done concurrently. Most existing systems, however, are single-user and limited to either AR or VR, thus constrained in the interaction techniques. We present XRDirector, a role-based collaborative immersive authoring system that enables designers to freely express interactions using AR and VR devices as puppets to manipulate virtual objects in 3D physical space. In XRDirector, we adapt roles known from filmmaking to structure the authoring process and help coordinate multiple designers in immersive authoring tasks. We study how novice AR/VR creators can take advantage of the roles and modes in XRDirector to prototype complex scenes with animated 3D characters, light effects, and camera movements, and also simulate interactive system behavior in a Wizard of Oz style. XRDirector's design was informed by case studies around complex 3D movie scenes and AR/VR games, as well as workshops with novice AR/VR creators. We show that XRDirector makes it easier and faster to create AR/VR scenes without the need for coding, characterize the issues in coordinating designers between AR and VR, and identify the strengths and weaknesses of each role and mode to mitigate the issues.

Significant tool support exists for the development of mixed reality (MR) applications; however, there is a lack of tools for analyzing MR experiences. We elicit requirements for future tools through interviews with 8 university research, instructional, and media teams using AR/VR in a variety of domains. While we find a common need for capturing how users perform tasks in MR, the primary differences were in terms of heuristics and metrics relevant to each project. Particularly in the early project stages, teams were uncertain about what data should, and even could, be collected with MR technologies. We designed the Mixed Reality Analytics Toolkit (MRAT) to instrument MR apps via visual editors without programming and enable rapid data collection and filtering for visualizations of MR user sessions. With MRAT, we contribute flexible interaction tracking and task definition concepts, an extensible set of heuristic techniques and metrics to measure task success, and visual inspection tools with in-situ visualizations in MR. Focusing on a multi-user, cross-device MR crisis simulation and triage training app as a case study, we then show the benefits of using MRAT, not only for user testing of MR apps, but also performance tuning throughout the design process.

We explore 360 paper prototyping to rapidly create AR/VR prototypes from paper and bring them to life on AR/VR devices. Our approach is based on a set of emerging paper prototyping templates specifically for AR/VR. These templates resemble the key components of many AR/VR interfaces, including 2D representations of immersive environments, AR marker overlays and face masks, VR controller models and menus, and 2D screens and HUDs. To make prototyping with these templates effective, we developed 360proto, a suite of three novel physical--digital prototyping tools: (1) the 360proto Camera for capturing paper mockups of all components simply by taking a photo with a smartphone and seeing 360-degree panoramic previews on the phone or stereoscopic previews in Google Cardboard; (2) the 360proto Studio for organizing and editing captures, for composing AR/VR interfaces by layering the captures, and for making them interactive with Wizard of Oz via live video streaming; (3) the 360proto App for running and testing the interactive prototypes on AR/VR capable mobile devices and headsets. Through five student design jams with a total of 86 participants and our own design space explorations, we demonstrate that our approach with 360proto is useful to create relatively complex AR/VR applications.

What is Mixed Reality (MR)? To revisit this question given the many recent developments, we conducted interviews with ten AR/VR experts from academia and industry, as well as a literature survey of 68 papers. We find that, while there are prominent examples, there is no universally agreed on, one-size-fits-all definition of MR. Rather, we identified six partially competing notions from the literature and experts' responses. We then started to isolate the different aspects of reality relevant for MR experiences, going beyond the primarily visual notions and extending to audio, motion, haptics, taste, and smell. We distill our findings into a conceptual framework with seven dimensions to characterize MR applications in terms of the number of environments, number of users, level of immersion, level of virtuality, degree of interaction, input, and output. Our goal with this paper is to support classification and discussion of MR applications' design and provide a better means to researchers to contextualize their work within the increasingly fragmented MR landscape.

Many recent studies have explored user-defined interactions for touch and gesture-based systems through end-user elicitation. While these studies have facilitated the user-end of the human-computer dialogue, the subsequent design of gesture representations to communicate gestures to the user vary in style and consistency. Our study explores how users interpret, enact, and refine gesture representations adapting techniques from recent elicitation studies. To inform our study design, we analyzed gesture representations from 30 elicitation papers and developed a taxonomy of design elements. We then conducted a partnered elicitation study with 30 participants producing 657 gesture representations accompanied by think-aloud data. We discuss design patterns and themes that emerged from our analysis, and supplement these findings with an in-depth look at users' mental models when perceiving and enacting gesture representations. Finally, based on the results, we provide recommendations for practitioners in need of "visual language" guidelines to communicate possible user actions.

The latest generations of smartphones with built-in AR capabilities enable a new class of mobile apps that merge digital and real-world content depending on a user's task, context, and preference. But even experienced mobile app designers face significant challenges: creating 2D/3D AR content remains difficult and time-consuming, and current mobile prototyping tools do not support AR views. There are separate tools for this; however, they require significant technical skill. This paper presents ProtoAR which supplements rapid physical prototyping using paper and Play-Doh with new mobile cross-device multi-layer authoring and interactive capture tools to generate mobile screens and AR overlays from paper sketches, and quasi-3D content from 360-degree captures of clay models. We describe how ProtoAR evolved over four design jams with students to enable interactive prototypes of mobile AR apps in less than 90 minutes, and discuss the advantages and insights ProtoAR can give designers.

Designers and researchers often rely on simple gesture recognizers like Wobbrock et al.'s $1 for rapid user interface prototypes. However, most existing recognizers are limited to a particular input modality and/or pre-trained set of gestures, and cannot be easily combined with other recognizers. In particular, creating prototypes that employ advanced touch and mid-air gestures still requires significant technical experience and programming skill. Inspired by $1's easy, cheap, and flexible design, we present the GestureWiz prototyping environment that provides designers with an integrated solution for gesture definition, conflict checking, and real-time recognition by employing human recognizers in a Wizard of Oz manner. We present a series of experiments with designers and crowds to show that GestureWiz can perform with reasonable accuracy and latency. We demonstrate advantages of GestureWiz when recreating gesture-based interfaces from the literature and conducting a study with 12 interaction designers that prototyped a multimodal interface with support for a wide range of novel gestures in about 45 minutes.

Many web pages developed today require navigation by visual interaction-seeing, hovering, pointing, clicking, and dragging with the mouse over dynamic page content. These forms of interaction are increasingly popular as developer trends have moved from static, logically structured pages to dynamic, interactive pages. However, they are also often inaccessible to blind web users who tend to rely on keyboard-based screen readers to navigate the web. Despite existing web accessibility standards, engineering web pages to be equally accessible via both keyboard and visuomotor mouse-based interactions is often not a priority for developers. Improving access to this kind of visual and interactive web content has been a long-standing goal of HCI researchers, but the barriers have proven to be too varied and unpredictable to be overcome by some of the proposed solutions: promoting guidelines and best practices, automatically generating accessible versions of pre-exisiting web pages, or developing human-assisted solutions, such as screen and cursor-sharing, which tend to diminish an end user's agency. In this paper we present a real-time, collaborative approach to helping blind web users overcome inaccessible parts of existing web pages. We introduce *Arboretum*, a new architecture that enables any web user to seamlessly hand off controlled parts of their browsing session to remote users, while maintaining control over the interface via a "propose and accept/reject" mechanism. We illustrate the benefit of Arboretum by using it to implement *Arbility*, a browser that allows blind users to hand off targeted visual interaction tasks to remote crowd workers. We evaluate the entire system in a study with 9 blind web users, showing that Arbility allows them to interact with web content that was previously difficult to access via a screen reader alone.

360-degree video is increasingly used to create immersive user experiences; however, it is typically limited to a single user and not interactive. Recent studies have explored the potential of 360 video to support multi-user collaboration in remote settings. These studies identified several challenges with respect to 360 live streams, such as the lack of gaze awareness, out-of-sync views, and missed gestures. To address these challenges, we created 360Anywhere, a framework for 360 video-based multi-user collaboration that, in addition to allowing collaborators to view and annotate a 360 live stream, also supports projection of annotations in the 360 stream back into the real-world environment in real-time. This enables a range of collaborative augmented reality applications not supported with existing tools. We present the 360Anywhere framework and tools that allow users to generate applications tailored to specific collaboration and augmentation needs with support for remote collaboration. In a series of exploratory design sessions with users, we assess 360Anywhere's power and flexibility for three mobile ad-hoc scenarios. Using 360Anywhere, participants were able to set up and use fairly complex remote collaboration systems involving projective augmented reality in less than 10 minutes.

Augmented Reality (AR) developers face a proliferation of new platforms, devices, and frameworks. This often leads to applications being limited to a single platform and makes it hard to support collaborative AR scenarios involving multiple different devices. This paper presents XD-AR, a cross-device AR application development framework designed to unify input and output across hand-held, head-worn, and projective AR displays. XD-AR's design was informed by challenging scenarios for AR applications, a technical review of existing AR platforms, and a survey of 30 AR designers, developers, and users. Based on the results, we developed a taxonomy of AR system components and identified key challenges and opportunities in making them work together. We discuss how our taxonomy can guide the design of future AR platforms and applications and how cross-device interaction challenges could be addressed. We illustrate this when using XD-AR to implement two challenging AR applications from the literature in a device-agnostic way.

Several recent studies have highlighted the need to support parallel usage of multiple devices for cross-device use. Yet, most interfaces today are still designed for single-device use and require re-authoring to enable cross-device interaction. This paper presents two studies to inform the design of a new web browser with support for semi-automatic generation of cross-device interfaces. Based on the results of a recent study in which users manually customized web pages for cross-device use, our first study elicits from users how they might want to trigger popular cross-device patterns to transform single-device designs with relatively little effort. Our second study then examines how the emerging design patterns could be applied to the Alexa top 50 sites from 10 different genres. Based on these studies, we design semi-automatic techniques for page segmentation and distribution between multiple devices that can work on many existing web sites and require only minimal user input to switch between different cross-device designs. Finally, we discuss possible extensions to the Chrome web browser to make the techniques available for a wide range of desktop, mobile, and wearable devices, and successfully test them on popular web sites.

The ability to quickly interact with any nearby appliance from a mobile device would allow people to perform a wide range of one-time tasks (e.g., printing a document in an unfamiliar office location). However, users currently lack this capability, and must instead manually configure their devices for each appliance they want to use. To address this problem, we created Snap-To-It, a system that allows users to opportunistically interact with any appliance simply by taking a picture of it. Snap-To-It shares the image of the appliance a user wants to interact with over a local area network. Appliances then analyze this image (along with the user's location and device orientation) to see if they are being "selected," and deliver the corresponding control interface to the user's mobile device. Snap-To-It's design was informed by two technology probes that explored how users would like to select and interact with appliances using their mobile phone. These studies highlighted the need to be able to select hardware and software via a camera, and identified several novel use cases not supported by existing systems (e.g., interacting with disconnected objects, transferring settings between appliances). In this paper, we show how Snap-To-It's design is informed by our probes and how developers can utilize our system. We then show that Snap-To-It can identify appliances with over 95.3% accuracy, and demonstrate through a two-month deployment that our approach is robust to gradual changes to the environment.

There is a significant gap in the body of research on cross-device interfaces. Research has largely focused on enabling them technically, but when and how users want to use cross-device interfaces is not well understood. This paper presents an exploratory user study with XDBrowser, a cross-device web browser we are developing to enable non-technical users to adapt existing single-device web interfaces for cross-device use while viewing them in the browser. We demonstrate that an end-user customization tool like XDBrowser is a powerful means to conduct user-driven elicitation studies useful for understanding user preferences and design requirements for cross-device interfaces. Our study with 15 participants elicited 144 desirable multi-device designs for five popular web interfaces when using two mobile devices in parallel. We describe the design space in this context, the usage scenarios targeted by users, the strategies used for designing cross-device interfaces, and seven concrete mobile multi-device design patterns that emerged. We discuss the method, compare the cross-device interfaces from our users and those defined by developers in prior work, and establish new requirements from observed user behavior. In particular, we identify the need to easily switch between different interface distributions depending on the task and to have more fine-grained control over synchronization.

The physical constraints of smartwatches limit the range and complexity of tasks that can be completed. Despite interface improvements on smartwatches, the promise of enabling productive work remains largely unrealized. This paper presents WearWrite, a system that enables users to write documents from their smartwatches by leveraging a crowd to help translate their ideas into text. WearWrite users dictate tasks, respond to questions, and receive notifications of major edits on their watch. Using a dynamic task queue, the crowd receives tasks issued by the watch user and generic tasks from the system. In a week-long study with seven smartwatch users supported by approximately 29 crowd workers each, we validate that it is possible to manage the crowd writing process from a watch. Watch users captured new ideas as they came to mind and managed a crowd during spare moments while going about their daily routine. WearWrite represents a new approach to getting work done from wearables using the crowd.

In this tutorial, I will describe key technical and design challenges of developing cross-device interfaces, which are not only able to adapt to a wide variety of device characteristics and user preferences, but can actually migrate a user's tasks from one device to another and even distribute the interface between multiple devices jointly used by one or more users. Starting with a review of the cross-device design concepts, authoring tools and interaction techniques presented in the ISS and larger HCI literature, I will give a comprehensive overview of the state of the art. I will then analyze the current tools for design and development of cross-device interfaces, and present one of these tools, XDBrowser, in detail. Participants will get hands-on experience with designing cross-device versions of existing interfaces using such authoring tools created for non-technical users. Finally, I will discuss open issues and provide an outlook based on the results from recent cross-device workshops I co-organized or participated in at CHI, EICS, and DIS.

Personal cloud services such as Dropbox are used increasingly to support collaborative work, even though they typically have poor support for tracking files and users' activities and collaborators often rely on other communication channels to be notified of changes. We present a meta-cloud storage service, MUBox, that, independent of a particular cloud storage service, provides improved support for collaboration. First, users can switch to activity views that list user activities rather than files, which is an example of an increasingly available feature in popular cloud storage clients. Second, MUBox introduces multi-user aware folder views that embed information on the last changes performed by collaborators. These folder views are enhanced based on a new concept of shadow files which act as placeholders for files that have been moved or renamed. A user study (N=16) with realistic folder exploration tasks shows that activity views have a significant effect on the accuracy and confidence of users in workspace awareness tasks, while shadow files significantly improve the speed, accuracy and confidence of users in traceability tasks. We describe how existing services could implement these features as well as a new concept for voting on changes to shared folders that could improve asynchronous collaboration.

While there are many established methodologies for information systems development, designing by example has not been formally explored and applied previously. Our work is also motivated by the desire to explore interface-driven development techniques that could complement existing approaches such as model-driven engineering with the goal of reducing the need for modelling and reengineering of existing applications and interfaces, while still supporting the development task. We explore the example-based technique for rapid development of powerful and flexible information management tools based on the example of Adobe Photoshop Lightroom, a system that was originally designed to support the workflow of digital photographers in a flexible way. We analyse experiments in which two new systems---one for managing collections of research papers and another for software project management---were developed based on the Lightroom paradigm. We derive a conceptual framework for engineering by example and assess the method by comparing it to traditional model-driven engineering.

Current GUI builders provide a design environment for user interfaces that target either a single type or fixed set of devices, and provide little support for scenarios in which the user interface, or parts of it, are distributed over multiple devices. Distributed user interfaces have received increasing attention over the past years. There are different, often model-based, approaches that focus on technical issues. This paper presents XDStudio--a new GUI builder designed to support interactive development of cross-device web interfaces. XDStudio implements two complementary authoring modes with a focus on the design process of distributed user interfaces. First, simulated authoring allows designing for a multi-device environment on a single device by simulating other target devices. Second, on-device authoring allows the design process itself to be distributed over multiple devices, as design and development take place on the target devices themselves. To support interactive development for multi-device environments, where not all devices may be present at design and run-time, XDStudio supports switching between the two authoring modes, as well as between design and use modes, as required. This paper focuses on the design of XDStudio, and evaluates its support for two distribution scenarios.

Web designers currently face the increased proliferation and diversity of new touch devices which pose major challenges to the design task. This paper presents W3Touch - an interface instrumentation toolkit for web designers to collect user performance data for different device characteristics in order to help them identify potential design problems for touch interaction. Web designers can visualise the data aggregated by W3Touch and use simple metrics to automate the adaptation process for many different viewing and interaction contexts. In a series of experiments with web designers and users, we show that W3Touch is able to detect interaction problems that are hard to find using conventional methods and demonstrate how the tool was successfully used to automate the desktop-to-mobile migration of Wikipedia as an example.

Despite the fact that screen sizes and average screen resolutions have dramatically increased over the past few years, little attention has been paid to the design of web sites for large, high-resolution displays that are now becoming increasingly used both in enterprise and consumer spaces. We present a study of how the visual area of the browser window is currently utilised by news web sites at different widescreen resolutions. The analysis includes measurements of space taken up by the article content, embedded ads and the remaining components as they appear in the viewport of the web browser. The results show that the spatial distribution of page elements does not scale well with larger viewing sizes, which leads to an increasing amount of unused screen real estate and unnecessary scrolling. We derive a number of device-sensitive metrics to measure the quality of web page layout in different viewing contexts, which can guide the design of flexible layout templates that scale effectively on large screens.

Despite the fact that average screen size and resolution have dramatically increased, many of today's web sites still do not scale well in larger viewing contexts. The upcoming HTML5 and CSS3 standards propose features that can be used to build more flexible web page layouts, but their potential to accommodate a wider range of display environments is currently relatively unexplored. We examine the proposed standards to identify the most promising features and report on experiments with a number of adaptive layout mechanisms that support the required forms of adaptation to take advantage of greater screen real estates, such as automated scaling of text and media. Special attention is given to the effective use of multi-column layout, a brand new feature for web design that contributes to optimising the space occupied by text, but at the same time still poses problems in predominantly continuous vertical-scrolling browsing behaviours. The proposed solutions were integrated in a flexible layout template that was then applied to an existing news web site and tested on users to identify the adaptive features that best support reading comfort and efficiency.