We present Anu.js, a toolkit for web-based immersive analytics (IA). The IA design space is vast, multi-faceted, and everchanging, challenging development in the absence of robust authoring support. The web is a popular platform for visualization applications, research, and teaching, and by leveraging the benefits of web technologies and adopting imperative authoring paradigms we can achieve the necessary expressiveness, compatibility, and ergonomics to support IA research and development. Anu.js adapts D3’s data-binding model to 3D contexts, granting fine-grained control over the creation, representation, animation, performance, and interaction of 3D scene-graphs. Additionally, Anu.js offers declarative prefabs to support common visualization elements and interactions, and synergizes with popular visualization libraries which allows developers to leverage these proven utilities. We demonstrate Anu.js’s potential through our diverse example gallery, expert evaluation, and potential future applications. Through this, Anu.js empowers developers in accelerating the creation of novel and bespoke visualizations for immersive web-based applications.

Additive models of interaction performance, such as the Keystroke-Level Model (KLM), are tools that allow designers to compare and optimize the performance of user interfaces by summing the predicted times for the atomic components of a specific interaction to predict the total time it would take to complete that interaction. There has been extensive work in creating such additive models for 2D interfaces, but this approach has rarely been explored for 3D user interfaces. We propose a KLM-style additive model, based on existing atomic task models in the literature, to predict task completion time for 3D interaction tasks. We performed two studies to evaluate the feasibility of this approach across multiple input modalities, with one study using a simple menu selection task and the other a more complex manipulation task. We found that several of the models from the literature predicted actual task performance with less than 20% error in both the menu selection and manipulation study. Overall, we found that additive models can predict both absolute and relative performance of input modalities with reasonable accuracy.

Head-worn augmented reality (AR) allows audiences to be immersed and engaged in stories told by live presenters. While presenters may also be in AR to have the same level of immersion and awareness as their audience, this symmetric presentation style may diminish important social cues such as eye contact. In this work, we examine the effects this (a)symmetry has on engagement, group awareness, and social interaction in co-located one-on-one augmented presentations. We developed a presentation system incorporating 2D/3D content that audiences can view and interact with in AR, with presenters controlling and delivering the presentation in either a symmetric style in AR, or an asymmetric style with a handheld tablet. We conducted a within- and between-subjects evaluation with 12 participant pairs to examine the differences between these symmetric and asymmetric presentation modalities. From our findings, we extracted four themes and derived strategies and guidelines for designers interested in augmented presentations.

Mixed Reality (MR) has introduced innovative remote interaction opportunities in enterprise settings. Physical space sharing is a core aspect of MR, showing promise in remote training, meeting and guidance scenarios. However, this feature could pose security and privacy risks of exposing sensitive enterprise or employee data, as MR spatial sharing uses external cameras to capture physical space information. Despite these potential risks, little research has addressed the specific privacy and security challenges and their mitigations, motivating us to take an initial step in understanding this problem domain. In this poster, we aim to understand when employees share physical object and space during remote collaboration, the privacy and security concerns they have, and the use cases for physical space sharing they envision for their work. We conclude by outlining the next steps to address the challenges of designing privacy and security systems for employees and enterprise to minimize the risk associated with MR spatial sharing.

Hybrid options at conferences, which support in-person and remote attendance, have increasingly become the norm in order to broaden participation and promote sustainability. However, hybrid conferences are challenging, where in-person and remote attendees often have disjoint, parallel experiences with limited opportunity to interact with each other. To explore the potential for facilitating social interaction between in-person and remote conference attendees, we designed and built SocialMiXR, a research prototype that uses WebXR technologies to align the physical and virtual worlds into one hybrid space for socialization. We deployed SocialMiXR in a three-day field study with 14 in-person and remote attendees of an engineering conference. Our qualitative results demonstrate that participants felt they were together in the same conference experience, and formed meaningful connections with each other. At the same time, they faced difficulties balancing different realities and capabilities given their separate contexts. We discuss implications for the design of hybrid social experiences at conferences.

Performance models of interaction, such as Fitts' law, are important tools for predicting and explaining human motor performance and for designing high-performance user interfaces. Extensive prior work has proposed such models for the 3D interaction task of distal pointing, in which the user points their hand or a device at a distant target in order to select it. However, there is no consensus on how to compute the index of difficulty for distal pointing tasks. We present a preliminary study suggesting that existing models may not be sufficient to model distal pointing performance with current virtual reality technologies. Based on these results, we hypothesized that both the form of the model and the standard method for collecting empirical data for pointing tasks might need to change in order to achieve a more accurate and valid distal pointing model. In our main study, we used a new methodology to collect distal pointing data and evaluated traditional models, purely ballistic models, and two-part models. Ultimately, we found that the best model used a simple Fitts'-law-style index of difficulty with angular measures of amplitude and width.

Future augmented reality (AR) glasses empower users to view personal applications and services anytime and anywhere without being restricted by physical locations and the availability of physical screens. In typical everyday activities, people move around to carry out different tasks and need a variety of information on the go. Existing interfaces in AR do not support these use cases well, especially when the number of applications increases. We explore the usability of three world-referenced approaches that move AR applications with users as they transition among different locations, featuring different levels of AR app availability: (1) always using a menu to manually open an app when needed; (2) automatically suggesting a relevant subset of all apps; and (3) carrying all apps with the users to the new location. Through a controlled study and a relatively more ecologically-valid study in AR, we reached better understandings on the performance trade-offs and observed the impact of various everyday contextual factors on these interfaces in more realistic AR settings. Our results shed light on how to better support the mobile information needs of users in everyday life in future AR interfaces.

Although access to sufficient screen space is crucial to knowledge work, workers often find themselves with limited access to display infrastructure in remote or public settings. While virtual displays can be used to extend the available screen space through extended reality (XR) head-worn displays (HWD), we must better understand the implications of working with them in public settings from both users' and bystanders' viewpoints. To this end, we conducted two user studies. We first explored the usage of a hybrid AR display across real-world settings and tasks. We focused on how users take advantage of virtual displays and what social and environmental factors impact their usage of the system. A second study investigated the differences between working with a laptop, an AR system, or a VR system in public. We focused on a single location and participants performed a predefined task to enable direct comparisons between the conditions while also gathering data from bystanders. The combined results suggest a positive acceptance of XR technology in public settings and show that virtual displays can be used to accompany existing devices. We highlighted some environmental and social factors. We saw that previous XR experience and personality can influence how people perceive the use of XR in public. In addition, we confirmed that using XR in public still makes users stand out and that bystanders are curious about the devices, yet have no clear understanding of how they can be used.

Virtual displays enabled through head-worn augmented reality have unique characteristics that can yield extensive amounts of screen space. Existing research has shown that increasing the space on a computer screen can enhance usability. Since virtual displays offer the unique ability to present content without rigid physical space constraints, they provide various new design possibilities. Therefore, we must understand the trade-offs of layout choices when structuring that space. We propose a single Canvas approach that eliminates boundaries from traditional multi-monitor approaches and instead places windows in one large, unified space. Our user study compared this approach against a multi-monitor setup, and we considered both purely virtual systems and hybrid systems that included a physical monitor. We looked into usability factors such as performance, accuracy, and overall window management. Results show that Canvas displays can cause users to compact window layouts more than multiple monitors with snapping behavior, even though such optimizations may not lead to longer window management times. We did not find conclusive evidence of either setup providing a better user experience. Multi-Monitor displays offer quick window management with snapping and a structured layout through subdivisions. However, Canvas displays allow for more control in placement and size, lowering the amount of space used and, thus, head rotation. Multi-Monitor benefits were more prominent in the hybrid configuration, while the Canvas display was more beneficial in the purely virtual configuration.

Gaze-adaptive interfaces can enable intuitive hands-free augmented reality (AR) interaction but unintentional selection (i.e. “Midas Touch”) can have serious consequences during high-stakes real-world AR use. In the present study, we assessed how simulated gaze-adaptive AR interfaces, implementing single and dual gaze inputs, influence Soldiers’ human performance and user experience (UX) in a fast-paced virtual reality marksmanship task. In Experiment 1, we investigated 1- and 2-stage dwell-based interfaces, finding confirmatory dual gaze dwell input effectively reduced Midas Touch but also reduced task performance and UX compared to an always-on (AO) interface. In Experiment 2, we investigated gaze depth-based interfaces, finding similar negative impacts of confirmatory dwell on Midas Touch, task performance, and UX. Overall, compared to the AO interface, single gaze input interfaces (e.g. single dwell or gaze depth threshold) reduced viewing of task-irrelevant information and yielded similar task performance and UX despite being prone to Midas Touch. Broadly, our findings demonstrate that AR users performing fast-paced dynamic tasks can tolerate some unintentional activation of AR displays if reliable and rapid information access is maintained and point to the need to develop and refine gaze depth estimation algorithms and novel gaze depth-based interfaces that provide rapid access to AR display content.

Explainable AI (XAI) has established itself as an important component of AI-driven interactive systems. With Augmented Reality (AR) becoming more integrated in daily lives, the role of XAI also becomes essential in AR because end-users will frequently interact with intelligent services. However, it is unclear how to design effective XAI experiences for AR. We propose XAIR, a design framework that addresses when, what, and how to provide explanations of AI output in AR. The framework was based on a multi-disciplinary literature review of XAI and HCI research, a large-scale survey probing 500+ end-users’ preferences for AR-based explanations, and three workshops with 12 experts collecting their insights about XAI design in AR. XAIR’s utility and effectiveness was verified via a study with 10 designers and another study with 12 end-users. XAIR can provide guidelines for designers, inspiring them to identify new design opportunities and achieve effective XAI designs in AR.

Empowered by the rich sensory capabilities and the advancements in artificial intelligence (AI), head-worn displays (HWD) could understand the user’s contexts and provide just-in-time assistance to users’ tasks to augment their everyday lives. However, there has been limited understanding of how users perceive interacting with AI services, and how different factors impact the user experience in HWD applications. In this research, we investigated broadly what user and system factors play important roles in human-AI experiences during an AI-assisted spatial task. We conducted a user study to simulate an everyday scenario where augmented reality (AR) glasses could provide suggestions/assistance. We researched three AI system factors (performance, initiation, transparency) with multiple user factors (personality traits, trust propensity, and prior trust with AI). We not only identified the impact of user traits such as the levels of conscientiousness and prior trust with the AI, but also found interesting interactions between them and system factors such as AI’s performance and initiation strategy. Based on the findings, we suggest that future AI assistance on HWD needs to take users’ individual characteristics into account and customize the system design accordingly.

Imagine in the future people comfortably wear augmented reality (AR) displays all day, how do we design interfaces that adapt to the contextual changes as people move around? In current operating systems, the majority of AR content defaults to staying at a fixed location until being manually moved by the users. However, this approach puts the burden of user interface (UI) transition solely on users. In this paper, we first ran a bodystorming design workshop to capture the limitations of existing manual UI transition approaches in spatially diverse tasks. Then we addressed these limitations by designing and evaluating three UI transition mechanisms with different levels of automation and controllability (low-effort manual, semi-automated, fully-automated). Furthermore, we simulated imperfect contextual awareness by introducing prediction errors with different costs to correct them. Our results provide valuable lessons about the trade-offs between UI automation levels, controllability, user agency, and the impact of prediction errors.

Glanceable Augmented Reality interfaces have the potential to provide fast and efficient information access for the user. However, where to place the virtual content and how to access them depend on the user context. We designed a Context-Aware AR interface that can intelligently adapt for two different contexts: solo and social. We evaluated information access using Context-Aware AR compared to current mobile phones and non-adaptive Glanceable AR interfaces. We found that in a solo scenario, compared to a mobile phone, the Context-Aware AR interface was preferred, easier, and significantly faster; it improved the user experience; and it allowed the user to better focus on their primary task. In the social scenario, we discovered that the mobile phone was slower, more intrusive, and perceived as the most difficult. Meanwhile, Context-Aware AR was faster for responding to information needs triggered by the conversation; it was preferred and perceived as the easiest for resuming conversation after information access; and it improved the user’s awareness of the other person’s facial expressions.

Referencing objects of interest is a common requirement in many collaborative tasks. Nonetheless, accurate object referencing at a distance can be challenging due to the reduced visibility of the objects or the collaborator and limited communication medium. Augmented Reality (AR) may help address the issues by providing virtual pointing rays to the target of common interest. However, such pointing ray techniques can face critical limitations in large outdoor spaces, especially when the environment model is unavailable. In this work, we evaluated two pointing ray techniques for distant object referencing in model-free AR from the literature: the Double Ray technique enhancing visual matching between rays and targets, and the Parallel Bars technique providing artificial orientation cues. Our experiment in outdoor AR involving participants as pointers and observers partially replicated results from a previous study that only evaluated observers in simulated AR. We found that while the effectiveness of the Double Ray technique is reduced with the additional workload for the pointer and human pointing errors, it is still beneficial for distant object referencing.

In the near future, augmented reality (AR) glasses are envisioned to become the next-generation personal computing platform. They could be always on and worn all day, delivering continuous and pervasive AR experiences for general-purpose everyday use cases. However, it remains unclear how we could enable unobtrusive and easy information access without distracting users, while being acceptable to use at the same time. To address this question, we implemented two prototypes based on the Glanceable AR paradigm, a promising way of managing and acquiring information through glancing at the periphery of AR head-worn displays (HWDs). We conducted two separate studies to evaluate our designs. In the first study, we obtained feedback from a large sample of participants of varied age and background about a video prototype that showcased some envisioned scenarios of using Glanceable AR for everyday tasks. In the second study, we asked participants to use a working prototype during authentic real-world activities for three days. We found that users appreciated the Glanceable AR approach. They found it less distracting or intrusive than existing devices in authentic everyday use cases, and would like to use the interface on a daily basis if the form factor of the AR headset was more like eyeglasses.

Augmented reality head-worn displays (AR HWDs) have the potential to assist personal computing and the acquisition of everyday information. In this research, we propose Glanceable AR, an interaction paradigm for accessing information in AR HWDs. In Glanceable AR, secondary information resides at the periphery of vision to stay unobtrusive and can be accessed by a quick glance whenever needed. We propose two novel hands-free interfaces: "head-glance", in which virtual contents are fixed to the user’s body and can be accessed by head rotation, and "gaze-summon" in which contents can be "summoned" into central vision by eye-tracked gazing at the periphery. We compared these techniques with a baseline heads-up display (HUD), which we call "eye-glance" interface in two dual-task scenarios. We found that the head-glance and eye-glance interfaces are more preferred and more efficient than the gaze-summon interface for discretionary information access. For a continuous monitoring task, the eye-glance interface was preferred. We discuss the implications of our findings for designing Glanceable AR interfaces in AR HWDs.

This paper presents our solution to the 2019 3DUI Contest challenge, which focuses on passive haptics. We aimed to provide a compelling user experience in virtual reality while overcoming the limitations of physical space and current tracking devices. To meet these goals, we designed a time travel scenario that incorporates several novel features. A time machine increases efficiency in the use of physical space. A passive haptic camera prop provides a help system that is integrated into the storyline. Finally, the concept of a “temporal stabilizer” provides a plausible way to reuse a single tracking device to track multiple passive haptic props.

Specifying points in three-dimensional space is essential in AR applications. Geometric triangulation is a straightforward way to specify points, but its naïve implementation has low precision. We designed two enhanced geometric techniques for 3D point marking: VectorCloud, which uses multiple rays to reduce jittering, and ImageRefinement, which allows 3D ray refinement to improve precision. Our experiments, conducted in both simulated and real AR, demonstrate that both techniques improve the precision of 3D point marking, and that ImageRefinement is superior to VectorCloud overall. These results are particularly relevant in the design of mobile AR systems for large outdoor areas.

Interactive visualizations are external cognitive artifacts aimed at supporting users' exploratory and sense-making activities. In recent years, there has been an explosion of commercial virtual reality (VR) head-mounted displays (HMD). These VR devices are meant to offer high levels of engagement and improve users' analytical exploration of the displayed content. However, given their rapid market introduction, the possible influences and usefulness that VR could bring in terms of supporting users' exploration with interactive visualizations remain largely underexplored. We attempt to fill this gap and provide results of an empirical study of an interactive visualization tool that we have developed for a VR HMD system. This tool is aimed at facilitating exploratory and analytical reasoning activities with 3D shapes and their transformational processes. Overall, the results show that the tool is supportive of users' exploratory and analytical activities based on the significant improvement in their post-experiment test scores (when compared to their pre-experiment ones) and their engagement level measured via a user engagement questionnaire and participants' comments. The results shed a positive light on the use of visualizations in VR environments and can inform the design of these tools of domains beyond 3D transformational geometry.