Explainable AI (XAI) offers solutions to the challenges of predictability and interpretability in adaptive interfaces, particularly in Augmented Reality (AR) systems that dynamically adapt information based on situational contexts. While traditional XAI methods highlight contextual factors influencing adaptations, they often overlook the user’s internal understanding, such as their expertise and contextual perceptions. This omission can result in explanations that feel redundant or obvious. We present XAIUI, a computational approach that generates tailored explanations by integrating the system’s adaptation model with a Bayesian model of the user’s internal representation. Two online studies evaluated XAIUI. In the first study (N = 77), participants ranked XAIUI ’s explanations as most preferred compared to four ablations ( \(\chi^{2}(4)=62.28, {\textrm{p}} < 0.001\) ). In the second study (N = 110), XAIUI ’s explanations were rated significantly less complex ( \(\chi^{2}(4)=840.855, {\textrm{p}} < 0.001\) ) than all ablations, except showing no explanation. Our results demonstrate XAIUI ’s ability to deliver user-centric, concise, and intuitive explanations, highlighting its potential to enhance AI-driven interfaces.

Advances in artificial intelligence open the possibility of predicting and recommending sequences of GUI commands to a user. An interesting question raised by this capability is how to present such recommendations to the user – as a sequential set of individual command recommendations, or as one aggregated recommendation consisting of multiple commands. In this paper we propose an interface for aggregated command recommendation and conduct controlled studies to compare sequential versus aggregated command recommendation across a range of simulated utility conditions. Our results indicate that aggregated command recommendation can improve overall task performance over sequential recommendation, and that this benefit comes from enabling users to rapidly recognize and use high-utility aggregated recommendations. The aggregated command recommendation approach also reduced deliberation time when evaluating and correcting imperfect sets of recommended commands.

Adaptive training of motor-skills, where the difficulty level of the training task is adapted optimally based on the learner’s skill levels, has been shown to enable higher learning gains compared to non-adaptive training. However, prior approaches rely on adapting physical tools that are tedious to design and build. This work investigates using augmented reality (AR) to achieve a similar objective of maintaining functional task difficulty – the difficulty experienced by the learner – at an optimal challenge point during adaptive training. A study prototype of an AR adaptive basketball training system was developed, wherein the learners train to throw a physical ball into a virtual AR hoop seen through a head-mounted device. Results from the study (N=16) aimed to measure the learning gains showed higher learning gains after adaptive AR training compared to non-adaptive AR training. An analysis of participant feedback, however, highlighted challenges with AR-based adaptive training, pointing to the need for a different design approach compared to the physical adaptive tools. Collectively, this exploratory study investigates the use of AR for adaptive motor-skill learning and lays the foundation for future research directions for the AR-tool design.

With the progress in Large Language Models (LLMs) and rapid development of wearable smart devices like smart glasses, there is a growing opportunity for users to interact with on-device virtual assistants through voice and gestures with ease. Although voice user interfaces (VUIs) have been widely studied, the potential uses of full-body gestures in VUIs that can fully understand users’ surroundings and gestures are relatively unexplored. In this two-phase research using a Wizard-of-Oz approach, we aim to investigate the role of gestures in VUI interactions and explore their design space. In an initial exploratory user study with six participants, we identify influential factors for VUI gestures and establish an initial design space. In the second phase, we conducted a user study with 12 participants to validate and refine our initial findings. Our results showed that users are open and ready to adopt and utilize gestures to interact with multi-modal VUIs, especially in scenarios with poor voice capture quality. The study also highlighted three key categories of gesture functions for enhancing multi-modal VUI interactions: context reference, alternative input, and flow control. Finally, we present a design space for multi-modal VUI gestures along with demonstrations to enlighten future design for coupling multi-modal VUIs with gestures.

Bare hand interaction in augmented or virtual reality (AR/VR) systems, while intuitive, often results in errors and frustration. However, existing methods, such as a static icon or a dynamic tutorial, can only inform simple and coarse hand gestures and lack corrective feedback. This paper explores various visualizations for enhancing precise hand interaction in VR. Through a comprehensive two-part formative study with 11 participants, we identified four types of essential information for visual guidance and designed different visualizations that manifest these information types. We further distilled four visual designs and conducted a controlled lab study with 15 participants to assess their effectiveness for various single- and double-handed gestures. Our results demonstrate that visual guidance significantly improved users’ gesture performance, reducing time and workload while increasing confidence. Moreover, we found that the visualization did not disrupt most users’ immersive VR experience or their perceptions of hand tracking and gesture recognition reliability.

Input false-positive errors, where a system recognizes an input action that the user did not perform, have been shown to be particularly costly for user experience. Recent work has suggested that eye-gaze behavior immediately following an input event can be used to detect whether the input was intended by a user or was the result of a false-positive error. The ability to detect these errors could enable systems that assist the user with error recovery, but little is currently known about how such error mediation techniques might be designed, or the benefits they could provide. This paper presents an initial investigation of the design of error mediation techniques, and an evaluation of their potential benefits. A controlled study demonstrated that error mediation techniques can save time when recovering from errors by helping users to notice and resolve these errors quickly when they occur.

Mixed reality (MR) provides users with novel affordances that allow them to overlay and experience reality in various visual manifestations. However, existing works mainly focused on using MR to augment a user’s present, which does not exhaust the full potential of contextual MR. In this paper, we empirically explore MR-Driven Near-Future Realities, a future multimodal MR experience that can overlay semantically-related augmentations within a user’s everyday life to allow them to preview, manipulate, and reflect on a near-future reality. We investigated this concept during a VR-based empirical study to understand users’ perceptions and opinions about MR-Driven Near-Future Realities. The results showed that users were positive about the concept of MR-Driven Near-Future Realities and were capable of eliciting near-future realities in various simulated real-world manifestations, but more demanding scenarios negatively affected their experience and task completion performance. Our goal is to spark meaningful and critical discussions about the use of MR-Driven Near-Future Realities to preview possible near-future realities in everyday life.

Adaptive user interfaces (AUIs) can improve user experience by automatically adapting how information and functionality are presented in a user interface. However, the dynamic nature and potentially numerous variations of AUIs make them challenging to author. In this paper, we present a generalized framework for defining adaptation as interpolations between UIs and introduce a computational approach for intelligently generating new variations of a UI from a small set of designs. Based on this approach, we develop FrameKit, an authoring tool with a programming-by-example interface that retains flexibility and control afforded by manual authoring while reducing effort through automatic generation. We demonstrate that FrameKit can support adaptations that typically require domain-specific toolkits, such as those found in context-aware applications, responsive UIs, and ability-based adaptation. We evaluated FrameKit with ten front-end developers, who successfully authored AUIs after a short tutorial session and suggested that FrameKit provides an effective mental model for AUI authoring.

Many XR devices use bare-hand gestures to reduce the need for handheld controllers. Such gestures, however, lead to false positive and false negative recognition errors, which detract from the user experience. While mediation techniques enable users to overcome recognition errors by clarifying their intentions via UI elements, little research has explored how mediation techniques should be designed in XR and how a user’s task and context may impact their design preferences. This research presents empirical studies about the impact of user perceived error costs on users’ preferences for three mediation technique designs, under different simulated scenarios that were inspired by real-life tasks. Based on a large-scale crowd-sourced survey and an immersive VR-based user study, our results suggest that the varying contexts within each task type can impact users’ perceived error costs, leading to different preferred mediation techniques. We further discuss the study implications of these results on future XR interaction design.

Gesture-based recognition systems are susceptible to input recognition errors and user errors, both of which negatively affect user experiences and can be frustrating to correct. Prior work has suggested that user gaze patterns following an input event could be used to detect input recognition errors and subsequently improve interaction. However, to be useful, error detection systems would need to detect various types of high-cost errors. Furthermore, to build a reliable detection model for errors, gaze behaviour following these errors must be manifested consistently across different tasks. Using data analysis and machine learning models, this research examined gaze dynamics following input events in virtual reality (VR). Across three distinct point-and-select tasks, we found differences in user gaze patterns following three input events: correctly recognized input actions, input recognition errors, and user errors. These differences were consistent across tasks, selection versus deselection actions, and naturally occurring versus experimentally injected input recognition errors. A multi-class deep neural network successfully discriminated between these three input events using only gaze dynamics, achieving an AUC-ROC-OVR score of 0.78. Together, these results demonstrate the utility of gaze in detecting interaction errors and have implications for the design of intelligent systems that can assist with adaptive error recovery.

Input recognition errors are common in gesture- and touch-based recognition systems, and negatively affect user experience and performance. When errors occur, systems are unaware of them, but the user's gaze following an error may provide valuable cues for error detection. A study was conducted using a manual serial selection task to investigate whether gaze could be used to discriminate user-initiated selections from injected false positive selection errors. Logistic regression models of gaze dynamics could successfully identify injected selection errors as early as 50 milliseconds following a selection, with performance peaking at 550 milliseconds. A two-phase gaze pattern was observed in which users exhibited high gaze motion immediately following errors, and then decreased gaze motion as the error was noticed. Together, these results provide the first demonstration that gaze dynamics can be used to detect input recognition errors, and open new possibilities for systems that can assist with error recovery.

The craft of improvisational quilting involves working without the use of a predefined pattern. Design decisions are made “in the fabric,” with design experimentation tightly interleaved with the creation of the final artifact. To investigate how this type of design process can be supported, and to address challenges faced by practitioners, this paper presents PatchProv, a system for supporting improvisational quilt design. Based on a review of popular books on improvisational quilting, a set of design principles and key challenges to improvisational quilt design were identified, and PatchProv was developed to support the unique aspects of this process. An evaluation with a small group of quilters showed enthusiasm for the approach and revealed further possibilities for how computational tools can support improvisational quilting and improvisational design practices more broadly.

Existing approaches to trading off false positive versus false negative errors in input recognition are based on imprecise ideas of how these errors affect user experience that are unlikely to hold for all situations. To inform dynamic approaches to setting such a tradeoff, two user studies were conducted on how relative preference for false positive versus false negative errors is influenced by differences in the temporal cost of error recovery, and high-level task factors (time pressure, multi-tasking). Participants completed a tile selection task in which false positive and false negative errors were injected at a fixed rate, and the temporal cost to recover from each of the two types of error was varied, and then indicated a preference for one error type or the other, and a frustration rating for the task. Responses indicate that the temporal costs of error recovery can drive both frustration and relative error type preference, and that participants exhibit a bias against false positive errors, equivalent to ∼1.5 seconds or more of added temporal recovery time. Several explanations for this bias were revealed, including that false positive errors impose a greater attentional demand on the user, and that recovering from false positive errors imposes a task switching cost.

This paper introduces Workflow graphs, or W-graphs, which encode how the approaches taken by multiple users performing a fixed 3D design task converge and diverge from one another. The graph's nodes represent equivalent intermediate task states across users, and directed edges represent how a user moved between these states, inferred from screen recording videos, command log data, and task content history. The result is a data structure that captures alternative methods for performing sub-tasks (e.g., modeling the legs of a chair) and alternative strategies of the overall task. As a case study, we describe and exemplify a computational pipeline for building W-graphs using screen recordings, command logs, and 3D model snapshots from an instrumented version of the Tinkercad 3D modeling application, and present graphs built for two sample tasks. We also illustrate how W-graphs can facilitate novel user interfaces with scenarios in workflow feedback, on-demand task guidance, and instructor dashboards.

3D design software is increasingly available to children through libraries, maker spaces, and for free on the web. This unprecedented availability has the potential to unleash children's creativity in cutting edge domains, but is limited by the steep learning curve of the software. Unfortunately, there is little past work studying the breakdowns faced by children in this domain-most past work has focused on adults in professional settings. In this paper, we present a study of online learning resources and help-seeking strategies available to children starting out with 3D design software. We find that children face a range of challenges when trying to learn 3D design independently-tutorials present instructions at a granularity that leads to overlooked and incorrectly-performed actions, and online help-seeking is largely ineffective due to challenges with query formulation and evaluating found information. Based on our findings, we recommend design directions for next-generation help and learning systems tailored to children.

Users of complex software applications often rely on inefficient or suboptimal workflows because they are not aware that better methods exist. In this paper, we develop and validate a hierarchical approach combining topic modeling and frequent pattern mining to classify the workflows offered by an application, based on a corpus of community-generated videos and command logs. We then propose and evaluate a design space of four different workflow recommender algorithms, which can be used to recommend new workflows and their associated videos to software users. An expert validation of the task classification approach found that 82% of the time, experts agreed with the classifications. We also evaluate our workflow recommender algorithms, demonstrating their potential and suggesting avenues for future work.

Instructors of 3D design workshops for children face many challenges, including maintaining awareness of students' progress, helping students who need additional attention, and creating a fun experience while still achieving learning goals. To help address these challenges, we developed Maestro, a workshop orchestration system that visualizes students' progress, automatically detects and draws attention to common challenges faced by students, and provides mechanisms to address common student challenges as they occur. We present the design of Maestro, and the results of a case-study evaluation with an experienced facilitator and 13 children. The facilitator appreciated Maestro's real-time indications of which students were successfully following her tutorial demonstration, and recognized the system's potential to "extend her reach" while helping struggling students. Participant interaction data from the study provided support for our follow-along detection algorithm, and the capability to remind students to use 3D navigation.

Learning a new software application can be a challenge, requiring the user to enter a new environment where their existing knowledge and skills do not apply, or worse, work against them. To ease this transition, we propose the idea of cross-application bridges that start with the interface of a familiar application, and gradually change their interaction model, tools, conventions, and appearance to resemble that of an application to be learned. To investigate this idea, we developed Blocks-to-CAD, a cross-application bridge from Minecraft-style games to 3D solid modeling. A user study of our system demonstrated that our modifications to the game did not hurt enjoyment or increase cognitive load, and that players could successfully apply knowledge and skills learned in the game to tasks in a popular 3D solid modeling application. The process of developing Blocks-to-CAD also revealed eight design strategies that can be applied to design cross-application bridges for other applications and domains.

A wide variety of tools for creating physical computing systems have been developed, but getting started in this domain remains challenging for novices. In this paper, we introduce test-driven physical computing tutorials, a novel application of interactive tutorial systems to better support users in building and programming physical computing systems. These tutorials inject interactive tests into the tutorial process to help users verify and understand individual steps before proceeding. We begin by presenting a taxonomy of the types of tests that can be incorporated into physical computing tutorials. We then present ElectroTutor, a tutorial system that implements a range of tests for both the software and physical aspects of a physical computing system. A user study suggests that ElectroTutor can improve users' success and confidence when completing a tutorial, and save them time by reducing the need to backtrack and troubleshoot errors made on previous tutorial steps.

Expert interaction techniques enable users to greatly improve their performance; however, to realize these advantages, the user must first acquire the skill necessary to use a technique, then choose to use it over competing novice techniques. This article investigates several factors that may influence whether use of an expert technique persists when the context of use changes. Two studies examine the effect of changing performance requirements, and find that a high performance requirement imposed in a training context can effectively push users to adopt an expert technique, and that use of the technique is maintained when the requirement is subsequently reduced or removed. In a final study, performance requirement, high-level task, and environment of use are changed—participants played a training game to learn the menu for a drawing application, which they then used to complete a series of drawings over the following week. Participants exhibited a somewhat surprising “all-or-nothing” effect, using the expert technique nearly exclusively or not at all, and maintaining this behavior over a range of qualitatively different tasks. This suggests that switching to an expert technique involves a global change by the user, rather than an incremental change as suggested by previous work.

The peak-end rule is a psychological heuristic observing that people's retrospective assessment of an experience is strongly influenced by the intensity of the peak and final moments of that experience. We examine how aspects of game player experience are influenced by peak-end manipulations to the sequence of events in games that are otherwise objectively identical. A first experiment examines players' retrospective assessments of two games (a pattern matching game based on Bejeweled and a point-and-click reaction game) when the sequence of difficulty is manipulated to induce positive, negative and neutral peak-end effects. A second experiment examines assessments of a shootout game in which the balance between challenge and skill is similarly manipulated. Results across the games show that recollection of challenge was strongly influenced by peak-end effects; however, results for fun, enjoyment, and preference to repeat were varied -- sometimes significantly in favour of the hypothesized effects, sometimes insignificant, but never against the hypothesis.

Small touchscreens worn on the wrist are becoming increasingly common, but standard interaction techniques for these devices can be slow, requiring a series of coarse swipes and taps to perform an action. To support faster command selection on watches, we investigate two related interaction techniques that exploit spatial memory. WristTap uses multitouch to allow selection in a single action, and TwoTap uses a rapid combination of two sequential taps. In three quantitative studies, we investigate the design and performance of these techniques in comparison to standard methods. Results indicate that both techniques are feasible, able to accommodate large numbers of commands, and fast users are able to quickly learn the techniques and reach performance of ~1.0 seconds per selection, which is approximately one-third of the time of standard commercial techniques. We also provide insights into the types of applications for which these techniques are well-suited, and discuss how the techniques could be extended.

Command selection on large multi-touch surfaces can be difficult, because the large surface means that there are few landmarks to help users build up familiarity with controls. However, people's hands and fingers are landmarks that are always present when interacting with a touch display. To explore the use of hands as landmarks, we designed two hand-centric techniques for multi-touch displays -- one allowing 42 commands, and one allowing 160 -- and tested them in an empirical comparison against standard tab widgets. We found that the small version (HandMark-Fingers) was significantly faster at all stages of use, and that the large version (HandMark-Multi) was slower at the start but equivalent to tabs after people gained experience with the technique. There was no difference in error rates, and participants strongly preferred both of the HandMark menus over tabs. We demonstrate that people's intimate knowledge of their hands can be the basis for fast and feasible interaction techniques that can improve the performance and usability of interactive tables and other multi-touch systems.

In recent years, extensive research in the HCI literature has explored interactive techniques for digital fabrication. However, little attention in this body of work has examined how to involve and guide human workers in fabricating larger-scale structures. We propose a novel model of crowdsourced fabrication, in which a large number of workers and volunteers are guided through the process of building a pre-designed structure. The process is facilitated by an intelligent construction space capable of guiding individual workers and coordinating the overall build process. More specifically, we explore the use of smartwatches, indoor location sensing, and instrumented construction materials to provide real-time guidance to workers, coordinated by a foreman engine that manages the overall build process. We report on a three day deployment of our system to construct a 12-tall bamboo pavilion with assistance from more than one hundred volunteer workers, and reflect on observations and feedback collected during the exhibit.

The names, icons, and tooltips of commands in feature-rich software are an important source of guidance when locating and selecting amongst commands. Unfortunately, these cues can mislead users into believing that a command is appropriate for a given task, when another command would be more appropriate, resulting in wasted time and frustration. In this paper, we present command disambiguation techniques that inform the user of alternative commands before, during, and after an incorrect command has been executed. To inform the design of these techniques, we define categories of false-feedforward errors caused by misleading interface cues, and identify causes for each. Our techniques are the first designed explicitly to solve this problem in feature-rich software. A user study showed enthusiasm for the techniques, and revealed their potential to play a key role in learning of feature-rich software.

User comments posted to popular online tutorials constitute a rich additional source of information for readers, yet current designs for displaying user comments on tutorial webpages do little to support their use. Instead, comments are separated from the tutorial content they reference and tend to be ordered according to post date. We propose and evaluate the TaggedComments system, a new approach to displaying comments that users post to online tutorials. Using tags supplied by commenters, TaggedComments seeks to enhance the role of user comments by 1) improving their visibility, 2) allowing users to personalize their use of the comments according to their particular information needs, and 3) providing direct access to potentially helpful comments from the tutorial content. A laboratory evaluation with 16 participants shows that, in comparison to the standard comment layout, TaggedComments significantly improves users' subjective impressions of comment utility when interacting with Photoshop tutorials.

Short video demonstrations are effective resources for helping users to learn tools in feature-rich software. However manually creating demonstrations for the hundreds (or thousands) of individual features in these programs would be impractical. In this paper, we investigate the potential for identifying good tool demonstrations from within screen recordings of users performing real-world tasks. Using an instrumented image-editing application, we collected workflow video content and log data from actual end users. We then developed a heuristic for identifying demonstration clips, and had the quality of a sample set of clips evaluated by both domain experts and end users. This multi-step approach allowed us to characterize the quality of 'naturally occurring' tool demonstrations, and to derive a list of good and bad features of these videos. Finally, we conducted an initial investigation into using machine learning techniques to distinguish between good and bad demonstrations.

Users often make continued and sustained use of online resources to complement use of a desktop application. For example, users may reference online tutorials to recall how to perform a particular task. While often used in a coordinated fashion, the browser and desktop application provide separate, independent mechanisms for helping users find and re-find task-relevant information. In this paper, we describe InterTwine, a system that links information in the web browser with relevant elements in the desktop application to create interapplication information scent. This explicit link produces a shared interapplication history to assist in re-finding information in both applications. As an example, InterTwine marks all menu items in the desktop application that are currently mentioned in the front-most web page. This paper introduces the notion of interapplication information scent, demonstrates the concept in InterTwine, and describes results from a formative study suggesting the utility of the concept.

Web-based tutorials are a popular help resource for learning how to perform unfamiliar tasks in complex software. However, in their current form, web tutorials are isolated from the applications that they support. In this paper we present FollowUs, a web-tutorial system that integrates a fully-featured application into a web-based tutorial. This novel architecture enables community enhanced tutorials, which continuously improve as more users work with them. FollowUs captures video demonstrations of users as they perform a tutorial. Subsequent users can use the original tutorial, or choose from a library of captured community demonstrations of each tutorial step. We conducted a user study to test the benefits of making multiple demonstrations available to users, and found that users perform significantly better using our system with a library of multiple demonstrations in comparison to its equivalent baseline system with only the original authored content.

This paper presents a recognizer for identifying references to user interface components in online documentation. The recognizer first extracts phrases matching a list of known components, then employs a classifier to reject coincidental matches. We describe why this seemingly straightforward problem is challenging, then show how informal conventions in documentation writing can be leveraged to perform classification. Using the features identified in this paper, our approach achieves an average F1 score of 0.81, and can correctly distinguish between actual command references and coincidental matches in 93.7% of test cases.

This paper presents results from a study examining perceptions and practices of usability in the free/open source software (FOSS) community. 27 individuals associated with 11 different FOSS projects were interviewed to understand how they think about, act on, and are motivated to address usability issues. Our results indicate that FOSS project members possess rather sophisticated notions of software usability, which collectively mirror definitions commonly found in HCI textbooks. Our study also uncovered a wide range of practices that ultimately work to improve software usability. Importantly, these activities are typically based on close, direct interpersonal relationships between developers and their core users, a group of users who closely follow the project and provide high quality, respected feedback. These relationships, along with positive feedback from other users, generate social rewards that serve as the primary motivations for attending to usability issues on a day-to-day basis. These findings suggest a need to reconceptualize HCI methods to better fit this culture of practice and its corresponding value system.