Translating Computer-Aided Design (CAD) models into physical objects requires expertise and adjustments to navigate fabrication constraints. Makers develop this tacit knowledge by understanding materials, techniques, and practical requirements. Adjustments are typically shared with designer collaborators through sketches and text. However, this documentation lacks situated knowledge gained during fabrication and remains disconnected from the model.

Editing particle-system visual effects (VFX) is vital for digital storytelling, but achieving controllable, art-directable results remains challenging due to their multi-dimensional nature. Given a large collection of parameters, users must find the ones relevant to their creative goals—a task that requires a systematic understanding of the particle system and how parameters map to high-level intents, such as making a fire look angry. Elemental Alchemist is a generative interface that transforms user intent into contextualized controls for semantic editing of particle systems. The system introduces two components: a contextual brush palette that generates tools based on scene context, and a generative control panel that surfaces relevant technical parameters and abstracts them to generate mid-level semantic attributes and high-level conceptual controls. An evaluation with 10 novice and 5 expert VFX practitioners shows the system supported users in translating high-level creative goals into particle system parameters.

Creating 3D character animations traditionally requires significant time and effort from the animator. Advancements in generative methods now enable easy creation of multiple character animation variations for use or further editing. However, this capability introduces a new challenge in comparing character animations to select the best animation, which is challenging due to temporal misalignment and the large amount of spatial data. We present AnimationDiff, a visual comparison tool for generated character animations. AnimationDiff enables contextual comparisons in the intended scene and camera angle, and embedding of spatial information by combining established animation visualization techniques and easy switching between overlaid and side-by-side comparisons. AnimationDiff also supports filtering to handle information overload, and Temporal Lenses that visualize entire animations over time for overview, alignment, and comparison. We evaluated AnimationDiff in a user study, showcasing its efficacy in animation comparison and providing design insights for comparing motion.

AI assistants are transforming creative and knowledge domains, holding similar promise for mechanical design via 3D CAD software. Yet, current AI assistance for CAD relies on geometry or command history, lacking rich design intent. We investigate think-aloud computing as a lightweight approach to capture designers’ spoken intent and inform how future AI assistance could leverage this to provide in-situ feedback. Through a three-part study with 10 designers and 10 experts, we (1) recorded designers’ think-aloud verbalizations during 3D modelling, (2) compared expert feedback with and without think-aloud recordings, and (3) interviewed the original designers to evaluate feedback quality. Findings show that verbalizations surface rationale, future actions, and challenges — insights absent from geometric and command data — that enable feedback attuned to designers’ goals. By harnessing think-aloud data, we uncover when to intervene, what to prompt, and characteristics of effective feedback, paving the way for context-aware AI assistance for CAD.

Think-Aloud Computing, a method for capturing users’ verbalized thoughts during software tasks, allows eliciting rich contextual insights into evolving intentions, struggles, and decision-making processes of users in real-time. However, existing approaches face practical challenges: users often lack awareness of what is captured by the system, are not effectively encouraged to speak, and miss or are interrupted by system feedback. Additionally, thinking aloud should feel worthwhile for users due to the gained contextual AI assistance. To better support and harness Think-Aloud Computing, we introduce PointAloud, a suite of novel AI-driven pointer-centric interactions for in-the-moment verbalization encouragement, low-distraction system feedback, and contextually rich work process documentation alongside proactive AI assistance. Our user study with 12 participants provides insights into the value of pointer-centric think-aloud computing for work process documentation and human-AI co-creation. We conclude by discussing the broader implications of our findings and design considerations for pointer-centric and AI-supported Think-Aloud Computing workflows.

As an emergent process, creativity relies on explorations via sampling and prototyping for problem construction. These activities compile knowledge, provide a context enveloping the solution, and answer questions. With Generative AI, practitioners can go beyond sampling existing media towards instantly generating and remixing new ones. We refer to this convergence as ’protosampling’. Using existing literature we ground a definition for protosampling and operationalize it through Atelier, a canvas-like system that leverages a variety of generative image and video models for visual creation. Atelier: (1) blends the spaces for thinking and creation, where both references and generated assets co-exist in one space, (2) provides various encapsulated technical workflows that focus on the activity at hand, and (3) enables navigating emergence through interactive visualizations, smart search, and collections. Protosampling as a lens reframes creative work to emphasize the process itself and how seemingly disjointed thoughts can tightly interweave into a final solution.

In pre-production, filmmakers and 3D animation experts must rapidly prototype ideas to explore a film’s possibilities before full-scale production, yet conventional approaches involve trade-offs in efficiency and expressiveness. Hand-drawn storyboards often lack spatial precision needed for complex cinematography, while 3D previsualization demands expertise and high-quality rigged assets. To address this gap, we present PrevizWhiz, a system that leverages rough 3D scenes in combination with generative image and video models to create stylized video previews. The workflow integrates frame-level image restyling with adjustable resemblance, time-based editing through motion paths or external video inputs, and refinement into high-fidelity video clips. A study with filmmakers demonstrates that our system lowers technical barriers for film-makers, accelerates creative iteration, and effectively bridges the communication gap, while also surfacing challenges of continuity, authorship, and ethical consideration in AI-assisted filmmaking.

Virtual Production (VP) professionals often face challenges accessing tacit knowledge and creative intent, which are important in forming common ground with collaborators and in contributing more effectively and efficiently to the team. From our formative study (N=23) with a follow-up interview (N=6), we identified the significance and prevalence of this challenge. To help professionals access knowledge, we present GroundLink, a Unity add-on that surfaces meeting-derived knowledge directly in the editor to support establishing common ground. It features a meeting knowledge dashboard for capturing and reviewing decisions and comments, constraint-aware feedforward that proactively informs the editor environment, and cross-modal synchronization that provides referential links between the dashboard and the editor. A comparative study (N=12) suggested that GroundLink help users build common ground with their team while improving perceived confidence and ease of editing the 3D scene. An expert evaluation with VP professionals (N=5) indicated strong potential for GroundLink in real-world workflows.

Current AI writing tools, which rely on text prompts, poorly support the spatial and interactive nature of storytelling where ideas emerge from direct manipulation and play. We present PlayWrite, a mixed-reality system where users author stories by directly manipulating virtual characters and props. A multi-agent AI pipeline interprets these actions into Intent Frames—structured narrative beats visualized as rearrangeable story marbles on a timeline. A large language model then transforms the user’s assembled sequence into a final narrative. A user study (N=13) with writers from varying domains found that PlayWrite fosters a highly improvisational and playful process. Users treated the AI as a collaborative partner, using its unexpected responses to spark new ideas and overcome creative blocks. PlayWrite demonstrates an approach for co-creative systems that move beyond text to embrace direct manipulation and play as core interaction modalities.

Joanne Leong, David Ledo, Thomas Driscoll, Tovi Grossman, George Fitzmaurice, and Fraser Anderson. 2025. Paratrouper: Exploratory Creation of Character Cast Visuals Using Generative AI. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25). Association for Computing Machinery, New York, NY, USA, Article 189, 1–20.

Design thrives on feedback. However, gathering constant feedback throughout the design process can be labor-intensive and disruptive. We explore how AI can bridge this gap by providing effortless, ambient feedback. We introduce FeedQUAC, a lightweight design companion that delivers real-time, read-aloud, AI-generated commentary from diverse personas based on live screenshots of the designer’s workspace. FeedQUAC is always available, context-aware, ambient, playful, and iteration-aware. In a design probe with eight 3D CAD designers, participants highlighted convenience, playfulness, confidence boosts, and inspiration. Our findings suggest that ambient interaction is a valuable consideration for both designing and evaluating future creativity support systems.

Generative AI tools are increasingly being integrated into various workflows. Many powerful AI systems, such as ChatGPT, Gemini, and Grok, are accessed through simple chat interfaces. While these language-based interactions may seem intuitive and futuristic, they also bear a resemblance to early computer command lines. With our prototype system, Curompt (combined cursor+prompt), we explore how to integrate conversational interaction powered by generative AI into familiar graphical interfaces with direct manipulation. We focused on the 3D environment, where relying on language-based interactions alone proves to be challenging.

Cross-reality tasks, like creating or consuming virtual reality (VR) content, often involve inconvenient or distracting switches between desktop and VR. An initial formative study explores cross-reality switching habits, finding most switches are momentary “peeks” between interfaces, with specific habits determined by current context. The results inform a design space for context-aware “peeking” techniques that allow users to view or interact with desktop from VR, and vice versa, without fully switching. We implemented a set of peeking techniques and evaluated them in two levels of a cross-reality task: one requiring only viewing, and another requiring input and viewing. Peeking techniques made task completion faster, with increased input accuracy and reduced perceived workload.

Tutorial videos are a popular help source for learning feature-rich software. However, getting quick answers to questions about tutorial videos is difficult. We present an automated approach for responding to tutorial questions. By analyzing 633 questions found in 5,944 video comments, we identified different question types and observed that users frequently described parts of the video in questions. We then asked participants (N=24) to watch tutorial videos and ask questions while annotating the video with relevant visual anchors. Most visual anchors referred to UI elements and the application workspace. Based on these insights, we built AQuA, a pipeline that generates useful answers to questions with visual anchors. We demonstrate this for Fusion 360, showing that we can recognize UI elements in visual anchors and generate answers using GPT-4 augmented with that visual information and software documentation. An evaluation study (N=16) demonstrates that our approach provides better answers than baseline methods.

Editing character motion in Virtual Reality is challenging as it requires working with both spatial and temporal data using controls with multiple degrees-of-freedom. The spatial and temporal controls are separated, making it difficult to adjust poses over time and predict the effects across adjacent frames. To address this challenge, we propose TimeTunnel, an immersive motion editing interface that integrates spatial and temporal control for 3D character animation in VR. TimeTunnel provides an approachable editing experience via KeyPoses and Trajectories. KeyPoses are a set of representative poses automatically computed to concisely depict motion. Trajectories are 3D animation curves that pass through the joints of KeyPoses to represent in-betweens. TimeTunnel integrates spatial and temporal control by superimposing Trajectories and KeyPoses onto a 3D character. We conducted two studies to evaluate TimeTunnel. In our quantitative study, TimeTunnel reduced the amount of time required for editing motion, and saved effort in locating target poses. Our qualitative study with domain experts demonstrated how TimeTunnel is an approachable interface that can simplify motion editing, while still preserving a direct representation of motion.

Text-to-image AI are capable of generating novel images for inspiration, but their applications for 3D design workflows and how designers can build 3D models using AI-provided inspiration have not yet been explored. To investigate this, we integrated DALL-E, GPT-3, and CLIP within a CAD software in 3DALL-E, a plugin that generates 2D image inspiration for 3D design. 3DALL-E allows users to construct text and image prompts based on what they are modeling. In a study with 13 designers, we found that designers saw great potential in 3DALL-E within their workflows and could use text-to-image AI to produce reference images, prevent design fixation, and inspire design considerations. We elaborate on prompting patterns observed across 3D modeling tasks and provide measures of prompt complexity observed across participants. From our findings, we discuss how 3DALL-E can merge with existing generative design workflows and propose prompt bibliographies as a form of human-AI design history.

Creative practitioners rely on sampling to understand, explore, and construct problems; or gather resources for later use. Despite practitioners’ ability to experience immersive environments, sampling from them remains limited to primarily visual captures (e.g., screenshots, videos), which overlook the richness and variety of available media. To address these challenges, we describe “Immersive Sampling” as a new way to frame information gathering in the context of immersive environments. In the context of Immersive Sampling, practitioners engage in experiencing immersive environments while capturing, organizing, revisiting, and remixing found content. We situate this subset of tasks in literature and argue for their importance for emerging, future content creation domains. To further explore how Immersive Sampling might take place, we created VRicolage, a proof-of-concept prototype showcasing a set of interactions in Virtual Reality to sample, revisit, and remix captures. Given the democratization of immersive environments, Immersive Sampling provides practitioners with a means to collect, revisit, and remix digital materials.

New immersive 3D design tools enable the creation of spatial design recordings, capturing collaborative design activities. By reviewing captured spatial design sessions, which include user activities, workflows, and tool use, users can reflect on their own design processes, learn new workflows, and understand others’ design rationale. However, finding interesting moments in design activities can be challenging: they contain multimodal data (such as user motion and logged events) occurring over time which can be difficult to specify when searching, and are typically distributed over many sessions or recordings. We present Tesseract, a Worlds-in-Miniature-based system to expressively query VR spatial design recordings. Tesseract consists of the Search Cube interface acting as a centralized stage-to-search container, and four querying tools for specifying multimodal data to enable users to find interesting moments in past design activities. We studied ten participants who used Tesseract and found support for our miniature-based stage-to-search approach.

Crafting a rich and unique environment is crucial for fictional world-building, but can be difficult to achieve since illustrating a world from scratch requires time and significant skill. We investigate the use of recent multi-modal image generation systems to enable users iteratively visualize and modify elements of their fictional world using a combination of text input, sketching, and region-based filling. WorldSmith enables novice world builders to quickly visualize a fictional world with layered edits and hierarchical compositions. Through a formative study (4 participants) and first-use study (13 participants) we demonstrate that WorldSmith offers more expressive interactions with prompt-based models. With this work, we explore how creatives can be empowered to leverage prompt-based generative AI as a tool in their creative process, beyond current "click-once" prompting UI paradigms.

In early stages of creative processes, practitioners externalize and combine inspirational materials, using strategies such as mood board creation to achieve a desired vision and aesthetic. Yet, collecting and combining materials can be difficult: (1) mood boards bias towards 2D images, neglecting audio, video, and 3D models; (2) alternative externalizations such as prototypes are best suited for later stages and can be time-consuming and tedious to create; and (3) online searches lead to disjointed sources between different websites and assets in the file system. To address these challenges, we created MoodCubes, a system for rapid creation and manipulation of multimedia content. When adding content, MoodCubes decomposes objects (e.g., extracting colour palettes), suggests new materials without the need to search (e.g., 3D models, images, lighting effects), and provides filters to change the scene’s aesthetic. We studied eight creative professionals using MoodCubes, which suggested ways the system might advance existing design practices.

Despite an abundance of carefully-crafted tutorials, trial-and-error remains many people’s preferred way to learn complex software. Yet, approaches to facilitate trial-and-error (such as tooltips) have evolved very little since the 1980s. While existing mechanisms work well for simple software, they scale poorly to large feature-rich applications. In this paper, we explore new techniques to support trial-and-error in complex applications. We identify key benefits and challenges of trial-and-error, and introduce a framework with a conceptual model and design space. Using this framework, we developed three techniques: ToolTrack to keep track of trial-and-error progress; ToolTrip to go beyond trial-and-error of single commands by highlighting related commands that are frequently used together; and ToolTaste to quickly and safely try commands. We demonstrate how these techniques facilitate trial-and-error, as illustrated through a proof-of-concept implementation in the CAD software Fusion 360. We conclude by discussing possible scenarios and outline directions for future research on trial-and-error.

With the rapid development of creativity support tools, creative practitioners (e.g., designers, artists, architects) have to constantly explore and adopt new tools into their practice. While HCI research has focused on developing novel creativity support tools, little is known about creative practitioner’s values when exploring and adopting these tools. We collect and analyze 23 videos, 13 interviews, and 105 survey responses of creative practitioners reflecting on their values to derive a value framework. We find that practitioners value the tools’ functionality, integration into their current workflow, performance, user interface and experience, learning support, costs and emotional connection, in that order. They largely discover tools through personal recommendations. To help unify and encourage reflection from the wider community of CST stakeholders (e.g., systems creators, researchers, marketers, educators), we situate the framework within existing research on systems, creativity support tools and technology adoption.

Analysis of human motion data can reveal valuable insights about the utilization of space and interaction of humans with their environment. To support this, we present AvatAR, an immersive analysis environment for the in-situ visualization of human motion data, that combines 3D trajectories with virtual avatars showing people’s detailed movement and posture. Additionally, we describe how visualizations can be embedded directly into the environment, showing what a person looked at or what surfaces they touched, and how the avatar’s body parts can be used to access and manipulate those visualizations. AvatAR combines an AR HMD with a tablet to provide both mid-air and touch interaction for system control, as well as an additional overview device to help users navigate the environment. We implemented a prototype and present several scenarios to show that AvatAR can enhance the analysis of human motion data by making data not only explorable, but experienceable.

Virtual Reality (VR) has potential for productive knowledge work, however, midair pointing with controllers or hand gestures does not offer the precision and comfort of traditional 2D mice. Directly integrating mice into VR is difficult as selecting targets in a 3D space is negatively impacted by binocular rivalry, perspective mismatch, and improperly calibrated control-display (CD) gain. To address these issues, we developed Depth-Adaptive Cursor , a 2D-mouse driven pointing technique for 3D selection with depth-adaptation that continuously interpolates the cursor depth by inferring what users intend to select based on the cursor position, the viewpoint, and the selectable objects. Depth-Adaptive Cursor uses a novel CD gain tool to compute a usable range of CD gains for general mouse-based pointing in VR. A user study demonstrated that Depth-Adaptive Cursor significantly improved performance compared with an existing mouse-based pointing technique without depth-adaption in terms of time (21.2%), error (48.3%), perceived workload, and user satisfaction.

We present VideoPoseVR, a video-based animation authoring workflow using online videos to author character animations in VR. It leverages the state-of-the-art deep learning approach to reconstruct 3D motions from online videos, caption the motions, and store them in a motion dataset. Creators can import the videos, search in the dataset, modify the motion timeline, and combine multiple motions from videos to author character animations in VR. We implemented a proof-of-concept prototype and conducted a user study to evaluate the feasibility of the video-based authoring approach as well as gather initial feedback of the prototype. The study results suggest that VideoPoseVR was easy to learn for novice users to author animations and enable rapid exploration of prototyping for applications such as entertainment, skills training, and crowd simulations.

Co-creative AI tools provide a method of creative collaboration between a user and machine. One form of co-creative AI called generative design requires the user to input design parameters and wait substantial periods of time while the system computes design solutions. We explore this interaction dynamic by providing an embodied experience in VR. Calliope is a virtual reality (VR) system that enables users to explore and manipulate generative design solutions in real time. Calliope accounts for the typical idle times in the generative design process by using a virtual environment to encourage parallelized and embodied data-exploration and synthesis, while maintaining a tight human-in-the-loop collaboration with the underlying algorithms. In this paper we discuss design considerations informed by formative studies with generative designers and artists and provide design guidelines to aid others in the development of co-creative AI systems in virtual environments.

When users complete tasks on the computer, the knowledge they leverage and their intent is often lost because it is tedious or challenging to capture. This makes it harder to understand why a colleague designed a component a certain way or to remember requirements for software you wrote a year ago. We introduce think-aloud computing, a novel application of the think-aloud protocol where computer users are encouraged to speak while working to capture rich knowledge with relatively low effort. Through a formative study we find people shared information about design intent, work processes, problems encountered, to-do items, and other useful information. We developed a prototype that supports think-aloud computing by prompting users to speak and contextualizing speech with labels and application context. Our evaluation shows more subtle design decisions and process explanations were captured in think-aloud than via traditional documentation. Participants reported that think-aloud required similar effort as traditional documentation.

As VR technology becomes more available, VR applications will be increasingly used to present information visualizations. While data visualization in VR is an interesting topic, there remain questions about how effective or accurate such visualization can be. One known phenomenon with VR environments is that people tend to unconsciously compress or underestimate distances. However, it is unknown if or how this effect will alter the perception of data visualizations in VR. To this end, we replicate portions of Cleveland and McGill's foundational perceptual visualization studies, in VR. Through a series of three studies we find that distance compression does negatively affect estimations of actual lengths (heights of bars), but does not appear to impact relative comparisons. Additionally, by replicating the position-angle experiments, we find that (as with traditional 2D visualizations) people are better at relative length evaluations than relative angles. Finally, by looking at these open questions, we develop a series of best practices for performing data visualization in a VR environment.

People new to making and makerspaces often struggle with identifying what tools are available and where they are, understanding how to operate the tools, and predicting how their decisions will affect their final product. From literature on novices and our interviews with expert makers, we identified situation awareness support as one possible way to address some of the challenges faced by novices. We present a set of design goals intended to scaffold situation awareness in a makerspace, and MakeAware, a prototype system we implemented based on those design goals. MakeAware provides a combination of environmental cues, information about the project process, and background knowledge. In a preliminary evaluation, we found MakeAware can help novices make conscious choices during a project and put more emphasis on planning, thereby exhibiting traits associated with having situation awareness while making.

While synchronous one-on-one help for software learning is rich and valuable, it can be difficult to find and connect with someone who can provide assistance. Through a formative user study, we explore the idea of fixed-duration, one-on-one help sessions and find that 3 minutes is often enough time for novice users to explain their problem and receive meaningful help from an expert. To facilitate this type of interaction, we developed MicroMentor, an on-demand help system that connects users via video chat for 3-minute help sessions. MicroMentor automatically attaches relevant supplementary materials and uses contextual information, such as command history and expertise, to encourage the most qualified users to accept incoming requests. These help sessions are recorded and archived, building a bank of knowledge that can further help a broader audience. Through a user study, we find MicroMentor to be useful and successful in connecting users for short teaching moments.

This paper introduces Workflow graphs, or W-graphs, which encode how the approaches taken by multiple users performing a fixed 3D design task converge and diverge from one another. The graph's nodes represent equivalent intermediate task states across users, and directed edges represent how a user moved between these states, inferred from screen recording videos, command log data, and task content history. The result is a data structure that captures alternative methods for performing sub-tasks (e.g., modeling the legs of a chair) and alternative strategies of the overall task. As a case study, we describe and exemplify a computational pipeline for building W-graphs using screen recordings, command logs, and 3D model snapshots from an instrumented version of the Tinkercad 3D modeling application, and present graphs built for two sample tasks. We also illustrate how W-graphs can facilitate novel user interfaces with scenarios in workflow feedback, on-demand task guidance, and instructor dashboards.

Augmented Reality (AR) can assist with physical tasks such as object assembly through the use of situated instructions. These instructions can be in the form of videos, pictures, text or guiding animations, where the most helpful media among these is highly dependent on both the user and the nature of the task. Our work supports the authoring of AR tutorials for assembly tasks with little overhead beyond simply performing the task itself. The presented system, AuthAR reduces the time and effort required to build interactive AR tutorials by automatically generating key components of the AR tutorial while the author is assembling the physical pieces. Further, the system guides authors through the process of adding videos, pictures, text and animations to the tutorial. This concurrent assembly and tutorial generation approach allows for authoring of portable tutorials that fit the preferences of different end users.

Design professionals increasingly need to learn and use multiple feature-rich software to design complex artifacts in a variety of domains. In this work, we aimed to understand how design professionals learn new software features and seek help in today's modern workplace of interconnected spaces and increased access to shared online learning resources. We conducted 20 interviews with design professionals working in different disciplines such as architecture and planning, construction, structural design, and broadcast media, who used complex feature-rich applications on a daily basis. We found that despite the wide availability of learning resources on the web and decades of research dedicated to workplace knowledge management tools, design professionals still rely heavily on interpersonal help-seeking. Furthermore, while companies have devised a variety of technological channels to answer employees' help requests, such intermediate tools were often abandoned, and help-seeking conversations shifted to face-to-face learning episodes. Our findings point to design opportunities and shortcomings in the design of remote communication tools when seeking help for feature-rich software applications.

Expressive robots are useful in many contexts, from industrial to entertainment applications. However, designing expressive robot behaviors requires editing a large number of unintuitive control parameters. We present an interactive, data-driven system that allows editing of these complex parameters in a semantic space. Our system combines a physics-based simulation that captures the robot's motion capabilities, and a crowd-powered framework that extracts relationships between the robot's motion parameters and the desired semantic behavior. These relationships enable mixed-initiative exploration of possible robot motions. We specifically demonstrate our system in the context of designing emotionally expressive behaviors. A user-study finds the system to be useful for more quickly developing desirable robot behaviors, compared to manual parameter editing.

The recent proliferation of fabrication and making activities has introduced a large number of users to a variety of tools and equipment. Monitored, reactive and adaptive fabrication spaces are needed to provide personalized information, feedback and assistance to users. This paper explores the sensorization of making and fabrication activities, where the environment, tools, and users were considered to be separate entities that could be instrumented for data collection. From this exploration, we present the design of a modular system that can capture data from the varied sensors and infer contextual information. Using this system, we collected data from fourteen participants with varying levels of expertise as they performed seven representative making tasks. From the collected data, we predict which activities are being performed, which users are performing the activities, and what expertise the users have. We present several use cases of this contextual information for future interactive fabrication spaces.

Remotely instructing and guiding users in physical tasks has offered promise across a wide variety of domains. While it has been the subject of many research projects, current approaches are often limited in the communication bandwidth (lacking context, spatial information) or interactivity (unidirectional, asynchronous) between the expert and the learner. Systems that use Mixed-Reality systems for this purpose have rigid configurations for the expert and the learner. We explore the design space of bi-directional mixed-reality telepresence systems for teaching physical tasks, and present Loki, a novel system which explores the various dimensions of this space. Loki leverages video, audio and spatial capture along with mixed-reality presentation methods to allow users to explore and annotate the local and remote environments, and record and review their own performance as well as their peer's. The system design of Loki also enables easy transitions between different configurations within the explored design space. We validate its utility through a varied set of scenarios and a qualitative user study.

Growing interest in personal fabrication has resulted in many ways to ideate, design, and prototype, in addition to studies of who a maker is and the challenges they face. Less attention, however, has focused on the role of the environment in fabrication processes. By understanding how interactions with tools, fixtures, materials, and spaces shape workflows, we can better determine how to design the next generation of workshops, design tools, and fabrication equipment to support personal fabrication activities. To build this understanding, site visits and interviews at local makerspaces, fabrication studios, and workshops were conducted. These visits uncovered the rich practices and roadblocks generated by workshops today. The observations identified the importance of spatial layouts, territoriality and occupant agency, distributed knowledge, and organizational flux, among others, to design and fabrication processes. These observations were further synthesized into one possible direction for such spaces: hybrid workshops (i.e., environments that can leverage computation and responsive architecture to enhance a maker's ability to design and fabricate). This work identifies how such spaces could harness the rich practices and eliminate the challenges found with workshops today and discusses the technical innovations and philosophical questions that hybrid workshops will pose to the future of personal fabrication.

We present SymbiosisSketch, a hybrid sketching system that combines drawing in air (3D) and on a drawing surface (2D) to create detailed 3D designs of arbitrary scale in an augmented reality (AR) setting. SymbiosisSketch leverages the complementary affordances of 3D (immersive, unconstrained, life-sized) and 2D (precise, constrained, ergonomic) interactions for in situ 3D conceptual design. A defining aspect of our system is the ongoing creation of surfaces from unorganized collections of 3D curves. These surfaces serve a dual purpose: as 3D canvases to map strokes drawn on a 2D tablet, and as shape proxies to occlude the physical environment and hidden curves in a 3D sketch. SymbiosisSketch users draw interchangeably on a 2D tablet or in 3D within an ergonomically comfortable canonical volume, mapped to arbitrary scale in AR. Our evaluation study shows this hybrid technique to be easy to use in situ and effective in transcending the creative potential of either traditional sketching or drawing in air.

This paper presents Dream Lens, an interactive visual analysis tool for exploring and visualizing large-scale generative design datasets. Unlike traditional computer aided design, where users create a single model, with generative design, users specify high-level goals and constraints, and the system automatically generates hundreds or thousands of candidates all meeting the design criteria. Once a large collection of design variations is created, the designer is left with the task of finding the design, or set of designs, which best meets their requirements. This is a complicated task which could require analyzing the structural characteristics and visual aesthetics of the designs. Two studies are conducted which demonstrate the usability and usefulness of the Dream Lens system, and a generatively designed dataset of 16,800 designs for a sample design problem is described and publicly released to encourage advancement in this area.

Instructors of 3D design workshops for children face many challenges, including maintaining awareness of students' progress, helping students who need additional attention, and creating a fun experience while still achieving learning goals. To help address these challenges, we developed Maestro, a workshop orchestration system that visualizes students' progress, automatically detects and draws attention to common challenges faced by students, and provides mechanisms to address common student challenges as they occur. We present the design of Maestro, and the results of a case-study evaluation with an experienced facilitator and 13 children. The facilitator appreciated Maestro's real-time indications of which students were successfully following her tutorial demonstration, and recognized the system's potential to "extend her reach" while helping struggling students. Participant interaction data from the study provided support for our follow-along detection algorithm, and the capability to remind students to use 3D navigation.

A wide variety of tools for creating physical computing systems have been developed, but getting started in this domain remains challenging for novices. In this paper, we introduce test-driven physical computing tutorials, a novel application of interactive tutorial systems to better support users in building and programming physical computing systems. These tutorials inject interactive tests into the tutorial process to help users verify and understand individual steps before proceeding. We begin by presenting a taxonomy of the types of tests that can be incorporated into physical computing tutorials. We then present ElectroTutor, a tutorial system that implements a range of tests for both the software and physical aspects of a physical computing system. A user study suggests that ElectroTutor can improve users' success and confidence when completing a tutorial, and save them time by reducing the need to backtrack and troubleshoot errors made on previous tutorial steps.

We present an interactive tool to animate the visual elements of a static picture, based on simple sketch-based markup. While animated images enhance websites, infographics, logos, e-books, and social media, creating such animations from still pictures is difficult for novices and tedious for experts. Creating automatic tools is challenging due to ambiguities in object segmentation, relative depth ordering, and non-existent temporal information. With a few user drawn scribbles as input, our mixed initiative creative interface extracts repetitive texture elements in an image, and supports animating them. Our system also facilitates the creation of multiple layers to enhance depth cues in the animation. Finally, after analyzing the artwork during segmentation, several animation processes automatically generate kinetic textures that are spatio-temporally coherent with the source image. Our results, as well as feedback from our user evaluation, suggest that our system effectively allows illustrators and animators to add life to still images in a broad range of visual styles.

Sketching in immersive 3D virtual reality (VR) environments has great potential for a variety of interactive 3D design applications. Precisely sketching the intended strokes in mid-air, however, can be a challenge. In this paper, we present a set of controlled studies to analyze the factors affecting human ability to sketch freely in a 3D VR environment. In our first study, we directly compare traditional sketching on a physical surface to sketching in VR, with and without a physical surface to rest the stylus on. Our results indicate that the lack of a physical drawing surface is a major cause of inaccuracies in VR drawing, and that the effect is dependent on the orientation of the drawing surface. In a second experiment, we evaluate the extent to which visual guidance can compensate for the loss of sketching precision in VR. We found that while additional visual guidance improves positional accuracy, it can be detrimental to the aesthetic quality of strokes. We conclude by distilling our experimental findings into design guidelines for sketching tools in immersive 3D environments.

Physical construction and assembly tasks are often carried out by groups of collocated workers, and they can be difficult to coordinate. Group members must spend time deciding how to split up the task, how to assign subtasks to each other, and in what order subtasks should be completed. Informed by an observational study examining group coordination challenges, we built a task distribution system called WeBuild. Our custom algorithm dynamically assigns subtasks to workers in a group, taking into account factors such as the dependencies between subtasks and the skills of each group member. Each worker views personalized step-by-step instructions on a mobile phone, while a dashboard visualizes the entire process. An initial study found that WeBuild reduced the start-up time needed to coordinate and begin a task, and provides direction for future research to build on toward improving group efficiency and coordination for complex tasks.

Datasets which are identical over a number of statistical properties, yet produce dissimilar graphs, are frequently used to illustrate the importance of graphical representations when exploring data. This paper presents a novel method for generating such datasets, along with several examples. Our technique varies from previous approaches in that new datasets are iteratively generated from a seed dataset through random perturbations of individual data points, and can be directed towards a desired outcome through a simulated annealing optimization strategy. Our method has the benefit of being agnostic to the particular statistical properties that are to remain constant between the datasets, and allows for control over the graphical appearance of resulting output.

The dramatic decrease in price and increase in availability of hobbyist electronics has led to a wide array of embedded and interactive devices. While electronics have become more widespread, developing and prototyping the required circuitry for these devices is still difficult, requiring knowledge of electronics, components, and programming. In this paper, we present Trigger-Action-Circuits (TAC), an interactive system that leverages generative design to produce circuitry, firmware, and assembly instructions, based on high-level, behavioural descriptions. TAC is able to generate multiple candidate circuits from a behavioural description, giving the user a number of alternative circuits that may be best suited to their use case (e.g., based on cost, component availability or ease of assembly). The generated circuitry uses off-the-shelf, commodity electronics, not specialized hardware components, enabling scalability and extensibility. TAC supports a range of common components and behaviors that are frequently required for prototyping electronic circuits. A user study demonstrated that TAC helps users avoid problems encountered during circuit design and assembly, with users completing their circuits significantly faster than with traditional methods.

We present DreamSketch, a novel 3D design interface that combines the free-form and expressive qualities of sketching with the computational power of generative design algorithms. In DreamSketch, a user coarsely defines the problem by sketching the design context. Then, a generative design algorithm produces multiple solutions that are augmented as 3D objects in the sketched context. The user can interact with the scene to navigate through the generated solutions. The combination of sketching and generative algorithms enables designers to explore multiple ideas and make better informed design decisions during the early stages of design. Design study sessions with designers and mechanical engineers demonstrate the expressive nature and creative possibilities of DreamSketch.

Within this work, a novel metaphor, haunted design, is explored to challenge the definitions of "display" used today. Haunted design draws inspiration and vision from some of the most multi-modal and sensory diverse experiences that have been reported, the paranormal and hauntings. By synthesizing and deconstructing such phenomena, four novel opportunities to direct display design were uncovered, e.g., intensity, familiarly, tangibility, and shareability. A large-scale design probe, The Living Room, guided the ideation and prototyping of design concepts that exemplify facets of haunted design. By combining the opportunities, design concepts, and survey responses, a framework highlighting the importance of objects, their behavior, and the resulting phenomena to haunted design was developed. Given its emphasis on the odd and unusual, the haunted design metaphor should great spur conversation and alternative directions for future display-based user experiences.

There is a long tradition for crafting wearable objects directly on the body, such as garments, casts, and orthotics. However, these high-skill, analog practices have yet to be augmented by digital fabrication techniques. In this paper, we explore the use of hybrid fabrication workflows for on-body printing. We outline design considerations for creating on-body fabrication systems, and identify several human, machine, and material challenges unique to this endeavor. Based on our explorations, we present ExoSkin, a hybrid fabrication system for designing and printing digital artifacts directly on the body. ExoSkin utilizes a custom built fabrication machine designed specifically for on-body printing. We demonstrate the potential of on-body fabrication with a set of sample workflows, and share feedback from initial observation sessions.

We present ChronoFab, a 3D modeling tool to craft motion sculptures, tangible representations of 3D animated models, visualizing an object's motion with static, transient, ephemeral visuals that are left behind. Our tool casts 3D modeling as a dynamic art-form by employing 3D animation and dynamic simulation for the modeling of motion sculptures. Our work is inspired by the rich history of stylized motion depiction techniques in existing 3D motion sculptures and 2D comic art. Based on a survey of such techniques, we present an interface that enables users to rapidly explore and craft a variety of static 3D motion depiction techniques, including motion lines, multiple stroboscopic stamps, sweeps and particle systems, using a 3D animated object as input. In a set of professional and non-professional usage sessions, ChronoFab was found to be a superior tool for the authoring of motion sculptures, compared to traditional 3D modeling workflows, reducing task completion times by 79%.

We present a sketching tool for crafting animated illustrations that contain the exaggerated dynamics of stylized 2D animations. The system provides a set of motion amplifiers which implement a set of established principles of 2D animation. These amplifiers break down a complex animation effect into independent, understandable chunks. Each amplifier imposes deformations to an underlying grid, which in turn updates the corresponding strokes. Users can combine these amplifiers at will when applying them to an existing animation, promoting rapid experimentation. By leveraging the freeform nature of sketching, our system allows users to rapidly sketch, record motion, explore exaggerated dynamics using the amplifiers, and fine-tune their animations. Practical results confirm that users with no prior experience in animation can produce expressive animated illustrations quickly and easily.

Sliders and Visual Analogue Scales (VASs) are input mechanisms which allow users to specify a value within a predefined range. At a minimum, sliders and VASs typically consist of a line with the extreme values labeled. Additional decorations such as labels and tick marks can be added to give information about the gradations along the scale and allow for more precise and repeatable selections. There is a rich history of research about the effect of labelling in discrete scales (i.e., Likert scales), however the effect of decorations on continuous scales has not been rigorously explored. In this paper we perform a 2,000 user, 250,000 trial online experiment to study the effects of slider appearance, and find that decorations along the slider considerably bias the distribution of responses received. Using two separate experimental tasks, the trade-offs between bias, accuracy, and speed-of-use are explored and design recommendations for optimal slider implementations are proposed.

We present RetroFab, an end-to-end design and fabrication environment that allows non-experts to retrofit physical interfaces. Our approach allows for changing the layout and behavior of physical interfaces. Unlike customizing software interfaces, physical interfaces are often challenging to adapt because of their rigidity. With RetroFab, a new physical interface is designed that serves as a proxy interface for the legacy controls that are now operated by actuators. RetroFab makes this concept of retrofitting devices available to non-experts by automatically generating an enclosure structure from an annotated 3D scan. This enclosure structure holds together actuators, sensors as well as components for the redesigned interface. To allow retrofitting a wide variety of legacy devices, the RetroFab design tool comes with a toolkit of 12 components. We demonstrate the versatility and novel opportunities of our approach by retrofitting five domestic objects and exploring their use cases. Preliminary user feedback reports on the experience of retrofitting devices with RetroFab.

In recent years, extensive research in the HCI literature has explored interactive techniques for digital fabrication. However, little attention in this body of work has examined how to involve and guide human workers in fabricating larger-scale structures. We propose a novel model of crowdsourced fabrication, in which a large number of workers and volunteers are guided through the process of building a pre-designed structure. The process is facilitated by an intelligent construction space capable of guiding individual workers and coordinating the overall build process. More specifically, we explore the use of smartwatches, indoor location sensing, and instrumented construction materials to provide real-time guidance to workers, coordinated by a foreman engine that manages the overall build process. We report on a three day deployment of our system to construct a 12-tall bamboo pavilion with assistance from more than one hundred volunteer workers, and reflect on observations and feedback collected during the exhibit.

Dynamic effects such as waves, splashes, fire, smoke, and explosions are an integral part of stylized animations. However, such dynamics are challenging to produce, as manually sketching key-frames requires significant effort and artistic expertise while physical simulation tools lack sufficient expressiveness and user control. We present an interactive interface for designing these elemental dynamics for animated illustrations. Users draw with coarse-scale energy brushes which serve as control gestures to drive detailed flow particles which represent local velocity fields. These fields can convey both realistic and artistic effects based on user specification. This painting metaphor for creating elemental dynamics simplifies the process, providing artistic control, and preserves the fluidity of sketching. Our system is fast, stable, and intuitive. An initial user evaluation shows that even novice users with no prior animation experience can create intriguing dynamics using our system.

Mobile devices offer constant connectivity to the world, which can negatively affect in-person interaction. Current approaches to minimizing the social disruption and improving the subtlety of interactions tend to focus on the development of inconspicuous devices that provide basic input or output. This paper presents a more general approach to subtle interaction and demonstrates how a number of principles from magic can be leveraged to improve subtlety. It also presents a framework that can be used to classify subtle interfaces along with a modular set of novel interfaces that fit within this framework. Lastly, the paper presents a new evaluation paradigm specifically designed to assess the subtlety of interactions. This paradigm is used to compare traditional approaches to our new subtle approaches. We find our new approaches are over five times more subtle than traditional interactions, even when participants are aware of the technologies being used.

Skin-based input has become an increasingly viable interaction model for user interfaces, however it has yet to be explored outside the domain of mobile computing. In this paper, we examine skin as an interactive input surface for gestural 3D modeling-to-fabrication systems. When used as both the input surface and base canvas for digital design, skin-input can enable non-experts users to intuitively create precise forms around highly complex physical contexts: our own bodies. In this paper, we outline design considerations when creating interfaces for such systems. We then discuss interaction techniques for three different modes of skin-centric modeling: direct, parametric, and generative. We also present Tactum, a new fabrication-aware design system that captures a user's skin-centric gestures for 3D modeling directly on the body. Lastly, we show sample artifacts generated with our system, and share a set of observations from design professionals.

Scatterplots are an effective and commonly used technique to show the relationship between two variables. However, as the number of data points increases, the chart suffers from "over-plotting" which obscures data points and makes the underlying distribution of the data difficult to discern. Reducing the opacity of the data points is an effective way to address over-plotting, however, setting the individual point opacity is a manual task performed by the chart designer. We present a user-driven model of opacity scaling for scatter plots built from crowd-sourced responses to opacity scaling tasks using several synthetic data distributions, and then test our model on a collection of real-world data sets.

Text entry for smart eyewear is generally limited to speech-based input due to constraints of the input channels. However, many smart eyewear devices are now including a side touchpad making gesture-based text entry feasible. The Swipeboard technique, recently proposed for ultra-small touch screens such as smart watches, may be particularly suitable for smart eyewear: unlike other recent text-entry techniques for small devices, it supports eyes-free input. We investigate the limitations and feasibility of implementing Swipeboard on smart eyewear, using the side touch pad for input. Our first study reveals usability and recognition problems of using the side touch pad to perform the required gestures. To address these problems, we propose SwipeZone, which replaces diagonal gestures with zone-specific swipes. In a text entry study, we show that our redesign achieved a WPM rate of 8.73, 15.2% higher than Swipeboard, with a statistically significant improvement in the last half of the study blocks.

In this work, we explore moveables, i.e., interactive papercraft that harness user interaction to generate visual effects. First, we present a survey of children's books that captured the state of the art of moveables. The results of this survey were synthesized into a moveable taxonomy and informed MoveableMaker, a new tool to assist users in designing, generating, and assembling moveable papercraft. MoveableMaker supports the creation and customization of a number of moveable effects and employs moveable-specific features including animated tooltips, automatic instruction generation, constraint-based rendering, techniques to reduce material waste, and so on. To understand how MoveableMaker encourages creativity and enhances the workflow when creating moveables, a series of exploratory workshops were conducted. The results of these explorations, including the content participants created and their impressions, are discussed, along with avenues for future research involving moveables.

The growth of mobile and wearable technologies has made it often difficult to understand what people in our surroundings are doing with their technology. In this paper, we introduce the concept of candid interaction: techniques for providing awareness about our mobile and wearable device usage to others in the vicinity. We motivate and ground this exploration through a survey on current attitudes toward device usage during interpersonal encounters. We then explore a design space for candid interaction through seven prototypes that leverage a wide range of technological enhancements, such as Augmented Reality, shape memory muscle wire, and wearable projection. Preliminary user feedback of our prototypes highlights the trade-offs between the benefits of sharing device activity and the need to protect user privacy.

Due to their limited input area, ultra-small devices, such as smartwatches, are even more prone to occlusion or the fat finger problem, than their larger counterparts, such as smart phones, tablets, and tabletop displays. We present NanoStylus -- a finger-mounted fine-tip stylus that enables fast and accurate pointing on a smartwatch with almost no occlusion. The NanoStylus is built from the circuitry of an active capacitive stylus, and mounted within a custom 3D-printed thimble-shaped housing unit. A sensor strip is mounted on each side of the device to enable additional gestures. A user study shows that NanoStylus reduces error rate by 80%, compared to traditional touch interaction and by 45%, compared to a traditional stylus. This high precision pointing capability, coupled with the implemented gesture sensing, gives us the opportunity to explore a rich set of interactive applications on a smartwatch form factor.

Task-based analysis is a common and effective way to measure expertise levels of software users. However, such assessments typically require in-person laboratory studies and inherently require knowledge of the user's task. Today, there is no accepted method for assessing a user's expertise levels outside of a lab, during a user's own home or work environment activities. In this article, we explore the feasibility of software applications automatically inferring a user's expertise levels, based on the user's in situ usage patterns. We outline the potential usage metrics that may be indicative of expertise levels and then perform a study, where we capture such metrics, by installing logging software in the participants' own workplace environments. We then invite those participants into a laboratory study and perform a more traditional task-based assessment of expertise. Our analysis of the study examines if metrics captured in situ, without any task knowledge, can be indicative of user expertise levels. The results show the existence of significant correlations between metrics calculated from in situ usage logs, and task-based user expertise assessments from our laboratory study. We discuss the implications of the results and how future software applications may be able to measure and leverage knowledge of the expertise of its users.

In 2009 we presented the idea of using collaborative filtering within a complex software application to help users learn new and relevant commands (Matejka et al. 2009). This project continued to evolve and we explored the design space of a contextual software command recommender system and completed a four-week user study (Li et al. 2011). We then expanded the scope of our project by implementing CommunityCommands, a fully functional and deployable recommender system. CommunityCommands was made available as a publically available plug-in download for Autodesk‟s flagship software application AutoCAD. During a one-year period, the recommender system was used by more than 1100 AutoCAD users. In this paper, we present our system usage data and payoff. We also provide an in-depth discussion of the challenges and design issues associated with developing and deploying the front end AutoCAD plug-in and its back end system. This includes a detailed description of the issues surrounding cold start and privacy. We also discuss how our practical system architecture was designed to leverage Autodesk‟s existing Customer Involvement Program (CIP) data to deliver in-product contextual recommendations to endusers. Our work sets important groundwork for the future development of recommender systems within the domain of end-user software learning assistance.

We present CADament, a gamified multiplayer tutorial system for learning AutoCAD. Compared with existing gamified software tutorial systems, CADament generates engaging learning experience through competitions. We investigate two variations of our game, where over-the-shoulder learning was simulated by providing viewports into other player's screens. We introduce an empirical lab study methodology where participants compete with one another, and we study knowledge transfer effects by tracking the migration of strategies between players during the study session. Our study shows that CADament has an advantage over pre-authored tutorials for improving learners' performance, increasing motivation, and stimulating knowledge transfer.

The emergence of smart devices (e.g., smart watches and smart eyewear) is redefining mobile interaction from the solo performance of a smart phone, to a symphony of multiple devices. In this paper, we present Duet -- an interactive system that explores a design space of interactions between a smart phone and a smart watch. Based on the devices' spatial configurations, Duet coordinates their motion and touch input, and extends their visual and tactile output to one another. This transforms the watch into an active element that enhances a wide range of phone-based interactive tasks, and enables a new class of multi-device gestures and sensing techniques. A technical evaluation shows the accuracy of these gestures and sensing techniques, and a subjective study on Duet provides insights, observations, and guidance for future work.

3D modelers often wish to showcase their models for sharing or review purposes. This may consist of generating static viewpoints of the model or authoring animated fly-throughs. Manually creating such views is often tedious and few automatic methods are designed to interactively assist the modelers with the view authoring process. We present a view authoring assistance system that supports the creation of informative view points, view paths, and view surfaces, allowing modelers to author the interactive navigation experience of a model. The key concept of our implementation is to analyze the model's workflow history, to infer important regions of the model and representative viewpoints of those areas. An evaluation indicated that the viewpoints generated by our algorithm are comparable to those manually selected by the modeler. In addition, participants of a user study found our system easy to use and effective for authoring viewpoint summaries.

We present Draco, a sketch-based interface that allows artists and casual users alike to add a rich set of animation effects to their drawings, seemingly bringing illustrations to life. While previous systems have introduced sketch-based animations for individual objects, our contribution is a unified framework of motion controls that allows users to seamlessly add coordinated motions to object collections. We propose a framework built around kinetic textures, which provide continuous animation effects while preserving the unique timeless nature of still illustrations. This enables many dynamic effects difficult or not possible with previous sketch-based tools, such as a school of fish swimming, tree leaves blowing in the wind, or water rippling in a pond. We describe our implementation and illustrate the repertoire of animation effects it supports. A user study with professional animators and casual users demonstrates the variety of animations, applications and creative possibilities our tool provides.

Short video demonstrations are effective resources for helping users to learn tools in feature-rich software. However manually creating demonstrations for the hundreds (or thousands) of individual features in these programs would be impractical. In this paper, we investigate the potential for identifying good tool demonstrations from within screen recordings of users performing real-world tasks. Using an instrumented image-editing application, we collected workflow video content and log data from actual end users. We then developed a heuristic for identifying demonstration clips, and had the quality of a sample set of clips evaluated by both domain experts and end users. This multi-step approach allowed us to characterize the quality of 'naturally occurring' tool demonstrations, and to derive a list of good and bad features of these videos. Finally, we conducted an initial investigation into using machine learning techniques to distinguish between good and bad demonstrations.

Ultra-small smart devices, such as smart watches, have become increasingly popular in recent years. Most of these devices rely on touch as the primary input modality, which makes tasks such as text entry increasingly difficult as the devices continue to shrink. In the sole pursuit of entry speed, the ultimate solution is a shorthand technique (e.g., Morse code) that sequences tokens of input (e.g., key, tap, swipe) into unique representations of each character. However, learning such techniques is hard, as it often resorts to rote memory. Our technique, Swipeboard, leverages our spatial memory of a QWERTY keyboard to learn, and eventually master a shorthand, eyes-free text entry method designed for ultra-small interfaces. Characters are entered with two swipes; the first swipe specifies the region where the character is located, and the second swipe specifies the character within that region. Our study showed that with less than two hours' training, Tested on a reduced word set, Swipeboard users achieved 19.58 words per minute (WPM), 15% faster than an existing baseline technique.

We present Kitty, a sketch-based tool for authoring dynamic and interactive illustrations. Artists can sketch animated drawings and textures to convey the living phenomena, and specify the functional relationship between its entities to characterize the dynamic behavior of systems and environments. An underlying graph model, customizable through sketching, captures the functional relationships between the visual, spatial, temporal or quantitative parameters of its entities. As the viewer interacts with the resulting dynamic interactive illustration, the parameters of the drawing change accordingly, depicting the dynamics and chain of causal effects within a scene. The generality of this framework makes our tool applicable for a variety of purposes, including technical illustrations, scientific explanation, infographics, medical illustrations, children's e-books, cartoon strips and beyond. A user study demonstrates the ease of usage, variety of applications, artistic expressiveness and creative possibilities of our tool.

We present Video Lens, a framework which allows users to visualize and interactively explore large collections of videos and associated metadata. The primary goal of the framework is to let users quickly find relevant sections within the videos and play them back in rapid succession. The individual UI elements are linked and highly interactive, supporting a faceted search paradigm and encouraging exploration of the data set. We demonstrate the capabilities and specific scenarios of Video Lens within the domain of professional baseball videos. A user study with 12 participants indicates that Video Lens efficiently supports a diverse range of powerful yet desirable video query tasks, while a series of interviews with professionals in the field demonstrates the framework's benefits and future potential.

3D printers offer extraordinary flexibility for prototyping the shape and mechanical function of objects. We investigate how 3D models can be modified to facilitate the creation of interactive objects that offer dynamic input and output. We introduce a general technique for supporting the rapid prototyping of interactivity by removing interior material from 3D models to form internal pipes. We describe this new design space of pipes for interaction design, where variables include openings, path constraints, topologies, and inserted media. We then present PipeDream, a tool for routing such pipes through the interior of 3D models, integrated within a 3D modeling program. We use two distinct routing algorithms. The first has users define pipes' terminals, and uses path routing and physics-based simulation to minimize pipe bending energy, allowing easy insertion of media post-print. The second allows users to supply a desired internal shape to which we fit a pipe route: for this we describe a graph-routing algorithm. We present several prototypes created using our tool to show its flexibility and potential.

One of the fundamental operations in today's user interfaces is pointing to targets, such as menus, buttons, and text. Making an error when selecting those targets in real-life user interfaces often results in some cost to the user. However, the existing target-directed pointing models do not consider the cost of error when predicting task completion time. In this paper, we present a model based on expected value theory that predicts the impact of the error cost on the user's completion time for target-directed pointing tasks. We then present a target-directed pointing user study, which results show that time-based costs of error significantly impact the user's performance. Our results also show that users perform according to an expected completion time utility function and that optimal performance computed using our model gives good prediction of the observed task completion times.

Web-based tutorials are a popular help resource for learning how to perform unfamiliar tasks in complex software. However, in their current form, web tutorials are isolated from the applications that they support. In this paper we present FollowUs, a web-tutorial system that integrates a fully-featured application into a web-based tutorial. This novel architecture enables community enhanced tutorials, which continuously improve as more users work with them. FollowUs captures video demonstrations of users as they perform a tutorial. Subsequent users can use the original tutorial, or choose from a library of captured community demonstrations of each tutorial step. We conducted a user study to test the benefits of making multiple demonstrations available to users, and found that users perform significantly better using our system with a library of multiple demonstrations in comparison to its equivalent baseline system with only the original authored content.

Online streaming video systems have become extremely popular, yet navigating to target scenes of interest can be a challenge. While recent techniques have been introduced to enable real-time seeking, they break down for large videos, where scrubbing the timeline causes video frames to skip and flash too quickly to be comprehendible. We present Swifter, a new video scrubbing technique that displays a grid of pre-cached thumbnails during scrubbing actions. In a series of studies, we first investigate possible design variations of the Swifter technique, and the impact of those variations on its performance. Guided by these results we compare an implementation of Swifter to the previously published Swift technique, in addition to the approaches utilized by YouTube and Netfilx. Our results show that Swifter significantly outperforms each of these techniques in a scene locating task, by a factor of up to 48%.

We present Patina, an application independent system for collecting and visualizing software application usage data. Patina requires no instrumentation of the target application, all data is collected through standard window metrics and accessibility APIs. The primary visualization is a dynamic heatmap overlay which adapts to match the content, location, and shape of the user interface controls visible in the active application. We discuss a set of design guidelines for the Patina system, describe our implementation of the system, and report on an initial evaluation based on a short-term deployment of the system.

YouMove is a novel system that allows users to record and learn physical movement sequences. The recording system is designed to be simple, allowing anyone to create and share training content. The training system uses recorded data to train the user using a large-scale augmented reality mirror. The system trains the user through a series of stages that gradually reduce the user's reliance on guidance and feedback. This paper discusses the design and implementation of YouMove and its interactive mirror. We also present a user study in which YouMove was shown to improve learning and short-term retention by a factor of 2 compared to a traditional video demonstration.

General-purpose software applications are usually not tailored for a specific user with specific tasks, strategies or preferences. In order to achieve optimal performance with such applications, users typically need to transition to an alternative efficient behavior. Often, features of such alternative behaviors are not initially accessible and first need to be customized. However, few research works formally study and empirically measure what drives a user to customize. In this paper, we describe the challenges involved in empirically studying customization behaviors, and propose a methodology for formally measuring the impact of potential customization factors. We then demonstrate this methodology by studying the impact of different customization factors on customization behaviors. Our results show that increasing exposure and awareness of customization features, and adding social influence can significantly affect the user's customization behavior.

We investigate implanted user interfaces that small devices provide when implanted underneath human skin. Such devices always stay with the user, making their implanted user interfaces available at all times. We discuss four core challenges of implanted user interfaces: how to sense input through the skin, how to produce output, how to communicate amongst one another and with external infrastructure, and how to remain powered. We investigate these four challenges in a technical evaluation where we surgically implant study devices into a specimen arm. We find that traditional interfaces do work through skin. We then demonstrate how to deploy a prototype device on participants, using artificial skin to simulate implantation. We close with a discussion of medical considerations of implanted user interfaces, risks and limitations, and project into the future.

Tutorials and sample workflows for complicated, feature-rich software packages are widely available online. As a result users must differentiate between workflows to choose the most suitable one for their task. We present Delta, an interactive workflow visualization and comparison tool that helps users identify the tradeoffs between workflows. We conducted an initial study to identify the set of attributes users attend to when comparing workflows, finding that they consider result quality, their knowledge of commands, and the efficiency of the workflow. We then designed Delta to surface these attributes at three granularities: a high-level, clustered view; an intermediate-level list view that contains workflow summaries; and a low-level detail view that allows users to compare two individual workflows. Finally, we conducted an evaluation of Delta on a small corpus of 30 workflows and found that the intermediate list view provided the best information density. We conclude with thoughts on how such a workflow comparison system could be scaled up to larger corpora in the future.

We first conduct a study using abstracted video content to measure the effects of latency on video scrubbing performance and find that even very small amounts of latency can significantly degrade navigation performance. Based on these results, we present Swift, a technique that supports real-time scrubbing of online videos by overlaying a small, low resolution copy of the video during video scrubbing, and snapping back to the high resolution video when the scrubbing is completed or paused. A second study compares the Swift technique to traditional online video players on a collection of realistic live motion videos and content-specific search tasks which finds the Swift technique reducing completion times by as much as 72% even with a relatively low latency of 500ms. Lastly, we demonstrate that the Swift technique can be easily implemented using modern HTML5 web standards.

We present Waken, an application-independent system that recognizes UI components and activities from screen captured videos, without any prior knowledge of that application. Waken can identify the cursors, icons, menus, and tooltips that an application contains, and when those items are used. Waken uses frame differencing to identify occurrences of behaviors that are common across graphical user interfaces. Candidate templates are built, and then other occurrences of those templates are identified using a multi-phase algorithm. An evaluation demonstrates that the system can successfully reconstruct many aspects of a UI without any prior application-dependant knowledge. To showcase the design opportunities that are introduced by having this additional meta-data, we present the Waken Video Player, which allows users to directly interact with UI components that are displayed in the video.

We present GamiCAD, a gamified in-product, interactive tutorial system for first time AutoCAD users. We introduce a software event driven finite state machine to model a user's progress through a tutorial, which allows the system to provide real-time feedback and recognize success and failures. GamiCAD provides extensive real-time visual and audio feedback that has not been explored before in the context of software tutorials. We perform an empirical evaluation of GamiCAD, comparing it to an equivalent in-product tutorial system without the gamified components. In an evaluation, users using the gamified system reported higher subjective engagement levels and performed a set of testing tasks faster with a higher completion ratio.

We present Magic Finger, a small device worn on the fingertip, which supports always-available input. Magic Finger inverts the typical relationship between the finger and an interactive surface: with Magic Finger, we instrument the user's finger itself, rather than the surface it is touching. Magic Finger senses touch through an optical mouse sensor, enabling any surface to act as a touch screen. Magic Finger also senses texture through a micro RGB camera, allowing contextual actions to be carried out based on the particular surface being touched. A technical evaluation shows that Magic Finger can accurately sense 22 textures with an accuracy of 98.9%. We explore the interaction design space enabled by Magic Finger, and implement a number of novel interaction techniques that leverage its unique capabilities.

Despite the prominence of multi-touch technologies, there has been little work investigating its integration into the desktop environment. Bringing multi-touch into desktop computing would give users an additional input channel to leverage, enriching the current interaction paradigm dominated by a mouse and keyboard. We provide two main contributions in this domain. First, we describe the results from a study we performed, which systematically evaluates the various potential regions within the traditional desktop configuration that could become multi-touch enabled. The study sheds light on good or bad regions for multi-touch, and also the type of input most appropriate for each of these regions. Second, guided by the results from our study, we explore the design space of multi-touch-integrated desktop experiences. A set of new interaction techniques are coherently integrated into a desktop prototype, called Magic Desk, demonstrating potential uses for multi-touch enabled desktop configurations.

Despite being an important channel for end-user assistance, few studies have directly investigated the interactions that occur in modern-day practice of software product support. We present results from a multi-dimensional analysis of product support activities at a leading design software company. We carried out a quantitative analysis of existing support requests, a survey with product support specialists, and follow-up interviews to understand the current practices in product support. In particular, we investigated the utility of different multimedia formats that modern web-based support systems enable. Our results showed that despite the value that these formats bring to support tasks, support specialists still face bottlenecks in remotely resolving software problems. We conclude by highlighting several opportunities in HCI for improving diagnosis and resolution of software issues over the web.

Despite the growing research on usability in the pre-development phase, we know little about post-deployment usability activities. To characterize these activities, we surveyed 333 full-time usability professionals and consultants working in large and small corporations from a wide range of industries. Our results show that, as a whole, usability professionals are currently not playing a substantial role in the post-deployment phase compared to other phases of user-centered design, but when they do, practitioners find their interactions quite valuable. We highlight opportunities in HCI research and practice to bridge this gap by working more closely with software support and maintenance teams. We also raise the need to understand what might be called 'usability maintenance,' that is, the process and procedures, by which usability is maintained after deployment.

In this paper we present Ambient Help, a system that supports opportunistic learning by providing automatic, context-sensitive learning resources while a user works. Multiple videos and textual help resources are presented ambiently on a secondary display. We define and examine a collection of design consideration for this type of interface. After describing our implementation details, we report on an experiment which shows that Ambient Help supports finding more helpful information, while not having a negative impact on the user's productivity, as compared to a traditional help condition.

Although touch-screen laptops are increasing in popularity, users still do not comfortably rely on touch in these environments, as current software interfaces were not designed for being used by the finger. In this paper, we first demonstrate the benefits of using touch as a complementary input modality along with the keyboard and mouse or touchpad in a laptop setting. To alleviate the frustration users experience with touch, we then design two techniques, TouchCuts, a single target expansion technique, and ,i>TouchZoom,/i>, a multiple target expansion technique. Both techniques facilitate the selection of small icons, by detecting the finger proximity above the display surface, and expanding the target as the finger approaches. In a controlled evaluation, we show that our techniques improve performance in comparison to both the computer mouse and a baseline touch-based target acquisition technique. We conclude by discussing other application scenarios that our techniques support.

We present Medusa, a proximity-aware multi-touch tabletop. Medusa uses 138 inexpensive proximity sensors to: detect a user's presence and location, determine body and arm locations, distinguish between the right and left arms, and map touch point to specific users and specific hands. Our tracking algorithms and hardware designs are described. Exploring this unique design, we develop and report on a collection of interactions enabled by Medusa in support of multi-user collaborative design, specifically within the context of Proxi-Sketch, a multi-user UI prototyping tool. We discuss design issues, system implementation, limitations, and generalizable concepts throughout the paper.

Users of complex software applications frequently need to consult documentation, tutorials, and support resources to learn how to use the software and further their understand-ing of its capabilities. Existing online help systems provide limited context awareness through "what's this?" and simi-lar techniques. We examine the possibility of making more use of the user's current context in a particular application to provide useful help resources. We provide an analysis and taxonomy of various aspects of application context and how they may be used in retrieving software help artifacts with web browsers, present the design of a context-aware augmented web search system, and describe a prototype implementation and initial user study of this system. We conclude with a discussion of open issues and an agenda for further research.

We describe Sketch-Sketch Revolution, a new tutorial system that allows any user to experience the success of drawing content previously created by an expert artist. Sketch-Sketch Revolution not only guides users through the application user interface, it also provides assistance with the actual sketching. In addition, the system offers an authoring tool that enables artists to create content and then automatically generates a tutorial from their recorded workflow history. Sketch-Sketch Revolution is a unique hybrid tutorial system that combines in-product, content-centric and reactive tutorial methods to provide an engaging learning experience. A qualitative user study showed that our system successfully taught users how to interact with a drawing application user interface, gave users confidence they could recreate expert content, and was uniformly considered useful and easy to use.

We describe TwitApp, an enhanced micro-blogging system integrated within AutoCAD for design sharing. TwitApp integrates rich content and still keeps the sharing transaction cost low. In TwitApp, tweets are organized by their project, and users can follow or unfollow each individual project. We introduce the concept of automatic tweet drafting and other novel features such as enhanced real-time search and integrated live video streaming. The TwitApp system leverages the existing Twitter micro-blogging system. We also contribute a study which provides insights on these concepts and associated designs, and demonstrates potential user excitement of such tools.

We present IP-QAT, a new community-based question and answer system for software users. Unlike most community forums, IP-QAT is integrated into the actual software application, allowing users to easily post questions, answers and tips without having to leave the application. Our in-product implementation is context-aware and shows relevant posts based on a user's recent activity. It is also designed with minimal transaction costs to encourage users to easily post, include annotated images and file attachments, as well as tag their posts with relevant UI components. We describe a robust cloud-based system implementation, which allowed us to release IP-QAT to 37 users for a 2 week field study. Our study showed that IP-QAT increased user contributions, and subjectively, users found our system more useful and easier to use, in comparison to the existing commercial discussion board.

We examine the use of modern recommender system technology to aid command awareness in complex software applications. We first describe our adaptation of traditional recommender system algorithms to meet the unique requirements presented by the domain of software commands. A user study showed that our item-based collaborative filtering algorithm generates 2.1 times as many good suggestions as existing techniques. Motivated by these positive results, we propose a design space framework and its associated algorithms to support both global and contextual recommendations. To evaluate the algorithms, we developed the CommunityCommands plug-in for AutoCAD. This plug-in enabled us to perform a 6-week user study of real-time, within-application command recommendations in actual working environments. We report and visualize command usage behaviors during the study, and discuss how the recommendations affected users behaviors. In particular, we found that the plug-in successfully exposed users to new commands, as unique commands issued significantly increased.

We investigate the use of on-line contextual video assistance to improve the learnability of software functionality. After discussing motivations and design goals for such forms of assistance, we present our new technique, ToolClips. ToolClips augment traditional tooltips to provide users with quick and contextual access to both textual and video assistance. In an initial study we found that users successfully integrated ToolClip usage into the flow of their primary tasks to overcome learnability difficulties. In a second study, we found that with ToolClips, users successfully completed 7 times as many unfamiliar tasks, in comparison to using a commercial professionally developed on-line help system. Users also retained the information obtained from ToolClips, performing tasks significantly faster one week later.

Despite the considerable quantity of research directed towards multitouch technologies, a set of standardized UI components have not been developed. Menu systems provide a particular challenge, as traditional GUI menus require a level of pointing precision inappropriate for direct finger input. Marking menus are a promising alternative, but have yet to be investigated or adapted for use within multitouch systems. In this paper, we first investigate the human capabilities for performing directional chording gestures, to assess the feasibility of multitouch marking menus. Based on the positive results collected from this study, and in particular, high angular accuracy, we discuss our new multitouch marking menu design, which can increase the number of items in a menu, and eliminate a level of depth. A second experiment showed that multitouch marking menus perform significantly faster than traditional hierarchal marking menus, reducing acquisition times in both novice and expert usage modalities.

MouseLight is a spatially-aware standalone mobile projector with the form factor of a mouse that can be used in combination with digital pens on paper. By interacting with the projector and the pen bimanually, users can visualize and modify the virtually augmented contents on top of the paper, and seamlessly transition between virtual and physical information. We present a high fidelity hardware prototype of the system and demonstrate a set of novel interactions specifically tailored to the unique properties of MouseLight. MouseLight differentiates itself from related systems such as PenLight in two aspects. First, MouseLight presents a rich set of bimanual interactions inspired by the ToolGlass interaction metaphor, but applied to physical paper. Secondly, our system explores novel displaced interactions, that take advantage of the independent input and output that is spatially aware of the underneath paper. These properties enable users to issue remote commands such as copy and paste or search. We also report on a preliminary evaluation of the system which produced encouraging observations and feedback.

We describe Chronicle, a new system that allows users to explore document workflow histories. Chronicle captures the entire video history of a graphical document, and provides links between the content and the relevant areas of the history. Users can indicate specific content of interest, and see the workflows, tools, and settings needed to reproduce the associated results, or to better understand how it was constructed to allow for informed modification. Thus, by storing the rich information regarding the document's history workflow, Chronicle makes any working document a potentially powerful learning tool. We outline some of the challenges surrounding the development of such a system, and then describe our implementation within an image editing application. A qualitative user study produced extremely encouraging results, as users unanimously found the system both useful and easy to use.

It is well-accepted that learnability is an important aspect of usability, yet there is little agreement as to how learnability should be defined, measured, and evaluated. In this paper, we present a survey of the previous definitions, metrics, and evaluation methodologies which have been used for software learnability. Our survey of evaluation methodologies leads us to a new question-suggestion protocol, which, in a user study, was shown to expose a significantly higher number of learnability issues in comparison to a more traditional think-aloud protocol. Based on the issues identified in our study, we present a classification system of learnability issues, and demonstrate how these categories can lead to guidelines for addressing the associated challenges.

We explore the use of multi-finger input to emulate full mouse functionality, such as the tracking state, three buttons, and chording. We first present the design space for such techniques, which serves as a guide for the systematic investigation of possible solutions. We then perform a series of pilot studies to come up with recommendations for the various aspects of the design space. These pilot studies allow us to arrive at a recommended technique, the SDMouse. In a formal study, the SDMouse was shown to significantly improve performance in comparison to previously developed mouse emulation techniques.

Digital pen systems, originally designed to digitize annotations made on physical paper, are evolving to permit a wider variety of applications. Although the type and quality of pen feedback (e.g., haptic, audio, and visual) have a huge impact on advancing the digital pen technology, dynamic visual feedback has yet to be fully investigated. In parallel, miniature projectors are an emerging technology with the potential to enhance visual feedback for small mobile computing devices. In this paper we present the PenLight system, which is a testbed to explore the interaction design space and its accompanying interaction techniques in a digital pen embedded with a spatially-aware miniature projector. Using our prototype, that simulates a miniature projection (via a standard video projector), we visually augment paper documents, giving the user immediate access to additional information and computational tools. We also show how virtual ink can be managed in single and multi-user environments to aid collaboration and data management. User evaluation with professional architects indicated promise of our proposed techniques and their potential utility in the paper-intensive domain of architecture.

We explore the use of modern recommender system technology to address the problem of learning software applications. Before describing our new command recommender system, we first define relevant design considerations. We then discuss a 3 month user study we conducted with professional users to evaluate our algorithms which generated customized recommendations for each user. Analysis shows that our item-based collaborative filtering algorithm generates 2.1 times as many good suggestions as existing techniques. In addition we present a prototype user interface to ambiently present command recommendations to users, which has received promising initial user feedback.

Inspired by our automotive and product design customers using large displays in design centers, visualization studios, and meeting rooms around the world, we have been exploring the use and potential of large display installations for almost a decade. Our research has touched on many aspects of this rich design space, from individual tools to complete systems, and has generally moved through the life cycle of a design artifact: from the creation phase, through communication and collaboration, to presentation and dissemination. As we attempt to preserve creative flow through the phases, we introduce social structures and constraints that drive the design of possible point solutions in the larger context of a digital design studio trail environment built in the lab. Although many of the interactions presented are viable across several design phases, this article focuses primarily on facilitating collaboration. We conclude with critical lessons learned of both what avenues have been fruitful and which roads to avoid. This article lightly covers the whole design process and attempts to inform readers of key factors to consider when designing for designers.

We describe a new type of graphical user interface widget called the "PieCursor." The PieCursor is based on the Tracking Menu technique and consists of a radial cluster of command wedges, is roughly the size of a cursor, and replaces the traditional cursor. The PieCursor technique merges the normal cursor function of pointing with command selection into a single action. A controlled experiment was conducted to compare the performance of rapid command and target selection using the PieCursor against larger versions of Tracking Menus and a status quo Toolbar configuration. Results indicate that for small clusters of tools (4 and 8 command wedges) the PieCursor can outperform the toolbar by 20.8% for coarse pointing. For fine pointing, the performance of the PieCursor degrades approximately to the performance found for the Toolbar condition.

Typical commercial 3D CAD tools provide modal tools such as pan, zoom, orbit, look, etc. to facilitate freeform navigation in a 3D scene. Mastering these navigation tools requires a significant amount of learning and even experienced computer users can find learning confusing and error-prone. To address this we have developed a concept called "Safe 3D Navigation" where we augment these modal tools with properties to reduce the occurance of confusing situations and improve the learning experience. In this paper we describe the major properties needed for safe navigation, the features we implemented to realize these properties, and usability tests on the effectiveness of these features. We conclude that indeed these properties do improve the learning experience for users that are new to 3D. Furthermore, many of the features we implemented for safe navigation are also very popular with experienced 3D users. As a result, these features have been integrated into six commercial 3D CAD applications and we recommend other application developers include these features to improve 3D navigation.

Literally hundreds of thousands of users of 2D computer-aided design (CAD) tools are in the difficult process of transitioning to 3D CAD tools. A common problem for these users is disorientation in the abstract virtual 3D environments that occur while developing new 3D scenes. To help address this problem, we present a novel in-scene 3D widget called the ViewCube as a 3D orientation indicator and controller. The ViewCube is a cube-shaped widget placed in a corner of the window. When acting as an orientation indicator, the ViewCube turns to reflect the current view direction as the user re-orients the scene using other tools. When used as an orientation controller, the ViewCube can be dragged, or the faces, edges, or corners can be clicked on, to easily orient the scene to the corresponding view. We conducted a formal experiment to measure the performance of the ViewCube comparing: (1) ArcBall-style dragging using the ViewCube for manual view switching, (2) clicking on face/edge/corner elements of the ViewCube for automated view switching and (3) clicking on a dedicated row of buttons for automated view switching. The results indicate that users prefer and are almost twice as fast at using the ViewCube with dragging compared to clicking techniques, independent of a number of ViewCube representations that we examined.

We describe a new interactive system for 3D design review, built to exploit the visual perception cue of motion parallax, in order to enhance shape perception and aesthetic evaluation. Traditional CAD applications typically use "bookmarked" static views for design evaluation. In our system, we replace static views with moving "shots" interspersed with cinematic visual transitions. Furthermore, users can access shots by picking object features on the 3D model, which invokes a spatial search over all shots, selects the ones deemed relevant to the feature, and plays back those shots. In this paper we describe the rationale behind this system, the main components, the implementation of the search technique, and the initial user evaluation of the system.

We describe a new interaction technique, called a spotlight, for directing the visual attention of an audience when viewing data or presentations on large wall-sized displays. A spotlight is simply a region of the display where the contents are displayed normally while the remainder of the display is somewhat darkened. In this paper we define the behavior of spotlights, show unique affordances of the technique, and discuss design characteristics. We also report on experiments that show the benefit of using the spotlight a large display and standard desktop configuration. Our results suggest that the spotlight is preferred over the standard cursor and outperforms it by a factor of 3.4 on a wall-sized display.

We describe a new interaction technique, called HoverCam, for navigating around 3D objects at close proximity. When a user is closely inspecting an object, the camera motions needed to move across its surface can become complex. For tasks such as 3D painting or modeling small detail features, users will often try to keep the camera a small distance above the surface. To achieve this automatically, HoverCam intelligently integrates tumbling, panning, and zooming camera controls into a single operation. This allows the user to focus on the task at hand instead of continuously managing the camera position and orientation. In this paper we show unique affordances of the technique and define the behavior and implementation of HoverCam. We also show how the technique can be used for navigating about data sets without well-defined surfaces such as point clouds and curves in space.

We describe a new widget and interaction technique, known as a "Frisbee," for interacting with areas of a large display that are difficult or impossible to access directly. A frisbee is simply a portal to another part of the display. It consists of a local "telescope" and a remote "target". The remote data surrounded by the target is drawn in the telescope and interactions performed within it are applied on the remote data. In this paper we define the behavior of frisbees, show unique affordances of the widget, and discuss design characteristics. We have implemented a test application and report on an experiment that shows the benefit of using the frisbee on a large display. Our results suggest that the frisbee is preferred over walking back and forth to the local and remote spaces at a distance of 4.5 feet.

We describe a new type of graphical user interface widget, known as a "tracking menu." A tracking menu consists of a cluster of graphical buttons, and as with traditional menus, the cursor can be moved within the menu to select and interact with items. However, unlike traditional menus, when the cursor hits the edge of the menu, the menu moves to continue tracking the cursor. Thus, the menu always stays under the cursor and close at hand.In this paper we define the behavior of tracking menus, show unique affordances of the widget, present a variety of examples, and discuss design characteristics. We examine one tracking menu design in detail, reporting on usability studies and our experience integrating the technique into a commercial application for the Tablet PC. While user interface issues on the Tablet PC, such as preventing round trips to tool palettes with the pen, inspired tracking menus, the design also works well with a standard mouse and keyboard configuration.

We review the Boom Chameleon, a novel input/output device consisting of a flat-panel display mounted on a tracked mechanical armature. The display acts as a physical window into 3D virtual environments, through which a one-to-one mapping between real and virtual space is preserved. The Boom Chameleon is further augmented with a touch-screen and a microphone/speaker combination. We created a 3D annotation application that exploits this unique configuration in order to simultaneously capture viewpoint, voice and gesture information. Results of an informal user study show that the Boom Chameleon annotation facilities have the potential to be an effective, and intuitive system for reviewing 3D designs.

Previous systems have explored the challenges of designing an interface for automotive styling which combine the metaphor of 2D drawing using physical tape with the simultaneous creation and management of a corresponding virtual 3D model. These systems have been limited to only 2D planar curves while typically the principal characteristic curves of an automotive design are three dimensional and non-planar. We present a system which addresses this limitation. Our system allows a designer to construct these non-planar 3D curves by drawing a series of 2D curves using the 2D tape drawing technique and interaction style. These results are generally applicable to the interface design of 3D modeling applications and also to the design of arm's length interaction on large scale display systems

This paper describes StyleCam, an approach for authoring 3D viewing experiences that incorporate stylistic elements that are not available in typical 3D viewers. A key aspect of StyleCam is that it allows the author to significantly tailor what the user sees and when they see it. The resulting viewing experience can approach the visual richness and pacing of highly authored visual content such as television commercials or feature films. At the same time, StyleCam allows for a satisfying level of interactivity while avoiding the problems inherent in using unconstrained camera models. The main components of StyleCam are camera surfaces which spatially constrain the viewing camera; animation clips that allow for visually appealing transitions between different camera surfaces; and a simple, unified, interaction technique that permits the user to seamlessly and continuously move between spatial-control of the camera and temporal-control of the animated transitions. Further, the user's focus of attention is always kept on the content, and not on extraneous interface widgets. In addition to describing the conceptual model of StyleCam, its current implementation, and an example authored experience, we also present the results of an evaluation involving real users.

We introduce the Boom Chameleon, a novel input/output device consisting of a flat-panel display mounted on a tracked mechanical boom. The display acts as a physical window into 3D virtual environments, through which a one-to-one mapping between real and virtual space is preserved. The Boom Chameleon is further augmented with a touch-screen and a microphone/speaker combination. We present a 3D annotation application that exploits this unique configuration in order to simultaneously capture viewpoint, voice and gesture information. Design issues are discussed and results of an informal user study on the device and annotation software are presented. The results show that the Boom Chameleon annotation facilities have the potential to be an effective, easy to learn and operate 3D design review system.

We present an alternate interface for 3D modeling for use on large scale displays. The interface integrates several concepts specifically selected and enhanced for large scale interaction. These include 2D construction planes spatially integrated in a 3D volume, enhanced orthographic views, smooth transitions between 2D and 3D views, tape drawing as the primary curve and line creation technique, visual viewpoint markers, and continuous two-handed interaction.

Rotating a piece of paper while drawing is an integral and almost subconscious part of drawing with pencil and paper. In a similar manner, the advent of lightweight pen-based computers allow digital artwork to be rotated while drawing by rotating the entire computer. Given this type of manipulation we explore the implications for the user interface to support artwork orientation. First we describe an exploratory study to further motivate our work and characterize how artwork is manipulated while drawing. After presenting some possible UI approaches to support artwork orientation, we define a new solution called a rotating user interface (RUIs). We then discuss design issues and requirements for RUIs based on our exploratory study.

The proliferation of multiple toolbars and UI widgets around the perimeter of application windows is an indication that the traditional GUI design of a single menubar is not sufficient to support large scale applications with numerous functions. In this paper we describe a new widget which is an enhancement of the traditional menubar which dramatically increases menu-item capacity. This widget, called the Hotbox combines several GUI techniques which are generally used independently: accelerator keys, modal dialogs, pop-up/pull down menus, radial menus, marking menus and menubars. These techniques are fitted together to create a single, easy to learn yet fast to operate GUI widget which can handle significantly more menu-items than the traditional GUI menubar. We describe the design rationale of the Hotbox and its effectiveness in a large scale commercial application. While the Hotbox was developed for a particular application domain, the widget itself and the design rationale are potentially useful in other domains.

We explore a new input device and a set of interaction techniques to facilitate direct manipulation of curves and surfaces. The input device, called ShapeTapeTM, is a continuous bend and twist sensitive strip that encourages manipulations that use both hands and, at times, all 10 fingers. We explore this input and interaction design space through a set of usage scenarios for creating and editing curves and surfaces as well as consider general interactions such as command access and camera controls. This investigation is carried out by extending Alias|wavefront’s modeling and animation package, Maya. CR

Tape drawing is the art of creating sketches on large scale upright surfaces using black photographic tape. Typically used in the automotive industry, it is an important part of the automotive design process that is currently not computerized. We analyze and describe the unique aspects of tape drawing, and use this knowledge to design and implement a digital tape drawing system. Our system retains the fundamental interaction and visual affordances of the traditional media while leveraging the power of the digital media. Aside from the practical aspect of our work, the interaction techniques developed have interesting implications for current theories of human bimanual interaction.

Article Free Access Share on The Rockin'Mouse: integral 3D manipulation on a plane Authors: Ravin Balakrishnan Dept. of Computer Science, University of Toronto, Toronto, Ontario, Canada M5S 1A1 and Alias/Wavefront, 110 Richmond Street East, Toronto, Ontario, Canada M5C 1P1 Dept. of Computer Science, University of Toronto, Toronto, Ontario, Canada M5S 1A1 and Alias/Wavefront, 110 Richmond Street East, Toronto, Ontario, Canada M5C 1P1View Profile , Thomas Baudel Alias/Wavefront, 110 Richmond Street East, Toronto, Ontario, Canada M5C 1P1 Alias/Wavefront, 110 Richmond Street East, Toronto, Ontario, Canada M5C 1P1View Profile , Gordon Kurtenbach Alias/Wavefront, 110 Richmond Street East, Toronto, Ontario, Canada M5C 1P1 Alias/Wavefront, 110 Richmond Street East, Toronto, Ontario, Canada M5C 1P1View Profile , George Fitzmaurice Alias/Wavefront, 110 Richmond Street East, Toronto, Ontario, Canada M5C 1P1 Alias/Wavefront, 110 Richmond Street East, Toronto, Ontario, Canada M5C 1P1View Profile Authors Info & Claims CHI '97: Proceedings of the ACM SIGCHI Conference on Human factors in computing systemsMarch 1997 Pages 311–318https://doi.org/10.1145/258549.258778Published:27 March 1997Publication History 84citation1,262DownloadsMetricsTotal Citations84Total Downloads1,262Last 12 Months49Last 6 weeks5 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

This paper reports on the experimental evaluation of a GraspableUser Interface that employs a "spacemultiplexing" input scheme in which each function to be controlled has a dedicated physical transducer, each occupying its own space.This input style contrasts the more traditional "time-multiplexing" input scheme which uses one device (such as the mouse) to control different functions at different points in time.A tracking experiment was conducted to compare a traditional GUI design with its time-multiplex input scheme versus a Graspable UI design having a space-multiplex input scheme.We found that the space-multiplex conditions out perform the time-multiplex conditions.In addition, we found that the use of specialized physical form factors for the input devices instead of generic form factors provide a performance advantage.We rugue that the specialized devices serve as both visual and tactile functional reminders of the associated tool assignment as well as facilitate manipulation due to the customized form factors.

An experimental GUI paradigm is presented which is based on the design goals of maximizing the amount of screen used for application data, reducing the amount that the UI diverts visual attentions from the application data, and increasing the quality of input. In pursuit of these goals, we integrated the non-standard UI technologies of multi-sensor tablets, toolglass, transparent UI components, and marking menus. We describe a working prototype of our new para-digm, the rationale behind it and our experiences introduc-ing it into an existing application. Finally, we presents some ot the lessons learned: prototypes are useful to break the barriers imposed by conventional GUI design and some of their ideas can still be retrofitted seamlessly into products. Furthermore, the added functionality is not measured only in terms of user performance, but also by the quality of interaction, which allows artists to create new graphic vocabularies and graphic styles.

Article Bricks: laying the foundations for graspable user interfaces Share on Authors: George W. Fitzmaurice Dynamic Graphics Project, CSRI, University of Toronto, Toronto, Ontario, Canada M5S 1A4 Dynamic Graphics Project, CSRI, University of Toronto, Toronto, Ontario, Canada M5S 1A4View Profile , Hiroshi Ishii NTT Human Interface Lab, 1-2356 Take, Yokosuka-Shi, Kanagawa, 238-03 Japan NTT Human Interface Lab, 1-2356 Take, Yokosuka-Shi, Kanagawa, 238-03 JapanView Profile , William A. S. Buxton Dynamic Graphics Project, CSRI, University of Toronto, Toronto, Ontario, Canada M5S 1A4 and Alias Research Inc., 110 Richmond Street East, Toronto, Ontario, Canada M5C 1P1 Dynamic Graphics Project, CSRI, University of Toronto, Toronto, Ontario, Canada M5S 1A4 and Alias Research Inc., 110 Richmond Street East, Toronto, Ontario, Canada M5C 1P1View Profile Authors Info & Claims CHI '95: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsMay 1995 Pages 442–449https://doi.org/10.1145/223904.223964Online:01 May 1995Publication History 528citation5,214DownloadsMetricsTotal Citations528Total Downloads5,214Last 12 Months324Last 6 weeks32 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We are exploring how virtual reahty theories can be applied toward palmtop computers. In our prototype, called the Chameleon, a small 4-inch hand-held monitor acts as a palmtop computer with the capabihties of a Silicon graphics workstation. A 6D input device and a response button are attached to tbe small monitor to detect user gestures and input selections for issuing commands. An experiment was conducted to evaluate our design and to see how well depth could be perceived in the small screen compared to a large 21-inch screen, and the extent to which movement of the small display (in a palmtop virtual reality condition) could improve depth perception, Results show that with very little training, perception of depth in the palmtop virtual reality condition is about as good as corresponding depth perception in a large (but static) display. Variations to the initial design are also discussed, along with issues to be explored in future research, Our research suggests that palmtop virtual reality may support effective navigation and search and retrieval, in rich and portable information spaces.

The Envoy Framework addresses a need for computer-based assistants or agents that operate in conjunction with users' existing applications, helping them perform tedious, repetitive, or time-consuming tasks more easily and efficiently. Envoys carry out missions for users by invoking envoy-aware applications called operatives and inform users of mission results via envoy-aware applications called informers. The distributed, open architecture developed for Envoys is derived from an analysis of the best characteristics of existing agent systems. This architecture has been designed as a model for how agent technology can be seamlessly integrated into the electronic desktop. It defines a set of application programmer's interfaces so that developers may convert their software to envoy-aware applications. A subset of the architecture described in this paper has been implemented in an Envoy Framework prototype.