AI co-writing systems challenge long held ideals about agency and ownership in the creative process, thereby hindering widespread adoption. To address this, we investigate conceptions of agency and ownership in AI creative co-writing. Drawing on insights from a review of commercial systems, we developed three co-writing systems with similar functionality but differing interface metaphors: agentic, tool-like, and magical. Through interviews with creative writers (n=18), we explored the role of these metaphors in participants’ sense of control and authorship. Our analysis resulted in a taxonomy of agency and ownership subtypes and underscored how metaphorical framings afforded participants different conceptions of agency and ownership. We conclude with recommendations for the design of AI co-writing systems, emphasizing the role of metaphors in participants’ creative practice.

Advances in artificial intelligence open the possibility of predicting and recommending sequences of GUI commands to a user. An interesting question raised by this capability is how to present such recommendations to the user – as a sequential set of individual command recommendations, or as one aggregated recommendation consisting of multiple commands. In this paper we propose an interface for aggregated command recommendation and conduct controlled studies to compare sequential versus aggregated command recommendation across a range of simulated utility conditions. Our results indicate that aggregated command recommendation can improve overall task performance over sequential recommendation, and that this benefit comes from enabling users to rapidly recognize and use high-utility aggregated recommendations. The aggregated command recommendation approach also reduced deliberation time when evaluating and correcting imperfect sets of recommended commands.

We explore the use of multimodal input to predict the landing position of a ray pointer while selecting targets in a virtual reality (VR) environment. We first extend a prior 2D Kinematic Template Matching technique to include head movements. This new technique, Head-Coupled Kinematic Template Matching, was found to improve upon the existing 2D approach, with an angular error of 10.0° when a user was 40% of the way through their movement. We then investigate two additional models that incorporated eye gaze, which were both found to further improve the predicted landing positions. The first model, Gaze-Coupled Kinematic Template Matching resulted in angular error of 6.8° for reciprocal target layouts and 9.1° for random target layouts, when a user was 40% of the way through their movement. The second model, Hybrid Kinematic Template Matching, resulted in angular error of 5.2° for reciprocal target layouts and 7.2° for random target layouts when a user was 40% of the way through their movement. We also found that using just the current gaze location resulted in sufficient predictions in many conditions. We reflect on our results by discussing the broader implications of utilizing multimodal input to inform selection predictions in VR.

The rapid growth of AR platforms, combined with the rising predictive power of intelligent systems, will fundamentally change interactive computing. Interaction will increasingly happen on the go, causing I/O to become constrained, ultimately leading to reliance on user intent prediction for aid. In this pictorial, we argue that to support the development of such systems, new predictive UI toolkits are required. We place the reader in the shoes of an App designer and outline the challenges that will be faced. We then describe a new predictive toolkit, leveraging Fuzzy Widgets, or “Fidgets” as the main UI building block. Fidgets extend Responsive Design into the realm of intelligent systems, to adapt not only to spatial constraints, but to system predictions as well. We then describe a working implementation of a predictive music application, built using our described framework, showcasing its benefits and range of adaptive abilities.

This innovative practice full paper finds that story-telling through animatronics can offer a combination of creativity and engineering in order to engage students in an inquiry-based approach to STEM learning. Animatronics is the art and science of bringing a story to life through robotic puppetry. Typically reserved for trained engineers with ample resources, animatronics can be an expressive yet inaccessible medium. We aim to allow students to write, create, and perform an animatronic story, learning valuable technical skills along the way. Our study, which took place over the course of eight weeks, investigates the use of our Paper Animatronics Kit in a Grade 2 and Grade 6 classroom, following the progression of their creative writing, animatronic building, and mentoring processes. We present results of our qualitative analysis of student and teacher interviews and group discussions.

Program visualizations help to form useful mental models of how programs work, and to reason and debug code. But these visualizations exist at a fixed level of abstraction, e.g., line-by-line. In contrast, programmers switch between many levels of abstraction when inspecting program behavior. Based on results from a formative study of hand-designed program visualizations, we designed CrossCode, a web-based program visualization system for JavaScript that leverages structural cues in syntax, control flow, and data flow to aggregate and navigate program execution across multiple levels of abstraction. In an exploratory qualitative study with experts, we found that CrossCode enabled participants to maintain a strong sense of place in program execution, was conducive to explaining program behavior, and helped track changes and updates to the program state.

Voice assistants have afforded users rich interaction opportunities to access information and issue commands in a variety of contexts. However, some users feel uneasy or creeped out by voice assistants, leading to a decreased desire to use them. As there has yet to be a comprehensive understanding of the factors that cause users to perceive voice assistants as being creepy, this research developed an empirical scale to measure the creepiness inherent in various voice assistants. Utilizing prior scale creation methodologies, a 7-item Perceived Creepiness of Voice Assistants Scale (PCAS) was created and validated. The scale measures how creepy a new voice assistant would be for users of voice assistants. The scale was developed to ensure that researchers and designers can evaluate the next generation of voice assistants before such voice assistants are released to the wider public.

Many XR devices use bare-hand gestures to reduce the need for handheld controllers. Such gestures, however, lead to false positive and false negative recognition errors, which detract from the user experience. While mediation techniques enable users to overcome recognition errors by clarifying their intentions via UI elements, little research has explored how mediation techniques should be designed in XR and how a user’s task and context may impact their design preferences. This research presents empirical studies about the impact of user perceived error costs on users’ preferences for three mediation technique designs, under different simulated scenarios that were inspired by real-life tasks. Based on a large-scale crowd-sourced survey and an immersive VR-based user study, our results suggest that the varying contexts within each task type can impact users’ perceived error costs, leading to different preferred mediation techniques. We further discuss the study implications of these results on future XR interaction design.

We demonstrate rich inferences about unaugmented everyday objects and hand object interactions by measuring minute skin surface deformations at the wrist using a sensing technique based on capacitance. The wristband prototype infers muscle and tendon tension, pose, and motion, which we then map to force (9 users, 13.66 +/- 9.84 N regression error on classes 0–49.1 N), grasp (9 users, 81 +/- 7 % classification accuracy on 6 grasps), and continuous interaction (10 users, 99 +/- 1 % discrimination accuracy between 6 interactions, 89–97 % accuracy on 3 states within each interaction) using basic machine learning models.

Gestural interaction has evolved from a set of novel interaction techniques developed in research labs, to a dominant interaction modality used by millions of users everyday. Despite its widespread adoption, the design of appropriate gesture vocabularies remains a challenging task for developers and designers. Existing research has largely used Expert-Led, User-Led, or Computationally-Based methodologies to design gesture vocabularies. These methodologies leverage the expertise, experience, and capabilities of experts, users, and systems to fulfill different requirements. In practice, however, none of these methodologies provide designers with a complete, multi-faceted perspective of the many factors that influence the design of gesture vocabularies, largely because a singular set of factors has yet to be established. Additionally, these methodologies do not identify or emphasize the subset of factors that are crucial to consider when designing for a given use case. Therefore, this work reports on the findings from an exhaustive literature review that identified 13 factors crucial to gesture vocabulary design and examines the evaluation methods and interaction techniques commonly associated with each factor. The identified factors also enable a holistic examination of existing gesture design methodologies from a factor-oriented viewpoint and highlighting the strengths and weaknesses of each methodology. This work closes with proposals of future research directions of developing an iterative user-centered and factor-centric gesture design approach as well as establishing an evolving ecosystem of factors that are crucial to gesture design.

Gaze-based interaction is a fast and ergonomic type of hands-free interaction that is often used with augmented and virtual reality when pointing at targets. Such interaction, however, can be cumbersome whenever user, tracking, or environmental factors cause eye tracking errors. Recent research has suggested that fallback modalities could be leveraged to ensure stable interaction irrespective of the current level of eye tracking error. This work thus presents Weighted Pointer interaction, a collection of error-aware pointing techniques that determine whether pointing should be performed by gaze, a fallback modality, or a combination of the two, depending on the level of eye tracking error that is present. These techniques enable users to accurately point at targets when eye tracking is accurate and inaccurate. A virtual reality target selection study demonstrated that Weighted Pointer techniques were more performant and preferred over techniques that required the use of manual modality switching.

This work explores the design of marking menus for gaze-based AR/VR menu selection by expert and novice users. It first identifies and explains the challenges inherent in ocular motor control and current eye tracking hardware, including overshooting, incorrect selections, and false activations. Through three empirical studies, we optimized and validated design parameters to mitigate these errors while reducing completion time, task load, and eye fatigue. Based on the findings from these studies, we derived a set of design guidelines to support gaze-based marking menus in AR/VR. To overcome the overshoot errors found with eye-based expert marking menu behaviour, we developed StickyPie, a marking menu technique that enables scale-independent marking input by estimating saccade landing positions. An evaluation of StickyPie revealed that StickyPie was easier to learn than the traditional technique (i.e., RegularPie) and was 10% more efficient after 3 sessions.

The craft of improvisational quilting involves working without the use of a predefined pattern. Design decisions are made “in the fabric,” with design experimentation tightly interleaved with the creation of the final artifact. To investigate how this type of design process can be supported, and to address challenges faced by practitioners, this paper presents PatchProv, a system for supporting improvisational quilt design. Based on a review of popular books on improvisational quilting, a set of design principles and key challenges to improvisational quilt design were identified, and PatchProv was developed to support the unique aspects of this process. An evaluation with a small group of quilters showed enthusiasm for the approach and revealed further possibilities for how computational tools can support improvisational quilting and improvisational design practices more broadly.

In virtual reality (VR) environments, asymmetric bimanual interaction techniques can increase users’ input bandwidth by complementing their perceptual and motor systems (e.g., using the dominant hand to select 3D UI controls anchored around the non-dominant arm). However, it is unclear how to optimize the layout of such 3D UI controls for near-body and mid-air interactions. We evaluate the performance and limitations of non-dominant arm-anchored 3D UIs in VR environments through a bimanual pointing study. Results demonstrated that targets appearing closer to the skin, located around the wrist, or placed on the medial side of the forearm could be selected more quickly than targets farther away from the skin, located around the elbow, or on the lateral side of the forearm. Based on these results, we developed Armstrong guidelines, demonstrated through a Unity plugin to enable designers to create performance-optimized arm-anchored 3D UI layouts.

Live streaming has become increasingly popular, with most streamers presenting their real-life appearance. However, Virtual YouTubers (VTubers), virtual 2D or 3D avatars that are voiced by humans, are emerging as live streamers and attracting a growing viewership in East Asia. Although prior research has found that many viewers seek real-life interpersonal interactions with real-person streamers, it is currently unknown what makes VTuber live streams engaging or how they are perceived differently than real-person streamers. We conducted an interview study to understand how viewers engage with VTubers and perceive the identities of the voice actors behind the avatars (i.e., Nakanohito). The data revealed that Virtual avatars bring unique performative opportunities which result in different viewer expectations and interpretations of VTuber behavior. Viewers intentionally upheld the disembodiment of VTuber avatars from their voice actors. We uncover the nuances in viewer perceptions and attitudes and further discuss the implications of VTuber practices to the understanding of live streaming in general.

Online video games support the development of social relationships through gameplay, however, gamers often cannot cultivate and maintain relationships based on social factors such as personality when using in-game matchmaking services. To address this, teammate matching sites external to games have emerged and enable gamers to offer to play games with others in exchange for payment. The affordances of these services are different from other existing gamer social sites, e.g., live streaming. Interviews were conducted with 16 dedicated users on Bixin, one of China’s largest paid teammate matching sites, to examine user motivations, practices, and perceptions. The interviews found that gamers selected paid teammates on Bixin using different criteria compared to in-game matchmaking services and emphasized the importance of real-life characteristics such as voice. To maintain connections, paid teammates often also extended communication to external communication services such as WeChat. Although most gamers expected to communicate with paid teammates as if they were friends, very few reported building real friendships with their matched counterparts.

Existing approaches to trading off false positive versus false negative errors in input recognition are based on imprecise ideas of how these errors affect user experience that are unlikely to hold for all situations. To inform dynamic approaches to setting such a tradeoff, two user studies were conducted on how relative preference for false positive versus false negative errors is influenced by differences in the temporal cost of error recovery, and high-level task factors (time pressure, multi-tasking). Participants completed a tile selection task in which false positive and false negative errors were injected at a fixed rate, and the temporal cost to recover from each of the two types of error was varied, and then indicated a preference for one error type or the other, and a frustration rating for the task. Responses indicate that the temporal costs of error recovery can drive both frustration and relative error type preference, and that participants exhibit a bias against false positive errors, equivalent to ∼1.5 seconds or more of added temporal recovery time. Several explanations for this bias were revealed, including that false positive errors impose a greater attentional demand on the user, and that recovering from false positive errors imposes a task switching cost.

The COVID-19 outbreak has resulted in a worldwide public health crisis. In such times of crisis, access to relevant and accurate information is critical. For many people in China, domestic social media platforms such as WeChat and Weibo have become dominant sources of COVID-19-related information and news. People have to evaluate the trustworthiness of COVID-19-related information and make sharing decisions using platforms that have to contend with government censorship policies, astroturfers, and other government interventions. We interviewed 33 Chinese WeChat users to understand how individuals were seeking COVID-19-related information and how they identified and evaluated specific COVID-19-related misinformation. This work exposes how COVID-19-related content with "positive energy" was prevalent on social media in China. A significant number of interviewees exhibited a willingness to prioritize information valence over veracity when evaluating and sharing content with others. Further, the work revealed how Chinese citizens' understanding of information ecosystems played an important role in their attitudes towards censorship and official media, and also influenced their evaluation of domestic and international information during a global crisis.

It is increasingly difficult for researchers to navigate and reach an understanding of a growing body of literature in a field of research. While past works in HCI and data visualization sought to support such activities, few investigated how these workflows are conducted in practice and how practices change in view of support tools. This work contributes a more holistic understanding of this space via a user-centered approach encompassing (a) a formative study on literature review practices of 15 researchers which informed (b) the design of LitSense, a proof-of-concept tool to support literature review workflows, and (c) a week-long study with 12 researchers performing a literature review with Litsense.

This paper presents a new technique to predict the ray pointer landing position for selection movements in virtual reality (VR) environments. The technique adapts and extends a prior 2D kinematic template matching method to VR environments where ray pointers are used for selection. It builds on the insight that the kinematics of a controller and Head-Mounted Display (HMD) can be used to predict the ray's final landing position and angle. An initial study provides evidence that the motion of the head is a key input channel for improving prediction models. A second study validates this technique across a continuous range of distances, angles, and target sizes. On average, the technique's predictions were within 7.3° of the true landing position when 50% of the way through the movement and within 3.4° when 90%. Furthermore, compared to a direct extension of Kinematic Template Matching, which only uses controller movement, this head-coupled approach increases prediction accuracy by a factor of 1.8x when 40% of the way through the movement.

The social media environment in China has become the dominant source of information and news over the past decade. This news environment has naturally suffered from challenges related to mis- and dis-information, encumbered by an increasingly complex landscape of factors and players including social media services, fact-checkers, censorship policies, and astroturfing. Interviews with 44 Chinese WeChat users were conducted to understand how individuals perceive misinformation and how it impacts their news consumption practices. Overall, this work exposes the diverse attitudes and coping strategies that Chinese users employ in complex social media environments. Due to the complex nature of censorship in China and participants' lack of understanding of censor-ship, they expressed varied opinions about its influence on the credibility of online information sources. Further, although most participants claimed that their opinions would not be easily swayed by astroturfers, many admitted that they could not effectively distinguish astroturfers from ordinary Internet users. Participants' inability to make sense of comments found online lead many participants to hold pro-censorship attitudes: the Government's Dividend.

Virtual reality has recently been adopted for use within the domain of visual analytics because it can provide users with an endless workspace within which they can be actively engaged and use their spatial reasoning skills for data analysis. However, virtual worlds need to utilize layouts and organizational schemes that are meaningful to the user and beneficial for data analysis. This paper presents DataHop, a novel visualization system that enables users to lay out their data analysis steps in a virtual environment. With a Filter, a user can specify the modification they wish to perform on one or more input data panels (i.e., containers of points), along with where output data panels should be placed in the virtual environment. Using this simple tool, highly intricate and useful visualizations may be generated and traversed by harnessing a user's spatial abilities. An exploratory study conducted with six virtual reality users evaluated the usability, affordances, and performance of DataHop for data analysis tasks, and found that spatially mapping one's workflow can be beneficial when exploring multidimensional datasets.

Prior research identified that physical paper documents have many positive attributes, for example natural tangibility and inherent physical flexibility. When documents are presented on digital devices, however, they can provide unique functionality to users, such as the ability to search, view dynamic multimedia content, and make use of indexing. This work explores the fusion of physical and digital paper documents. It first presents the results of a study that probed how users perform document-intensive analytical tasks when both physical and digital versions of documents were available. The study findings then informed the design of HoloDoc, a mixed reality system that augments physical artifacts with rich interaction and dynamic virtual content. Finally, we present the interaction techniques that HoloDoc affords, and the results of a second study that assessed HoloDoc's utility when working with digital and physical copies of academic articles.

Globalization has led to the destruction of many cultural practices, expressions, and knowledge found within local communities. These practices, defined by UNESCO as Intangible Cultural Heritage (ICH), have been identified, promoted, and safeguarded by nations, academia, organizations and local communities to varying degrees. Despite such efforts, many practices are still in danger of being lost or forgotten forever. With the increased popularity of livestreaming in China, some streamers have begun to use livestreaming to showcase and promote ICH activities. To better understand the practices, opportunities, and challenges inherent in sharing and safeguarding ICH through livestreaming, we interviewed 10 streamers and 8 viewers from China. Through our qualitative investigation, we found that ICH streamers had altruistic motivations and engaged with viewers using multiple modalities beyond livestreams. We also found that livestreaming encouraged real-time interaction and sociality, while non-live curated videos attracted attention from a broader audience and assisted in the archiving of knowledge.

Designers are often discouraged from creating data visualizations that omit or distort information, because they can easily be misleading. However, the same representations that could be used to deceive can provide benefits when chosen to appropriately align with user tasks. We present an interaction technique, Perceptual Glimpses, which allows for the transparent presentation of so-called 'deceptive' views of information that are made temporary using quasimodes. When presented using Perceptual Glimpses, message-level exaggeration caused by a truncated axis on a bar chart was reduced under some conditions, but users require guidance to avoid errors, and view presentation order may affect trust. When Perceptual Glimpses was extended to display a range of views that might otherwise be deceptive or difficult to understand if shown out of context, users were able to understand and leverage these transformations to perform a range of low-level tasks. Design recommendations and examples suggest extensions of the technique.

Virtual and Augmented Reality deliver engaging interaction experiences that can transport and extend the capabilities of the user. To ensure these paradigms are more broadly usable and effective, however, it is necessary to also deliver many of the conventional functions of a smartphone or personal computer. It remains unclear how conventional input tasks, such as text entry, can best be translated into virtual and augmented reality. In this paper we examine the performance potential of four alternative text entry strategies in virtual reality (VR). These four strategies are selected to provide full coverage of two fundamental design dimensions: i) physical surface association; and ii) number of engaged fingers. Specifically, we examine typing with index fingers on a surface and in mid-air and typing using all ten fingers on a surface and in mid-air. The central objective is to evaluate the human performance potential of these four typing strategies without being constrained by current tracking and statistical text decoding limitations. To this end we introduce an auto-correction simulator that uses knowledge of the stimulus to emulate statistical text decoding within constrained experimental parameters and use high-precision motion tracking hardware to visualise and detect fingertip interactions. We find that alignment of the virtual keyboard with a physical surface delivers significantly faster entry rates over a mid-air keyboard. Also, users overwhelmingly fail to effectively engage all ten fingers in mid-air typing, resulting in slower entry rates and higher error rates compared to just using two index fingers. In addition to identifying the envelopes of human performance for the four strategies investigated, we also provide a detailed analysis of the underlying features that distinguish each strategy in terms of its performance and behaviour.

An ideal Mixed Reality (MR) system would only present virtual information (e.g., a label) when it is useful to the person. However, deciding when a label is useful is challenging: it depends on a variety of factors, including the current task, previous knowledge, context, etc. In this paper, we propose a Reinforcement Learning (RL) method to learn when to show or hide an object's label given eye movement data. We demonstrate the capabilities of this approach by showing that an intelligent agent can learn cooperative policies that better support users in a visual search task than manually designed heuristics. Furthermore, we show the applicability of our approach to more realistic environments and use cases (e.g., grocery shopping). By posing MR object labeling as a model-free RL problem, we can learn policies implicitly by observing users' behavior without requiring a visual search model or data annotation.

We present Plane, Ray, and Point, a set of interaction techniques that utilizes shape constraints to enable quick and precise object alignment and manipulation in virtual reality. Users create the three types of shape constraints, Plane, Ray, and Point, by using symbolic gestures. The shape constraints are used like scaffoldings and limit and guide the movement of virtual objects that collide or intersect with them. The same set of gestures can be performed with the other hand, which allow users to further control the degrees of freedom for precise and constrained manipulation. The combination of shape constraints and bimanual gestures yield a rich set of interaction techniques to support object transformation. An exploratory study conducted with 3D design experts and novice users found the techniques to be useful in 3D scene design workflows and easy to learn and use.

We present PseudoBend, a haptic feedback technique that creates the illusion that a rigid device is being stretched, bent, or twisted. The method uses a single 6-DOF force sensor and a vibrotactile actuator to render grain vibrations to simulate the vibrations produced during object deformation based on the changes in force or torque exerted on a device. Because this method does not require any moving parts aside from the vibrotactile actuator, devices designed using this method can be small and lightweight. Psychophysical studies conducted using a prototype that implements this method confirmed that the method could be used to successfully create the illusion of deformation and could also change users' perception of stiffness by changing the virtual stiffness parameters.

Prior research has demonstrated that users are increasingly employing multiple devices during daily work. Currently, devices such as keyboards, cell phones, and tablets remain largely unaware of their role within a user's workflow. As a result, transitioning between devices is tedious, often to the degree that users are discouraged from taking full advantage of the devices they have within reach. This work explores the device ecologies used in desk-centric environments and complies the insights observed into SMAC, a simplified model of attention and capture that emphasizes the role of user-device proxemics, as mediated by hand placement, gaze, and relative body orientation, as well as inter-device proxemics. SMAC illustrates the potential of harnessing the rich, proxemic diversity that exists between users and their device ecologies, while also helping to organize and synthesize the growing body of literature on distributed user interfaces. An evaluation study using SMAC demonstrated that users could easily understand the tenants of user- and inter-device proxemics and found them to be valuable within their workflows.

The livestreaming industry in China is gaining greater traction than its European and North American counterparts and has a profound impact on the stakeholders' online and offline lives. An emerging genre of livestreaming that has become increasingly popular in China is outdoor livestreaming. With outdoor livestreams, streamers broadcast outdoor activities, travel, or socialize with passersby in outdoor settings, often for 6 or more hours, and viewers watch such streams for hours each day. However, given that professionally produced content about travel and outdoor activities are not very popular, it is currently unknown what makes this category of livestreams so engaging and how these techniques can be applied to other content or genres. Thus, we conducted a mixed methods study consisting of a survey (N=287) and interviews (N = 20) to understand how viewers watch and engage with outdoor livestreams in China. The data revealed that outdoor livestreams encompass many categories of content, environments and passersby behaviors create challenges and uncertainty for viewers and streamers, and viewers watch livestreams for surprising lengths of time (e.g., sometimes more than 5 continuous hours). We also gained insights into how live commenting and virtual gifting encourage engagement. Lastly, we detail how the behaviors of dedicated fans and casual viewers differ and provide implications for the design of livestreaming services that support outdoor activities.

Growing interest in personal fabrication has resulted in many ways to ideate, design, and prototype, in addition to studies of who a maker is and the challenges they face. Less attention, however, has focused on the role of the environment in fabrication processes. By understanding how interactions with tools, fixtures, materials, and spaces shape workflows, we can better determine how to design the next generation of workshops, design tools, and fabrication equipment to support personal fabrication activities. To build this understanding, site visits and interviews at local makerspaces, fabrication studios, and workshops were conducted. These visits uncovered the rich practices and roadblocks generated by workshops today. The observations identified the importance of spatial layouts, territoriality and occupant agency, distributed knowledge, and organizational flux, among others, to design and fabrication processes. These observations were further synthesized into one possible direction for such spaces: hybrid workshops (i.e., environments that can leverage computation and responsive architecture to enhance a maker's ability to design and fabricate). This work identifies how such spaces could harness the rich practices and eliminate the challenges found with workshops today and discusses the technical innovations and philosophical questions that hybrid workshops will pose to the future of personal fabrication.

Prewriting is the process of generating and organizing ideas before drafting a document. Although often overlooked by novice writers and writing tool developers, prewriting is a critical process that improves the quality of a final document. To better understand current prewriting practices, we first conducted interviews with writing learners and experts. Based on the learners' needs and experts' recommendations, we then designed and developed InkPlanner, a novel pen and touch visualization tool that allows writers to utilize visual diagramming for ideation during prewriting. InkPlanner further allows writers to sort their ideas into a logical and sequential narrative by using a novel widget - NarrativeLine. Using a NarrativeLine, InkPlanner can automatically generate a document outline to guide later drafting exercises. Inkplanner is powered by machine-generated semantic and structural suggestions that are curated from various texts. To qualitatively review the tool and understand how writers use InkPlanner for prewriting, two writing experts were interviewed and a user study was conducted with university students. The results demonstrated that InkPlanner encouraged writers to generate more diverse ideas and also enabled them to think more strategically about how to organize their ideas for later drafting.

We present a new handheld haptic device, Thor's Hammer, which uses propeller propulsion to generate ungrounded, 3-DOF force feedback. Thor's Hammer has six motors and propellers that generates strong thrusts of air without the need for physical grounding or heavy air compressors. With its location and orientation tracked by an optimal tracking system, the system can exert forces in arbitrary directions regardless of the device's orientation. Our technical evaluation shows that Thor's Hammer can apply up to 4 N of force in arbitrary directions with less than 0.11 N and 3.9° of average magnitude and orientation errors. We also present virtual reality applications that can benefit from the force feedback provided by Thor's Hammer. Using these applications, we conducted a preliminary user study and participants felt the experience more realistic and immersive with the force feedback.

Despite gaining traction in North America, live streaming has not reached the popularity it has in China, where live- streaming has a tremendous impact on the social behaviors of users. To better understand this socio-technological phenomenon, we conducted a mixed methods study of live streaming practices in China. We present the results of an online survey of 527 live streaming users, focusing on their broadcasting or viewing practices and the experiences they find most engaging. We also interviewed 14 active users to explore their motivations and experiences. Our data revealed the different categories of content that was broadcasted and how varying aspects of this content engaged viewers. We also gained insight into the role reward systems and fan group-chat play in engaging users, while also finding evidence that both viewers and streamers desire deeper channels and mechanisms for interaction in addition to the commenting, gifting, and fan groups that are available today.

Clinical practice is heavily reliant on the use of unstructured text to document patient stories due to its expressive and flexible nature. However, a physician's capacity to recover information from text for clinical overview is severely affected when records get longer and time pressure increases. Data visualization strategies have been explored to aid in information retrieval by replacing text with graphical summaries, though often at the cost of omitting important text features. This causes physician mistrust and limits real-world adoption. This work presents our investigation into the role and use of text in clinical practice, and reports on efforts to assess the best of both worlds---text and visualization---to facilitate clinical overview. We report on insights garnered from a field study, and the lessons learned from an iterative design process and evaluation of a text-visualization prototype, MedStory, with 14 medical professionals. The results led to a number of grounded design recommendations to guide visualization design to support clinical text overview.

Creating whimsical, personal data visualizations remains a challenge due to a lack of tools that enable for creative visual expression while providing support to bind graphical content to data. Many data analysis and visualization creation tools target the quick generation of visual representations, but lack the functionality necessary for graphics design. Toolkits and charting libraries offer more expressive power, but require expert programming skills to achieve custom designs. In contrast, sketching affords fluid experimentation with visual shapes and layouts in a free-form manner, but requires one to manually draw every single data point. We aim to bridge the gap between these extremes. We propose DataInk, a system supports the creation of expressive data visualizations with rigorous direct manipulation via direct pen and touch input. Leveraging our commonly held skills, coupled with a novel graphical user interface, DataInk enables direct, fluid, and flexible authoring of creative data visualizations.

Virtual Reality enables users to explore content whose physics are only limited by our creativity. Such limitless environments provide us with many opportunities to explore innovative ways to support productivity and collaboration. We present Spacetime, a scene editing tool built from the ground up to explore the novel interaction techniques that empower single user interaction while maintaining fluid multi-user collaboration in immersive virtual environment. We achieve this by introducing three novel interaction concepts: the Container, a new interaction primitive that supports a rich set of object manipulation and environmental navigation techniques, Parallel Objects, which enables parallel manipulation of objects to resolve interaction conflicts and support design workflows, and Avatar Objects, which supports interaction among multiple users while maintaining an individual users' agency. Evaluated by professional Virtual Reality designers, Spacetime supports powerful individual and fluid collaborative workflows.

Knowledge-sharing live streams are distinct from traditional educational videos, at least because of the large concurrently-viewing audience and the real-time discussions between viewers and the streamer. Though this creates unique opportunities for interactive learning, it also brings a challenge for creating a useful archive for post hoc learning. This paper presents the results of interviews with knowledge sharing streamers, their moderators, and viewers to understand current experiences and needs for sharing and learning knowledge through live streaming. Based on those findings, we built StreamWiki, a tool which leverages the viewers during live streams to produce useful archives of the interactive learning experience. On StreamWiki, moderators initiate tasks that viewers complete by conducting microtasks, such as writing a summary, commenting, and voting for informative comments. As a result, a summary document is built in real time. Through the tests of our prototype with streamers and viewers, we found that StreamWiki could help understanding the content and the context of the stream, during the stream and for post hoc learning.

When fabricating, it is common to follow a prescribed set of steps in a tutorial or how-to. While popular, such explicit knowledge resources have many inconsistencies and omissions, use static illustrations, and cannot adapt to drop-in makers or a maker's mistakes. To overcome many of these issues, this work presents Automatics, a novel explicit knowledge resource system that dynamically generates fabrication activities for one or more makers based on their current environmental and fabrication context. Automatics assigns tasks to makers based on the past tools and components the maker was working with, enables makers to recover from mistakes through model regeneration, suggests alternative tools if a needed tool is unavailable or in use, and allows multiple makers to drop-in throughout a fabrication activity. Initial usage and feedback from novice makers showed that Automatics increases the number of tasks that can be completed compared to paper instructions, decreases frustration, and improves one's understanding of the global context of assigned tasks during fabrication activities.

PhenoLines is a visual analysis tool for the interpretation of disease subtypes, derived from the application of topic models to clinical data. Topic models enable one to mine cross-sectional patient comorbidity data (e.g., electronic health records) and construct disease subtypes-each with its own temporally evolving prevalence and co-occurrence of phenotypes-without requiring aligned longitudinal phenotype data for all patients. However, the dimensionality of topic models makes interpretation challenging, and de facto analyses provide little intuition regarding phenotype relevance or phenotype interrelationships. PhenoLines enables one to compare phenotype prevalence within and across disease subtype topics, thus supporting subtype characterization, a task that involves identifying a proposed subtype's dominant phenotypes, ages of effect, and clinical validity. We contribute a data transformation workflow that employs the Human Phenotype Ontology to hierarchically organize phenotypes and aggregate the evolving probabilities produced by topic models. We introduce a novel measure of phenotype relevance that can be used to simplify the resulting topology. The design of PhenoLines was motivated by formative interviews with machine learning and clinical experts. We describe the collaborative design process, distill high-level tasks, and report on initial evaluations with machine learning experts and a medical domain expert. These results suggest that PhenoLines demonstrates promising approaches to support the characterization and optimization of topic models.

Current touch devices are adept at tracking finger touches, but cannot distinguish if multiple touches are caused by different fingers on a single hand, by fingers from both hands of a single user, or by different users. This limitation significantly reduces the possibilities for interaction techniques in touch interfaces. We present GhostID, a capacitive sensor that can differentiate the origins of multiple simultaneous touches. Our approach analyzes the signal ghosting, already present as an artifact in a frequency-division touch controller, to differentiate touches from the same hand or different hands of a single user (77% reliability at 60 fps) or from two different users (95% reliability at 60 fps). In addition to GhostID, we also develop a framework of user-differentiation capabilities for touch input devices, and illustrate a set of interaction techniques enabled by GhostID.

Despite the long development of Graphical User Interfaces, working with multiple graphical objects remains a challenge, due to the difficulties of forming complex selections, ambiguities of operations, and tediousness of repetitively unselect-reselect or ungroup-regroup objects. Instead of tackling them as individual problems, we attribute it to the lack of system support to the general selection-action cycles. We propose Collection Objects to not only support a single fast selection-action cycle but also allow multiple cycles to be chained together into a fluid workflow. Collection Objects unifies selection, grouping, and manipulation of aggregate selections into a single object, with which selection can be composed with various techniques, modified for later actions, grouped with objects inside still directly accessible, and quasi-moded for less context switching. We implemented Collection Object in the context of a vector drawing application with simultaneous pen and touch input. Results of an expert evaluation show that Collection Objects holds considerable promises for fluid interaction with multiple objects.

Cross-sectional phenotype studies are used by genetics researchers to better understand how phenotypes vary across patients with genetic diseases, both within and between cohorts. Analyses within cohorts identify patterns between phenotypes and patients (e.g., co-occurrence) and isolate special cases (e.g., potential outliers). Comparing the variation of phenotypes between two cohorts can help distinguish how different factors affect disease manifestation (e.g., causal genes, age of onset, etc.). PhenoStacks is a novel visual analytics tool that supports the exploration of phenotype variation within and between cross-sectional patient cohorts. By leveraging the semantic hierarchy of the Human Phenotype Ontology, phenotypes are presented in context, can be grouped and clustered, and are summarized via overviews to support the exploration of phenotype distributions. The design of PhenoStacks was motivated by formative interviews with genetics researchers: we distil high-level tasks, present an algorithm for simplifying ontology topologies for visualization, and report the results of a deployment evaluation with four expert genetics researchers. The results suggest that PhenoStacks can help identify phenotype patterns, investigate data quality issues, and inform data collection design.

Within this work, a novel metaphor, haunted design, is explored to challenge the definitions of "display" used today. Haunted design draws inspiration and vision from some of the most multi-modal and sensory diverse experiences that have been reported, the paranormal and hauntings. By synthesizing and deconstructing such phenomena, four novel opportunities to direct display design were uncovered, e.g., intensity, familiarly, tangibility, and shareability. A large-scale design probe, The Living Room, guided the ideation and prototyping of design concepts that exemplify facets of haunted design. By combining the opportunities, design concepts, and survey responses, a framework highlighting the importance of objects, their behavior, and the resulting phenomena to haunted design was developed. Given its emphasis on the odd and unusual, the haunted design metaphor should great spur conversation and alternative directions for future display-based user experiences.

Foldem, a novel method of rapid fabrication of objects with multi-material properties is presented. Our specially formulated Foldem sheet allows users to fabricate and easily assemble objects with rigid, bendable, and flexible properties using a standard laser-cutter. The user begins by creating his designs in a vector graphics software package. A laser cutter is then used to fabricate the design by selectively ablating/vaporizing one or more layers of the Foldem sheet to achieve the desired physical properties for each joint. Herein the composition of the Foldem sheet, as well as various design considerations taken into account while building and designing the method, are described. Sample objects made with Foldem are demonstrated, each showcasing the unique attributes of Foldem. Additionally, a novel method for carefully calibrating a laser cutter for precise ablation is presented.

We report on the Latency Hammer, a low-cost yet highaccuracy and high-precision automated tool that measures the interface latency of touchscreen devices. The Hammer directly measures latency by triggering a capacitive touch event on a device using an electrically actuated touch simulator, and a photo sensor to monitor the screen for a visual response. This allows us to measure the full end-toend latency of a touchscreen system exactly as it would be experienced by a user. The Hammer does not require human interaction to perform a measurement, enabling the acquisition of large datasets. We present the operating principles of the Hammer, and discuss its design and construction; full design documents are available online. We also present a series of tools and equipment that were built to assess and validate the performance of the Hammer, and demonstrate that it provides reliable latency measurements.

Advances in display and tracking technologies hold the promise of increasingly immersive augmented-reality experiences. Unfortunately, the on-demand generation of haptic experiences is lagging behind these advances in other feedback channels. We present Annexing Reality; a system that opportunistically annexes physical objects from a user's current physical environment to provide the best-available haptic sensation for virtual objects. It allows content creators to a priori specify haptic experiences that adapt to the user's current setting. The system continuously scans user's surrounding, selects physical objects that are similar to given virtual objects, and overlays the virtual models on to selected physical ones reducing the visual-haptic mismatch. We describe the developer's experience with the Annexing Reality system and the techniques utilized in realizing it. We also present results of a developer study that validates the usability and utility of our method of defining haptic experiences.

We present Object-Oriented Drawing, which replaces most WIMP UI with Attribute Objects. Attribute Objects embody the attributes of digital content as UI objects that can be manipulated through direct touch gestures. In the paper, the fundamental UI concepts are presented, including Attribute Objects, which may be moved, cloned, linked, and freely associated with drawing objects. Other functionalities, such as attribute-level blending and undo, are also demonstrated. We developed a drawing application based on the presented concepts with simultaneous touch and pen input. An expert assessment of our application shows that direct physical manipulation of Attribute Objects enables a user to quickly perform interactions which were previously tedious, or even impossible, with a coherent and consistent interaction experience throughout the entire interface.

Augmented and virtual reality have the potential of being indistinguishable from the real world. Holographic displays, including head mounted units, support this vision by creating rich stereoscopic scenes, with objects that appear to float in thin air - often within arm's reach. However, one has but to reach out and grasp nothing but air to destroy the suspension of disbelief. Snake-charmer is an attempt to provide physical form to virtual objects by revisiting the concept of Robotic Graphics or Encountered-type Haptic interfaces with current commodity hardware. By means of a robotic arm, Snake-charmer brings physicality to a virtual scene and explores what it means to truly interact with an object. We go beyond texture and position simulation and explore what it means to have a physical presence inside a virtual scene. We demonstrate how to render surface characteristics beyond texture and position, including temperature; how to physically move objects; and how objects can physically interact with the user's hand. We analyze our implementation, present the performance characteristics, and provide guidance for the construction of future physical renderers.

The differential diagnosis of hereditary disorders is a challenging task for clinicians due to the heterogeneity of phenotypes that can be observed in patients. Existing clinical tools are often text-based and do not emphasize consistency, completeness, or granularity of phenotype reporting. This can impede clinical diagnosis and limit their utility to genetics researchers. Herein, we present PhenoBlocks, a novel visual analytics tool that supports the comparison of phenotypes between patients, or between a patient and the hallmark features of a disorder. An informal evaluation of PhenoBlocks with expert clinicians suggested that the visualization effectively guides the process of differential diagnosis and could reinforce the importance of complete, granular phenotypic reporting.

Mobile devices offer constant connectivity to the world, which can negatively affect in-person interaction. Current approaches to minimizing the social disruption and improving the subtlety of interactions tend to focus on the development of inconspicuous devices that provide basic input or output. This paper presents a more general approach to subtle interaction and demonstrates how a number of principles from magic can be leveraged to improve subtlety. It also presents a framework that can be used to classify subtle interfaces along with a modular set of novel interfaces that fit within this framework. Lastly, the paper presents a new evaluation paradigm specifically designed to assess the subtlety of interactions. This paradigm is used to compare traditional approaches to our new subtle approaches. We find our new approaches are over five times more subtle than traditional interactions, even when participants are aware of the technologies being used.

This paper reports on two experiments designed to further our understanding of users' perception of latency in touch- based systems. The first experiment extends previous efforts to measure latency perception by reporting on a unified study in which direct and indirect form-factors are compared for both tapping and dragging tasks. Our results show significant effects from both form-factor and task, and inform system designers as to what input latencies they should aim to achieve in a variety of system types. A follow-up experiment investigates peoples' ability to perceive small improvements to latency in direct and indirect form-factors for tapping and dragging tasks. Our results provide guidance to system designers of the relative value of making improvements in latency that reduce but do not fully eliminate lag from their systems.

In this work, we explore moveables, i.e., interactive papercraft that harness user interaction to generate visual effects. First, we present a survey of children's books that captured the state of the art of moveables. The results of this survey were synthesized into a moveable taxonomy and informed MoveableMaker, a new tool to assist users in designing, generating, and assembling moveable papercraft. MoveableMaker supports the creation and customization of a number of moveable effects and employs moveable-specific features including animated tooltips, automatic instruction generation, constraint-based rendering, techniques to reduce material waste, and so on. To understand how MoveableMaker encourages creativity and enhances the workflow when creating moveables, a series of exploratory workshops were conducted. The results of these explorations, including the content participants created and their impressions, are discussed, along with avenues for future research involving moveables.

Printem film, a novel method for the fabrication of Printed Circuit Boards (PCBs) for small batch/prototyping use, is presented. Printem film enables a standard office inkjet or laser printer, using standard inks, to produce a PCB: the user prints a negative of the PCB onto the film, exposes it to UV or sunlight, and then tears-away the unneeded portion of the film, leaving-behind a copper PCB. PCBs produced with Printem film are as conductive as PCBs created using standard industrial methods. Herein, the composition of Printem film is described, and advantages of various materials discussed. Sample applications are also described, each of which demonstrates some unique advantage of Printem film over current prototyping methods: conductivity, flexibility, the ability to be cut with a pair of scissors, and the ability to be mounted to a rigid backplane.

The emergence of smart devices (e.g., smart watches and smart eyewear) is redefining mobile interaction from the solo performance of a smart phone, to a symphony of multiple devices. In this paper, we present Duet -- an interactive system that explores a design space of interactions between a smart phone and a smart watch. Based on the devices' spatial configurations, Duet coordinates their motion and touch input, and extends their visual and tactile output to one another. This transforms the watch into an active element that enhances a wide range of phone-based interactive tasks, and enables a new class of multi-device gestures and sensing techniques. A technical evaluation shows the accuracy of these gestures and sensing techniques, and a subjective study on Duet provides insights, observations, and guidance for future work.

We introduce Splash, a framework reducing development overhead for both data curators and visualization developers of client-server visualization systems. Splash streamlines the process of creating a multiple level-of-detail version of the data and facilitates progressive data download, thereby enabling real-time, on-demand navigation with existing visualization toolkits. As a result, system responsiveness is increased and the user experience is improved. We demonstrate the benefit of progressive loading for user interaction on slower networks. Additionally, case study evaluations of Splash with real-world data curators suggest that Splash supports iterative refinement of visualizations and promotes the use of exploratory data analysis.

The proliferation of inexpensive connected devices has created a situation where a person, at any given moment, is surrounded by interactive computers. Despite this fact, there are very few means by which a user may take advantage of this large number of screens. We present Conductor, a prototype framework which serves as an exemplar for the construction of cross-device applications. We present a series of interaction methods by which users can easily share information, chain tasks across devices, and manage sessions across devices. We also present a cross-device usage scenario which utilizes several cross-device applications built within our prototype framework. We also describe a user study, which helped us to understand how users will take advantage of a large number of devices in support of performance of a sense making task.

Large-scale touch surfaces have been widely studied in literature and adopted for public installations such as interactive billboards. However, current designs do not take into consideration that touching the interactive surface at different heights is not the same; for body-height displays, the bottom portion of the screen is within easier reach of the foot than the hand. We explore the design space of foot input on vertical surfaces, and propose three distinct interaction modalities: hand, foot tapping, and foot gesturing. Our design exploration pays particular attention to areas of the touch surface that were previously overlooked: out of hand's reach and close to the floor. We instantiate our design space with a working prototype of an interactive surface, in which we are able to distinguish between finger and foot tapping and extend the input area beyond the bottom of the display to support foot gestures.

Unlock gestures are performed by billions of users across the world multiple times a day. Beyond preventing accidental input on mobile devices, they currently serve little to no other purpose. In this paper, we explore how replacing the regular unlock screen with one that asks the user to perform a simple, optional task, can benefit a wealth of application domains, including data collection, personal-health metrics collection, and human intelligence tasks. We evaluate this concept, which we refer to as Slide to X. Further, we show that people are willing to perform microtasks presented through this interface and continue to do so throughout the day while they visit different locations as part of their daily routines. We then discuss how to implement this concept and demonstrate three applications.

We present Panelrama, a web-based framework for the construction of applications using distributed user interfaces (DUIs). Our implementation provides developers with low migration costs through built-in mechanisms for the synchronization of a UI state, requiring minimal changes to existing languages. Additionally, we describe a solution to categorize device characteristics and dynamically change UI allocation to best-fit devices. We illustrate the use of Panelrama through three sample applications which demonstrate its support for known interaction methods, we also present the results of a developer study, which validates our belief that cross-device application experiences can be easily implemented using our framework.

We present "Fast Multi-Touch" (FMT), an extremely high frame rate and low-latency multi-touch sensor based on a novel projected capacitive architecture that employs simultaneous orthogonal signals. The sensor has a frame rate of 4000 Hz and a touch-to-data output latency of only 40 microseconds, providing unprecedented responsiveness. FMT is demonstrated with a high-speed DLP projector yielding a touch-to-light latency of 110 microseconds.

A method of reducing the perceived latency of touch input by employing a model to predict touch events before the finger reaches the touch surface is proposed. A corpus of 3D finger movement data was collected, and used to develop a model capable of three granularities at different phases of movement: initial direction, final touch location, time of touchdown. The model is validated for target distances >= 25.5cm, and demonstrated to have a mean accuracy of 1.05cm 128ms before the user touches the screen. Preference study of different levels of latency reveals a strong preference for unperceived latency touchdown feedback. A form of 'soft' feedback, as well as other uses for this prediction to improve performance, is proposed.

Hotkeys are an efficient method of selecting commands on a keyboard. However, these shortcuts are often underused by users. We present Métamorphe, a novel keyboard with keys that can be individually raised and lowered to promote hotkeys usage. Métamorphe augments the output of traditional keyboards with haptic and visual feedback, and offers a novel design space for user input on raised keys (e.g., gestures such as squeezing or pushing the sides of a key). We detail the implementation of Métamorphe and discuss design factors. We also report two user studies. The first is a user-defined interface study that shows that the new input vocabulary is usable and useful, and provides insights into the mental models that users associate with raised keys. The second user study shows improved eyes-free selection performance for raised keys as well as the surrounding unraised keys.

Although advances in touchscreen technology have provided us with more precise devices, touchscreens are still laden with latency issues. Common commercial devices present with latency up to 125ms. Although these levels have been shown to impact users' perception of the responsiveness of the system [16], relatively little is known about the impact of latency on the performance of tasks common to direct-touch interfaces, such as direct physical manipulation.

Web applications designed for map exploration in local neighborhoods have become increasingly popular and important in everyday life. During the information-seeking process, users often revisit previously viewed locations, repeat earlier searches, or need to memorize or manually mark areas of interest. To facilitate rapid returns to earlier views during map exploration, we propose a novel algorithm to automatically generate map bookmarks based on a user's interaction. TrailMap, a web application based on this algorithm, is developed, providing a fluid and effective neighborhood exploration experience. A one-week study is conducted to evaluate TrailMap in users' everyday web browsing activities. Results showed that TrailMap's implicit bookmarking mechanism is efficient for map exploration and the interactive and visual nature of the tool is intuitive to users.

With the selection of devices encompassing a wider range of computing surfaces, along with the near ubiquity of wireless networks, the nature of the workspace has become distributed over multiple locations and digital artifacts. We interviewed 22 professionals across a wide range of industries about their use of artifacts in their workflows to dis-cover new cross-device interaction paradigms and issues. We explore the impact of today's cloud services and app-based computing on how devices are used together. Gaps in data management and cross-device interactions were identified as the main obstacles and opportunities for improvement for multi-device interaction.

SNOUT is a novel interface overlay designed for occasional no-hand or one-handed use of handheld capacitive touch devices. Inspired by the desire to use these devices in scenarios where visually focused bimanual input is awkward, we performed a pair of studies intended to evaluate the potential of the nose to provide touch input. These studies influenced our design principles, resulting in the construction of a 'nose mode' which enables object selection, continuous parameter control, and speech-based text entry. Selection is accomplished via a nose tap, using a colour overlay and peripheral colour feedback to correct mistakes. The other two techniques are activated by a nose tap, but use the accelerometer to control parameters and speech-to-text for text entry. An evaluation of SNOUT shows it to effectively render handheld capacitive touch devices operational in scenarios where they are presently unusable.

When the user is engaged with a real-world task it can be inappropriate or difficult to use a smartphone. To address this concern, we developed ShoeSense, a wearable system consisting in part of a shoe-mounted depth sensor pointing upward at the wearer. ShoeSense recognizes relaxed and discreet as well as large and demonstrative hand gestures. In particular, we designed three gesture sets (Triangle, Radial, and Finger-Count) for this setup, which can be performed without visual attention. The advantages of ShoeSense are illustrated in five scenarios: (1) quickly performing frequent operations without reaching for the phone, (2) discreetly performing operations without disturbing others, (3) enhancing operations on mobile devices, (4) supporting accessibility, and (5) artistic performances. We present a proof-of-concept, wearable implementation based on a depth camera and report on a lab study comparing social acceptability, physical and mental demand, and user preference. A second study demonstrates a 94-99% recognition rate of our recognizers.

Multi-touch technology lends itself to collaborative crowd interaction (CI). However, common tap-operated widgets are impractical for CI, since they are susceptible to accidental touches and interference from other users. We present a novel multi-touch interface called FlowBlocks in which every UI action is invoked through a small sequence of user actions: dragging parametric UI-Blocks, and dropping them over operational UI-Docks. The FlowBlocks approach is advantageous for CI because it a) makes accidental touches inconsequential; and b) introduces design parameters for mutual awareness, concurrent input, and conflict management. FlowBlocks was successfully used on the floor of a busy natural history museum. We present the complete design space and describe a year-long iterative design and evaluation process which employed the Rapid Iterative Test and Evaluation (RITE) method in a museum setting.

Software designed for direct-touch interfaces often utilize a metaphor of direct physical manipulation of pseudo "real-world" objects. However, current touch systems typically take 50-200ms to update the display in response to a physi-cal touch action. Utilizing a high performance touch de-monstrator, subjects were able to experience touch latencies ranging from current levels down to about 1ms. Our tests show that users greatly prefer lower latencies, and noticea-ble improvement continued well below 10ms. This level of performance is difficult to achieve in commercial compu-ting systems using current technologies. As an alternative, we propose a hybrid system that provides low-fidelity visu-al feedback immediately, followed by high-fidelity visuals at standard levels of latency.

We present Magic Finger, a small device worn on the fingertip, which supports always-available input. Magic Finger inverts the typical relationship between the finger and an interactive surface: with Magic Finger, we instrument the user's finger itself, rather than the surface it is touching. Magic Finger senses touch through an optical mouse sensor, enabling any surface to act as a touch screen. Magic Finger also senses texture through a micro RGB camera, allowing contextual actions to be carried out based on the particular surface being touched. A technical evaluation shows that Magic Finger can accurately sense 22 textures with an accuracy of 98.9%. We explore the interaction design space enabled by Magic Finger, and implement a number of novel interaction techniques that leverage its unique capabilities.

Touch screen surfaces large enough for ten-finger input have become increasingly popular, yet typing on touch screens pales in comparison to physical keyboards. We examine typing patterns that emerge when expert users of physical keyboards touch-type on a flat surface. Our aim is to inform future designs of touch screen keyboards, with the ultimate goal of supporting touch-typing with limited tactile feedback. To study the issues inherent to flat-glass typing, we asked 20 expert typists to enter text under three conditions: (1) with no visual keyboard and no feedback on input errors, then (2) with and (3) without a visual keyboard, but with some feedback. We analyzed touch contact points and hand contours, looking at attributes such as natural finger positioning, the spread of hits among individual keys, and the pattern of non-finger touches. We also show that expert typists exhibit spatially consistent key press distributions within an individual, which provides evidence that eyes-free touch-typing may be possible on touch surfaces and points to the role of personalization in such a solution. We conclude with implications for design.

In this paper, we explore the use of a projection-based handheld device to facilitate in-clinic doctor-patient communication. We present the user-centered design process used to understand the workflow of medical professionals and to identify challenges they currently face in communicating information to patients. Based on the lessons learned, we developed AnatOnMe, a prototype projection-based hand-held system for enhancing information exchange in the current practice of one medical sub-specialty, physical therapy. We then present the results of a controlled experiment to understand the desirability and learning tradeoffs of using AnatOnMe to teach medical concepts on three potential projection surfaces - wall, model, and patient body. Finally, we present results of two expert reviews of the system.

Direct touch manipulations enable the user to interact with the on-screen content in a direct and easy manner closely mimicking the spatial manipulations in the physical world. However, they also suffer from well-known issues of precision, occlusion and an inability to isolate different degrees of freedom in spatial manipulations. We present a set of interactions, called Rock & Rails, that augment existing direct touch manipulations with shape-based gestures, thus providing on-demand gain control, occlusion avoidance, and separation of constraints in 2D manipulation tasks. Using shape gestures in combination with direct-manipulations allows us to do this without ambiguity in detection and without resorting to manipulation handles, which break the direct manipulation paradigm. Our set of interactions were evaluated by 8 expert graphic designers and were found to be easy to learn and master, as well as effective in accomplishing a precise graphical layout task.

We present Medusa, a proximity-aware multi-touch tabletop. Medusa uses 138 inexpensive proximity sensors to: detect a user's presence and location, determine body and arm locations, distinguish between the right and left arms, and map touch point to specific users and specific hands. Our tracking algorithms and hardware designs are described. Exploring this unique design, we develop and report on a collection of interactions enabled by Medusa in support of multi-user collaborative design, specifically within the context of Proxi-Sketch, a multi-user UI prototyping tool. We discuss design issues, system implementation, limitations, and generalizable concepts throughout the paper.

Many touch screens remain inaccessible to blind users, and those approaches to providing access that do exist offer minimal support for interacting with large touch screens or spatial data. In this paper, we introduce a set of three software-based access overlays intended to improve the accessibility of large touch screen interfaces, specifically interactive tabletops. Our access overlays are called edge projection, neighborhood browsing, and touch-and-speak. In a user study, 14 blind users compared access overlays to an implementation of Apple's VoiceOver screen reader. Our results show that two of our techniques were faster than VoiceOver, that participants correctly answered more questions about the screen's layout using our techniques, and that participants overwhelmingly preferred our techniques. We developed several applications demonstrating the use of access overlays, including an accessible map kiosk and an accessible board game.

Emerging technologies provide platforms for new devices, applications, and user interfaces. These technologies have shown potential in early research, but their true utility and measures of success lie in their ability to reflect and enhance the capabilities of the people who use them.

Groups of users often have shared information needs -- for example, business colleagues need to conduct research relating to joint projects and students must work together on group homework assignments. In this paper, we introduce WeSearch, a collaborative Web search system designed to leverage the benefits of tabletop displays for face-to-face collaboration and organization tasks. We describe the design of WeSearch and explain the interactions it affords. We then describe an evaluation in which eleven groups used WeSearch to conduct real collaborative search tasks. Based on our study's findings, we analyze the effectiveness of the features introduced by WeSearch.

We present WeSpace -- a collaborative work space that integrates a large data wall with a multi-user multi-touch table. WeSpace has been developed for a population of scientists who frequently meet in small groups for data exploration and visualization. It provides a low overhead walk-up and share environment for users with their own personal applications and laptops. We present our year-long effort from initial ethnographic studies, to iterations of design, development and user testing, to the current experiences of these scientists carrying out their collaborative research in the WeSpace. We shed light on the utility, the value of the multi-touch table, the manifestation, usage patterns and the changes in their workflow that WeSpace has brought about.

We present Ripples, a system which enables visualizations around each contact point on a touch display and, through these visualizations, provides feedback to the user about successes and errors of their touch interactions. Our visualization system is engineered to be overlaid on top of existing applications without requiring the applications to be modified in any way, and functions independently of the application's responses to user input. Ripples reduces the fundamental problem of ambiguity of feedback when an action results in an unexpected behaviour. This ambiguity can be caused by a wide variety of sources. We describe the ambiguity problem, and identify those sources. We then define a set of visual states and transitions needed to resolve this ambiguity, of use to anyone designing touch applications or systems. We then present the Ripples implementation of visualizations for those states, and the results of a user study demonstrating user preference for the system, and demonstrating its utility in reducing errors.

Many research projects have demonstrated the benefits of bimanual interaction for a variety of tasks. When choosing bimanual input, system designers must select the input device that each hand will control. In this paper, we argue for the use of pen and touch two-handed input, and describe an experiment in which users were faster and committed fewer errors using pen and touch input in comparison to using either touch and touch or pen and pen input while performing a representative bimanual task. We present design principles and an application in which we applied our design rationale toward the creation of a learnable set of bimanual, pen and touch input commands.

The interoperability of disparate data types and sources has been a long standing problem and a hindering factor for the efficacy and efficiency in visual exploration applications. In this paper, we present a solution, called LivOlay, that enables the rapid visual overlay of live data rendered in different applications. Our tool addresses datasets in which visual registration of the information is necessary in order to allow for thorough understanding and visual analysis. We also discuss initial evaluation and user feedback of LivOlay.

We investigate the differences -- in terms of bothquantitative performance and subjective preference -- between direct-touch and mouse input for unimanual andbimanual tasks on tabletop displays. The results of twoexperiments show that for bimanual tasks performed ontabletops, users benefit from direct-touch input. However,our results also indicate that mouse input may be moreappropriate for a single user working on tabletop tasksrequiring only single-point interaction.

Volumetric displays, which provide a 360° view of imagery illuminated in true 3D space, are a promising platform for interactive 3D applications. However, presenting text in volumetric displays can be a challenge, as the text may not be oriented towards the user. This is especially problematic with multiple viewers, as the text could, for example, appear forwards to one user, and backwards to another. In a first experiment we determined the effects of 3D rotations on text readability. Based on the results, we developed and evaluated a new technique which optimizes text orientation for multiple viewers. This technique provided 33% faster group reading times in a collaborative experimental task.

Information shown on a tabletop display can appear distorted when viewed by a seated user. Even worse, the impact of this distortion is different depending on the location of the information on the display. In this paper, we examine how this distortion affects the perception of the basic graphical elements of information visualization shown on displays at various angles. We first examine perception of these elements on a single display, and then compare this to perception across displays, in order to evaluate the effectiveness of various elements for use in a tabletop and multi-display environment. We found that the perception of some graphical elements is more robust to distortion than others. We then develop recommendations for building data visualizations for these environments.

Touch is a compelling input modality for interactive devices; however, touch input on the small screen of a mobile device is problematic because a user's fingers occlude the graphical elements he wishes to work with. In this paper, we present LucidTouch, a mobile device that addresses this limitation by allowing the user to control the application by touching the back of the device. The key to making this usable is what we call pseudo-transparency: by overlaying an image of the user's hands onto the screen, we create the illusion of the mobile device itself being semi-transparent. This pseudo-transparency allows users to accurately acquire targets while not occluding the screen with their fingers and hand. Lucid Touch also supports multi-touch input, allowing users to operate the device simultaneously with all 10 fingers. We present initial study results that indicate that many users found touching on the back to be preferable to touching on the front, due to reduced occlusion, higher precision, and the ability to make multi-finger input.

Tables have historically played a key role in many real-time collaborative environments, often referred to as "war rooms". Today, these environments have been transformed by computational technology into spaces with large vertical displays surrounded by numerous desktop computers. However, despite significant research activity in the area of tabletop computing, very little is known about how to best integrate a digital tabletop into these multi-surface environments. In this paper, we identify various design requirements for the implementation of a system intended to support such an environment. We then present a set of designs that demonstrate how an interactive tabletop can be used in a real-time operations center to facilitate collaborative situation-assessment and decision-making.

In many environments, it is often the case that input is made to displays that are positioned non-traditionally relative to one or more users. This typically requires users to perform interaction tasks under transformed input-display spatial mappings, and the literature is unclear as to how such transformations affect performance. We present two experiments that explore the impact of display space position and input control space orientation on user's subjective preference and objective performance in a docking task. Our results provide guidelines as to optimal display placement and control orientation in collaborative computing environments with one or more shared displays.

Visual search is the subject of countless psychology studies in which people search for target items within a scene. The bulk of this literature focuses on the individual with the goal of understanding the human perceptual system. In life, visual search is performed not only by individuals, but also by groups -- a team of doctors may study an x-ray and a team of analysts may study a satellite photograph. In this paper, we examine the issues one should consider when searching as a group. We present the details of an experiment designed to investigate the impact of group size on visual search performance, and how different display configurations affected that performance. We asked individuals, pairs, and groups of four people to participate in a baggage screening task in which these teams searched simulated x-rays for prohibited items. Teams conducted these searches on single monitors, a row of four monitors, and on a single horizontal display. Our findings suggest that groups commit far fewer errors in visual search tasks, although they may perform slower than individuals under certain conditions. The interaction between group size and display configuration turned out to be an important factor as well.

In this paper, we discuss our adaptation of a single-display, single-user commercial application for use in a multi-device, multi-user environment. We wrap Google Earth, a popular geospatial application, in a manner that allows for synchronized coordinated views among multiple instances running on different machines in the same co-located environment. The environment includes a touch-sensitive tabletop display, three vertical wall displays, and a TabletPC. A set of interaction techniques that allow a group to manage and exploit this collection of devices is presented.

We explore the design space of a two-sided interactive touch table, designed to receive touch input from both the top and bottom surfaces of the table. By combining two registered touch surfaces, we are able to offer a new dimension of input for co-located collaborative groupware. This design accomplishes the goal of increasing the relative size of the input area of a touch table while maintaining its direct-touch input paradigm. We describe the interaction properties of this two-sided touch table, report the results of a controlled experiment examining the precision of user touches to the underside of the table, and a series of application scenarios we developed for use on inverted and two-sided tables. Finally, we present a list of design recommendations based on our experiences and observations with inverted and two-sided tables.

Volumetric displays provide interesting opportunities and challenges for 3D interaction and visualization, particularly when used in a highly interactive manner. We explore this area through the design and implementation of techniques for interactive direct manipulation of objects with a 3D volumetric display. Motion tracking of the user's fingers provides for direct gestural interaction with the virtual objects, through manipulations on and around the display's hemispheric enclosure. Our techniques leverage the unique features of volumetric displays, including a 360° viewing volume that enables manipulation from any viewpoint around the display, as well as natural and accurate perception of true depth information in the displayed 3D scene. We demonstrate our techniques within a prototype 3D geometric model building application.

The numeric keypads on mobile phones generally consist of 12 keys (0-9, *, #). Ambiguity arises when the 36-character alpha-numeric English alphabet is mapped onto this smaller number of keys. In this paper, we first present a taxonomy of the various techniques for resolving this ambiguity, dividing them into techniques that use consecutive actions to first select a character grouping and then a character from within that grouping, and those that use concurrent actions to achieve the same end. We then present the design and implementation of a chording approach to text entry that uses concurrent key presses. We conducted a controlled experiment that compared this chording technique to one-handed and two-handed versions of the commonly used MultiTap technique. The results show that the concurrent chording technique significantly outperforms both versions of the consecutive action MultiTap technique.

Volumetric displays provide interesting opportunities and challenges for 3D interaction and visualization, particularly when used in a highly interactive manner. We explore this area through the design and implementation of techniques for interactive direct manipulation of objects with a 3D volumetric display. Motion tracking of the user's fingers provides for direct gestural interaction with the virtual objects, through manipulations on and around the display's hemispheric enclosure. Our techniques leverage the unique features of volumetric displays, including a 360° viewing volume that enables manipulation from any viewpoint around the display, as well as natural and accurate perception of true depth information in the displayed 3D scene. We demonstrate our techniques within a prototype 3D geometric model building application.

TiltText, a new technique for entering text into a mobile phone is described. The standard 12-button text entry keypad of a mobile phone forces ambiguity when the 26- letter Roman alphabet is mapped in the traditional manner onto keys 2-9. The TiltText technique uses the orientation of the phone to resolve this ambiguity, by tilting the phone in one of four directions to choose which character on a particular key to enter. We first discuss implementation strategies, and then present the results of a controlled experiment comparing TiltText to MultiTap, the most common text entry technique. The experiment included 10 participants who each entered a total of 640 phrases of text chosen from a standard corpus, over a period of about five hours. The results show that text entry speed including correction for errors using TiltText was 23% faster than MultiTap by the end of the experiment, despite a higher error rate for TiltText. TiltText is thus amongst the fastest known language-independent techniques for entering text into mobile phones.

Hunter Gatherer is an interface that lets Web users carry out three main tasks: (1) collect components from within Web pages; (2) represent those components in a collection; (3) edit those component collections. Our research shows that while the practice of making collections of content from within Web pages is common, it is not frequent, due in large part to poor interaction support in existing tools. We engaged with users in task analysis as well as iterative design reviews in order to understand the interaction issues that are part of within-Web-page collection making and to design an interaction that would support that process.We report here on that design development, as well as on the evaluations of the tool that evolved from that process, and the future work stemming from these results, in which our critical question is: what happens to users perceptions and expectations of web-based information (their web-based information management practices) when they can treat this information as harvestable, recontextualizable data, rather than as fixed pages?