What would it mean for a digital assistant to stay with our cursor, rather than as a chatbot in a separate window? By staying near the user’s actions, such assistance promises lightweight, continuous, in-the-flow help, but also introduces unique risks of intrusion. However, little is known about the tasks it is suited for, and the design considerations it entails. This study offers an initial investigation of this space through a multi-stage design inquiry. First, a retrospective think-aloud study with nine participants reveals common inefficiencies in everyday information work where cursor-centric support may be valuable. Building on these observations, we proposed the Fairy Cursor as a design probe and developed a proof-of-concept environment to examine users’ interpretations, preferences and concerns around it. Our findings surfaced key challenges and opportunities in designing cursor-centric assistants, including how users tolerate presence, interpret initiatives, and balance assistance with ongoing engagement. We conclude with design implications for in-the-flow human–AI collaboration and outline directions for future systems.

Recent work identified clarity as one of the top quality attributes that notebook users value, but notebooks lack support for maintaining clarity throughout the exploratory phases of the notebook authoring workflow. We propose always-clear notebook authoring that supports both clarity and exploration, and present a Jupyter implementation called Tidynote. The key to Tidynote is three-fold: (1) a scratchpad sidebar to facilitate exploration, (2) cells movable between the notebook and the scratchpad to maintain organization, and (3) linear execution with state forks to clarify program state. An exploratory study (N=13) of open-ended data analysis tasks shows that Tidynote features holistically promote clarity throughout a notebook’s lifecycle, support realistic notebook tasks, and enable novel strategies for notebook clarity. These results suggest that Tidynote supports maintaining clarity throughout the entirety of notebook authoring.

Creative practitioners rely on tools to capture and manage ideas as a foundational aspect of their work. However, we have little knowledge about how idea management practices vary in different creative domains. Combining insights from qualitative surveys ( N \(\sim\) 200) and follow-up in-depth interviews ( n \(\sim\) 60) with creative professionals from four domains (interaction design, research, music, and graphics) of creative work, we report on (1) how ideas are externalized in practitioners’ archives, (2) what they consider important when choosing tools to capture ideas, and (3) how these tool collections resemble and differ from each other. Our cross-domain study demonstrates that participants’ tool use reflects idea capture characteristic needs as well as domain-specific views about the creative process. We conclude with a discussion about capturing as an externalizing activity, practitioners’ use of the term ideas , and four suggestions for directions in the design of creativity support tools.

Science and knowledge communication is the process of informing and engaging the public about a diverse array of topics, such as science, health, philosophy, and history. Effective science and knowledge communication is challenging because communicators need to balance several factors, such as the complexity of topics, viewers' diverse backgrounds, and the characteristics of the medium. With the widespread availability of design tools and platforms to create and share content, laypeople can disseminate knowledge widely through online platforms. Popular science and knowledge communication video channels on YouTube, for example, have millions or tens of millions of subscribers, as well as millions or billions of accumulated views. Even with the growing popularity of science and knowledge communication videos, there is little understanding of the practices the creators use to make and increase the reach of their videos, the challenges they encounter while doing so, and how these videos impact viewers. This paper reports on interviews conducted with 27 creators of popular science and knowledge communication videos on YouTube and 13 viewers of these creators' videos. We present the motivations of creators and viewers, the practices creators use for broad science and knowledge communication, and the challenges encountered by members of the community.

Audiovisual recordings of user studies and interviews provide important data in qualitative HCI research. Even when a textual transcription is available, researchers frequently turn to these recordings due to their rich information content. However, the temporal, unstructured nature of audiovisual recordings makes them less efficient to work with than text. Through interviews and a survey, we explored how HCI researchers work with audiovisual recordings. We investigated researchers’ transcription and annotation practice, their overall analysis workflow, and the prevalence of direct analysis of audiovisual recordings. We found that a key task was locating and analyzing inspectables, interesting segments in recordings. Since locating inspectables can be time consuming, participants look for detectables, visual or auditory cues that indicate the presence of an inspectable. Based on our findings, we discuss the potential for automation in locating detectables in qualitative audiovisual analysis.

Computational notebooks combine code, visualizations, and text in a single document. Researchers, data analysts, and even journalists are rapidly adopting this new medium. We present three studies of how they are using notebooks to document and share exploratory data analyses. In the first, we analyzed over 1 million computational notebooks on GitHub, finding that one in four had no explanatory text but consisted entirely of visualizations or code. In a second study, we examined over 200 academic computational notebooks, finding that although the vast majority described methods, only a minority discussed reasoning or results. In a third study, we interviewed 15 academic data analysts, finding that most considered computational notebooks personal, exploratory, and messy. Importantly, they typically used other media to share analyses. These studies demonstrate a tension between exploration and explanation in constructing and sharing computational notebooks. We conclude with opportunities to encourage explanation in computational media without hindering exploration.

Computational notebooks aim to support collaborative data analysis by combining code, visualizations, and text in a single easily shared document. Yet, as notebooks evolve and grow they often become difficult to navigate or understand, discouraging sharing and reuse. We present the design and evaluation of a Jupyter Notebook extension providing facilities for annotated cell folding. Through a lab study and multi-week deployment we find cell folding aids notebook navigation and comprehension, not only by the original author, but also by collaborators viewing the notebook in a meeting or revising it on their own. However, in some cases cell folding encouraged collaborators to overlook folded sections or spend longer reviewing a notebook before editing it. These findings extend our understanding of code folding's trade-offs to a new medium and demonstrate its benefits for everyday collaboration. We conclude by discussing how dynamic reorganization can support sharing and reuse of computational notebooks.

Resuming suspended activities is fundamental to knowledge work. As activities grow in complexity or fade from memory, they become increasingly difficult to cognitively resume. Motivated by the efficiency of images for cueing autobiographical memory, we conducted 2 studies of how visual histories of computer-mediated activity might aid users in restoring the mental context of suspended activities. In the first exploratory study we demonstrate that users provided a visual history consisting of small thumbnail images of their desktop can reconstruct detailed mental contexts of earlier computer work. In the second observational study we examine methods and cues that web developers, scientific writers, and graphic designers currently use to restore mental context. We find that participants from all three professions engage in situated sensemaking, reconstructing context by generating stories to explain cues in their documents with implicit meaning, such as sudden changes in writing quality or the shape of blocks of code. We identify characteristics of each activity that influence the information, artifacts, and strategies participants used to reconstruct context. We conclude that visual histories can help users reconstruct mental context but that challenges remain in designing lightweight image-based cues that help users reconstruct context for a range of activities.

In this paper we describe the development of a novel paper-digital interface for recording movement trajectories, designed to assist ethnographers and ethologists in analysis of social movement. While we focus on development of a system to aid analysis of elephant movement, the resulting interaction techniques and facilities are quite general. The paper highlights how our design evolved to balance the goals of researchers, their current practices, and the challenges of integrating the relatively unconstrained world of pen and paper with the relatively constrained world of digital systems.

Documentation of development is a critical aspect of the work of early childhood education teachers. Through field observations and interviews, we detail the process and tools used in documenting development for children age three months to five-years-old at one school. Teachers use paper forms and printed photos to record and analyze observations of development. The evolving final product is a paper portfolio of development. This practice relies heavily on the teacher's ability to objectively observe children in situ, create a record of the activity, and make salient the link between evidence and developmental milestones. We describe current paper and photo documentation practices in light of an online record keeping system that will be introduced at this school within the next year. The present analysis contributes to a growing literature on the role of paper and digital media in documentation.

Photographs are powerful and pervasive media that facilitate communication and support reminiscence. Adding audio narrations to traditional paper photographs combines the familiarity of paper photos with the nuanced richness of voice. We enable creation of and interaction with audio-enhanced paper photographs through custom software deployed on a digital pen. This paper reports on use of a paper-digital photo album by an older adult (Ethel, age 105), her extended family, and nursing staff over a five-month field deployment. The interactive photo album was found to be easy to use and accessible, effectively engaged Ethel's and family members' interest, and provided a focal point for communication and social interaction. Family and caregivers report improvements in Ethel's social interaction. We discuss the properties of our audio-enhanced paper approach that make it a promising medium for engaging older adults and distill general design considerations for paper-digital photo albums.

Hybrid paper-digital interfaces are a promising approach for supporting language activities. The familiarity of pen and paper makes it a particularly attractive media for many user groups, including young children. Digital pens enhance interaction with traditional paper content by playing and recording audio and recognizing handwriting and gestures. Currently, generating custom interactive paper documents involves some programming, limiting its use by many user groups (e.g., educators and families) who might especially benefit from application of hybrid paper-digital interfaces in their practices. To address this need, we developed an end-user Toolkit for Authoring Pen and Paper Language Activities (TAP & PLAY). This paper describes the iterative development of the toolkit, its accessibility for novice non-technical users, and use in three different contexts for early language learning. We demonstrate and document the system's usability, generality, and utility for people who want to create and tailor their own custom interactive paper-based language activities.

Researchers from many disciplines are taking advantage of increasingly inexpensive digital video to capture extensive records of human activity in real-world settings. The ability to record and share such data has created a critical moment in the practice and scope of behavioral research. While recent work is beginning to develop techniques for visualizing and interacting with integrated multimodal information collected during field research, navigating and analyzing these large datasets remains challenging and tools are especially needed to support the early stages of data exploration.

Large-scale display walls, and the high-resolution visualizations they support, promise to become ubiquitous. Natural interaction with them, especially in collaborative environments, is increasingly important and yet remains an on-going challenge. Part of the problem is a resolution mismatch between low-resolution input devices and high-resolution display walls. In addition, enabling concurrent use by multiple users is difficult - for example, how would this large workspace be managed for multiple users and what novel collaborative interactions could occur? In this paper, we present an overlay interface element superimposed on wall-display applications to help constrain interaction, focus attention on subsections of a display wall, and facilitate a collaborative multi-user workflow.

The intertwining of everyday life and computation, along with a new generation of inexpensive digital recording devices and storage facilities, is revolutionizing our ability to collect and analyze human activity data. Such ubiquitous data collection has an exciting potential to augment human cognition and radically improve information-intensive work. In this paper we introduce a system to aid the process of data collection and analysis during observational research by providing non-intrusive automatic capture of paper-based annotations. The system exploits current note-taking practices and incorporates digital pen technology. We describe the development, deployment and use of the system for interactive visualization and annotation of multiple stream of video and other types of time-based data.

Digital social media have transformed how we communicate and manage our relationships. Despite its portability, sketching as a social medium has been largely left behind. Given sketching's unique affordances for visual communication this absence is a real loss. Sketches convey visuo-spatial ideas directly, require minimal detail to render concepts, and show the peculiarities of handwriting. Sketching holds the promise to enrich how we communicate, and its ubiquity is critical for sharing information at opportune moments. We present the results of an exploratory field study of ubiquitous sketching for social media, documenting users' experiences with UbiSketch. This system integrates digital pens, paper, and mobile phones to support the transmission of paper sketches to online services. We learned that UbiSketch enabled participants to leverage sketching's unique affordances, that ubiquitous sketching creates a synergy with the practice of posting context-dependent information, and that it broadens and deepens social interaction.

After a stroke or brain injury, it may be more difficult to understand language and communicate with others. Speech-language therapy may help an individual regain language and cope with changes in their communication abilities. Our research examines the process of speech-language therapy with an emphasis on the practices of therapists working with adults with aphasia and apraxia of speech. This paper presents findings from field work undertaken to inform the design of a mixed paper-digital interface prototype using multimodal digital pens. We describe and analyze therapists' initial reactions to the system and present two case studies of use by older adults undergoing speech-language therapy. We discuss the utility of multimodal paper-digital interfaces to assist therapy and describe our vision of a system to help therapists independently create custom interactive paper materials for their clients.

This paper examines accessibility issues of surface computing with older adults and explores the appeal of surface computing for health care support. We present results from a study involving 20 older adults (age 60 to 88) performing gesture-based interactions on a multitouch surface. Older adults were able to successfully perform all actions on the surface computer, but some gestures that required two fingers (resize) and fine motor movement (rotate) were problematic. Ratings for ease of use and ease of performing each action as well as time required to figure out an action were similar to that of younger adults. Older adults reported that the surface computer was less intimidating, less frustrating, and less overwhelming than a traditional computer. The idea of using a surface computer for health care support was well-received by participants. We conclude with a discussion of design issues involving surface computing for older adults and use of this technology for health care.

The unbundling of the monolithic computer and its movement off the desktop and into the world are creating new work and living practices. For good and for ill, our professional, personal, and social activities are increasingly dependent on and intertwined with digital computation and communication facilities. Visual interfaces through which we typically interact with such systems date from Ivan Sutherland's seminal Sketchpad system in the 60's and now are commonplace on devices ranging from phones to ultrahigh resolution display walls. Interfaces continue to be a main focus of HCI and information visualization research and applications. In this talk I argue that focusing on the interface is only one of multiple important and necessary levels of design to support information-based activities and unless we attend more seriously to social, cognitive, and embodied levels, we will continue to fall far short of realizing the full potential of computer technology to aid thought, communication, and social interaction.

In this paper we compare the affordances of presenting educational material on a tabletop display with presenting the same material using traditional paper handouts. Ten pairs of undergraduate students used digital or paper materials to prepare for exams during four one-hour study sessions over the course of a term. Students studying with the tabletop display solved problems on their own before resorting to answer keys and repeated activities more often than students studying with paper documents. We summarize study activities and discuss the benefits and drawbacks of each medium.

We present Silicone iLluminated Active Peripherals (SLAP), a system of tangible, translucent widgets for use on multitouch tabletops. SLAP Widgets are cast from silicone or made of acrylic, and include sliders, knobs, keyboards, and buttons. They add tactile feedback to multi-touch tables, improving input accuracy. Using rear projection, SLAP Widgets can be relabeled dynamically, providing inexpensive, battery-free, and untethered augmentations. Furthermore, SLAP combines the flexibility of virtual objects with physical affordances. We evaluate how SLAP Widgets influence the user experience on tabletops compared to virtual controls. Empirical studies show that SLAPWidgets are easy to use and outperform virtual controls significantly in terms of accuracy and overall interaction time.

We present SLAPbook, an application using SLAP, translucent and tangible widgets for use on vision-based multi-touch tabletops in Single Display Groupware (SDG) environments. SLAP stands for Silicone ILluminated Active Peripherals and includes widgets such as sliders, knobs, keyboards, and buttons. The widgets and tactile feedback to multi-touch tables while simultaneously providing dynamic relabeling to tangible objects using the table's rear projection. SLAPbook provides multiple users the ability to add and edit content to a guestbook, browse other peoples' entries, and access personal data using a token-based personalization system. Interaction with the table takes place in the personal and public space so that users can make use of personal and shared controls to perform separate and coordinative actions.

Being mobile influences not only the types of information people seek but also the ways they attempt to access it. Mobile contexts present challenges of changing location and social context, restricted time for information access, and the need to share attentional resources among concurrent activities. Understanding mobile information needs and associated interaction challenges is fundamental to improving designs for mobile phones and related devices. We conducted a two-week diary study to better understand mobile information needs and how they are addressed. Our study revealed that depending on the time and resources available, as well as the situational context, people use diverse and, at times, ingenious ways to obtain needed information. We summarize key findings and discuss design implications for mobile technology.

This paper describes the design and evaluation of Shared Speech Interface (SSI), an application for an interactive multitouch tabletop display designed to facilitate medical conversations between a deaf patient and a hearing, non-signing physician. We employ a participatory design process involving members of the deaf community as well as medical and communication experts. We report results from an evaluation that compares conversation when facilitated by: (1) a digital table, (2) a human sign language interpreter, and (3) both a digital table and an interpreter. Our research reveals that tabletop displays have valuable properties for facilitating discussion between deaf and hearing individuals as well as enhancing privacy and independence. The contributions of this work include initial guidelines for cooperative group work technology for users with varying hearing abilities, discussion of benefits of participatory design with the deaf community, and lessons about using dictated speech on shared displays.

Tactile feedback allows devices to communicate with users when visual and auditory feedback are inappropriate. Unfortunately, current vibrotactile feedback is abstract and not related to the content of the message. This often clash-es with the nature of the message, for example, when sending a comforting message.

Paper persists as an integral component of active reading and other knowledge-worker tasks because it provides ease of use unmatched by digital alternatives. Paper documents are light to carry, easy to annotate, rapid to navigate, flexible to manipulate, and robust to use in varied environments. Interactions with paper documents create rich webs of annotation, cross reference, and spatial organization. Unfortunately, the resulting webs are confined to the physical world of paper and, as they accumulate, become increasingly difficult to store, search, and access. XLibris [Schilit et al. 1998] and similar systems address these difficulties by simulating paper with tablet PCs. While this approach is promising, it suffers not only from limitations of current tablet computers (e.g., limited screen space) but also from loss of invaluable paper affordances. In this article, we describe PapierCraft, a gesture-based command system that allows users to manipulate digital documents using paper printouts as proxies. Using an Anoto [Anoto 2002] digital pen, users can draw command gestures on paper to tag a paragraph, e-mail a selected area, copy selections to a notepad, or create links to related documents. Upon pen synchronization, PapierCraft executes the commands and presents the results in a digital document viewer. Users can then search the tagged information and navigate the web of annotated digital documents resulting from interactions with the paper proxies. PapierCraft also supports real time interactions across mix-media, for example, letting users copy information from paper to a Tablet PC screen. This article presents the design and implementation of the PapierCraft system and describes user feedback from initial use.

This work describes a multi-year ethnography-for-design study in a dental hygiene training program in San Diego, USA. We performed an ethnographic analysis of instructional practices used in clinical instruction, and helped design a new training laboratory, equipped with digital media technology, through which students and instructors could access a video blogging ('vlogging') system. We present an analysis of how the vlogging system transformed social and instructional interactions in clinical training. Additionally, we describe how the faculty's appropriation of vlog technology was initially challenged by the presentation of divergent methodology in vlog records, and increased transparency of teaching practices in video records.

We explore and extend the metaphor of "piles" to include computationally-enriched piles, portable regions of automation in Dynapad, a multiscale workspace. We describe these collection-management tools and how their design was informed by observing people organizing collections of personal digital photographs

IRYS is a tool for the replay and analysis of gaze and touch behavior during on-line activities. Essentially a multimodal VCR, it can record and replay computer screen activity and overlay this video with a synchronized spotlight of the user's attention, as measured by an eye-tracking and/or touch-tracking system. This cross-platform tool is particularly useful for detailed ethnographic analysis of natural on-line behavior involving multiple applications and windows in a continually changing workspace.

We are quickly passing through the historical moment when people work in front of a single computer, dominated by a small CRT and focused on tasks involving only local information. Networked computers are becoming ubiquitous and are playing increasingly significant roles in our lives and in the basic infrastructures of science, business, and social interaction. For human-computer interaction to advance in the new millennium we need to better understand the emerging dynamic of interaction in which the focus task is no longer confined to the desktop but reaches into a complex networked world of information and computer-mediated interactions. We think the theory of distributed cognition has a special role to play in understanding interactions between people and technologies, for its focus has always been on whole environments: what we really do in them and how we coordinate our activity in them. Distributed cognition provides a radical reorientation of how to think about designing and supporting human-computer interaction. As a theory it is specifically tailored to understanding interactions among people and technologies. In this article we propose distributed cognition as a new foundation for human-computer interaction, sketch an integrated research framework, and use selections from our earlier work to suggest how this framework can provide new opportunities in the design of digital work materials.

No abstract available.

Article KidPad: a design collaboration between children, technologists, and educators Share on Authors: Allison Druin Computer Science Department/College of Education, University of New Mexico, Farris Engineering Center, Albuquerque, New Mexico Computer Science Department/College of Education, University of New Mexico, Farris Engineering Center, Albuquerque, New MexicoView Profile , Jason Stewart Computer Science Department/College of Education, University of New Mexico, Farris Engineering Center, Albuquerque, New Mexico Computer Science Department/College of Education, University of New Mexico, Farris Engineering Center, Albuquerque, New MexicoView Profile , David Proft Computer Science Department/College of Education, University of New Mexico, Farris Engineering Center, Albuquerque, New Mexico Computer Science Department/College of Education, University of New Mexico, Farris Engineering Center, Albuquerque, New MexicoView Profile , Ben Bederson Computer Science Department/College of Education, University of New Mexico, Farris Engineering Center, Albuquerque, New Mexico Computer Science Department/College of Education, University of New Mexico, Farris Engineering Center, Albuquerque, New MexicoView Profile , Jim Hollan Computer Science Department/College of Education, University of New Mexico, Farris Engineering Center, Albuquerque, New Mexico Computer Science Department/College of Education, University of New Mexico, Farris Engineering Center, Albuquerque, New MexicoView Profile Authors Info & Claims CHI '97: Proceedings of the ACM SIGCHI Conference on Human factors in computing systemsMarch 1997 Pages 463–470https://doi.org/10.1145/258549.258866Published:27 March 1997 90citation2,232DownloadsMetricsTotal Citations90Total Downloads2,232Last 12 Months62Last 6 weeks8 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

No abstract available.

We describe the current status of Pad++, a zooming graphical interface that we are exploring as an alternative to traditional window and icon-based approaches to interface design. We discuss the motivation for Pad++, describe the implementation, and present prototype applications. In addition, we introduce an informational physics strategy for interface design and briefly compare it with metaphor-based design strategies.

No abstract available.

We describe two applications that illustrate the idea of computational wear in the domain of document processing. By graphically depicting the history of author and reader interactions with documents, these applications offer otherwise unavailable information to guide work. We discuss how their design accords with a theory of professional work and an informational physics perspective on interface design.

A belief in the efficacy of imitating face-to-face communication is an unquestioned presupposition of most current work on supporting communications in electronic media. In this paper we highlight problems with this presupposition and present an alternative proposal for grounding and motivating research and development that frames the issue in terms of needs, media, and mechanisms. To help elaborate the proposal we sketch a series of example projects and respond to potential criticisms.

In this paper, we introduce ephemeral interest groups for supporting informal communication. Ephemeral interest groups are electronic discussion groups that, in contrast to bulletin boards and the like, are short-lived and ad hoc. They are designed as a medium for informal discussions of items broadcast to a wider community. We have implemented a prototype system to explore ephemeral interest groups. We discuss the goals of the system, characterize its evolution over the last ten months of deployment, and sketch our plans for future developments.

The authors maintain that of particular importance for visualization excellence is an understanding of effective deictic facilities, especially new techniques made possible by computation. They explain what deixis is and why it is fundamental to visualization and they analyze some of the requirements for effective deixis in the context of emergent visualization technology.>

article Free Access Share on CHI research at MCC Authors: J. Hollan Microelectronics and Computer Technology Corporation Microelectronics and Computer Technology CorporationView Profile , B. Curtis Microelectronics and Computer Technology Corporation Microelectronics and Computer Technology CorporationView Profile Authors Info & Claims ACM SIGCHI BulletinVolume 20Issue SIMarch 1989 pp 147–149https://doi.org/10.1145/67450.67479Published:01 March 1989Publication History 0citation237DownloadsMetricsTotal Citations0Total Downloads237Last 12 Months14Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Direct manipulation has been lauded as a good form of interface design, and some interfaces that have this property have been well received by users. In this article we seek a cognitive account of both the advantages and disadvantages of direct manipulation interfaces. We identify two underlying phenomena that give rise to the feeling of directness. One deals with the information processing distance between the user's intentions and the facilities provided by the machine. Reduction of this distance makes the interface feel direct by reducing the effort required of the user to accomplish goals. The second phenomenon concerns the relation between the input and output vocabularies of the interface language. In particular, direct manipulation requires that the system provide representations of objects that behave as if they are the objects themselves. This provides the feeling of directness of manipulation.