What if text could be sculpted and refined like clay—or cultivated and pruned like a plant? Texterial reimagines text as a material that users can grow, sculpt, and transform. Current generative-AI models enable rich text operations, yet rigid, linear interfaces often mask such capabilities. We explore how the text-as-material metaphor can reveal AI-enabled operations, reshape the writing process, and foster compelling user experiences. A formative study shows that users readily reason with text-as-material, informing a conceptual framework that explains how material metaphors shift mental models and bridge gulfs of envisioning, execution, and evaluation in LLM-mediated writing. We present the design and evaluation of two technical probes: Text as Clay, where users refine text through gestural sculpting, and Text as Plants, where ideas grow serendipitously over time. This work expands the design space of writing tools by treating text as a living, malleable medium.

Despite Generative AI (GenAI) systems' potential for enhancing content creation, users often struggle to effectively integrate GenAI into their creative workflows. Core challenges include misalignment of AI-generated content with user intentions (intent elicitation and alignment), user uncertainty around how to best communicate their intents to the AI system (prompt formulation), and insufficient flexibility of AI systems to support diverse creative workflows (workflow flexibility). Motivated by these challenges, we created IntentTagger: a system for slide creation based on the notion of Intent Tags - small, atomic conceptual units that encapsulate user intent - for exploring granular and non-linear micro-prompting interactions for Human-GenAI co-creation workflows. Our user study with 12 participants provides insights into the value of flexibly expressing intent across varying levels of ambiguity, meta-intent elicitation, and the benefits and challenges of intent tag-driven workflows. We conclude by discussing the broader implications of our findings and design considerations for GenAI-supported content creation workflows.

Chat-based prompts respond with verbose linear-sequential texts, making it difficult to explore and refine ambiguous intents, back up and reinterpret, or shift directions in creative AI-assisted design work. AI-Instruments instead embody "prompts" as interface objects via three key principles: (1) Reification of user-intent as reusable direct-manipulation instruments; (2) Reflection of multiple interpretations of ambiguous user-intents (Reflection-in-intent) as well as the range of AI-model responses (Reflection-in-response) to inform design "moves" towards a desired result; and (3) Grounding to instantiate an instrument from an example, result, or extrapolation directly from another instrument. Further, AI-Instruments leverage LLM's to suggest, vary, and refine new instruments, enabling a system that goes beyond hard-coded functionality by generating its own instrumental controls from content. We demonstrate four technology probes, applied to image generation, and qualitative insights from twelve participants, showing how AI-Instruments address challenges of intent formulation, steering via direct manipulation, and non-linear iterative workflows to reflect and resolve ambiguous intents.

Via Research through Design (RtD), we explore the potential of headphones as a general-purpose input device for both foreground motion-gestures as well as background sensing of user activity. As a familiar wearable device, headphones offer a compelling site for head-situated interaction and sensing. Using emerging sensing modalities such as inertial motion, capacitive touch sensing, and depth cameras, our implemented prototypes explore sensing and interaction techniques that offer a range of compelling capabilities.

We present AdHocProx, a system that uses device-relative, inside-out sensing to augment co-located collaboration across multiple devices, without recourse to externally-anchored beacons – or even reliance on WiFi connectivity.

We present Escapement, a video prototyping tool that introduces a powerful new concept for prototyping screen-based interfaces by flexibly mapping sensor values to dynamic playback control of videos. This recasts the time dimension of video mock-ups as sensor-mediated interaction.

Structured note-taking forms such as sketchnoting, self-tracking journals, and bullet journaling go beyond immediate capture of information scraps. Instead, hand-drawn pride-in-craftmanship increases perceived value for sharing and display. But hand-crafting lists, tables, and calendars is tedious and repetitive. To support these practices digitally, Style Blink (“Style-Blocks+Ink”) explores handcrafted styling as a first-class object. Style-blocks encapsulate digital ink, enabling people to craft, modify, and reuse embellishments and decorations for larger structures, and apply custom layouts. For example, we provide interaction instruments that style ink for personal expression, inking palettes that afford creative experimentation, fillable pens that can be “loaded” with commands and actions to replace menu selections, techniques to customize inked structures post-creation by modifying the underlying handcrafted style-blocks and to re-layout the overall structure to match users’ preferred template. In effect, any ink stroke, notation, or sketch can be encapsulated as a style-object and re-purposed as a tool. Feedback from 13 users show the potential of style adaptation and re-use in individual sketching practices.

To better ground technical (systems) investigation and interaction design of cross-device experiences, we contribute an in-depth survey of existing multi-device practices, including fragmented workflows across devices and the way people physically organize and configure their workspaces to support such activity. Further, this survey documents a historically significant moment of transition to a new future of remote work, an existing trend dramatically accelerated by the abrupt switch to work-from-home (and having to contend with the demands of home-at-work) during the COVID-19 pandemic. We surveyed 97 participants, and collected photographs of home setups and open-ended answers to 50 questions categorized in 5 themes. We characterize the wide range of multi-device physical configurations and identify five usage patterns, including: partitioning tasks, integrating multi-device usage, cloning tasks to other devices, expanding tasks and inputs to multiple devices, and migrating between devices. Our analysis also sheds light on the benefits and challenges people face when their workflow is fragmented across multiple devices. These insights have implications for the design of multi-device experiences that support people’s fragmented workflows.

Sketchnoting is a form of visual note taking where people listen to, synthesize, and visualize ideas from a talk or other event using a combination of pictures, diagrams, and text. Little is known about the design space of this kind of visual note taking. With an eye towards informing the implementation of digital equivalents of sketchnoting, inking, and note taking, we introduce a classification of sketchnote styles and techniques, with a qualitative analysis of 103 sketchnotes, and situated in context with six semi-structured follow up interviews. Our findings distill core sketchnote components (content, layout, structuring elements, and visual styling) and dimensions of the sketchnote design space, classifying levels of conciseness, illustration, structure, personification, cohesion, and craftsmanship. We unpack strategies to address particular note taking challenges, for example dealing with constraints of live drawings, and discuss relevance for future digital inking tools, such as recomposition, styling, and design suggestions.

AirConstellations supports a unique semi-fixed style of cross-device interactions via multiple self-spatially-aware armatures to which users can easily attach (or detach) tablets and other devices. In particular, AirConstellations affords highly flexible and dynamic device formations where the users can bring multiple devices together in-air — with 2–5 armatures poseable in 7DoF within the same workspace — to suit the demands of their current task, social situation, app scenario, or mobility needs. This affords an interaction metaphor where relative orientation, proximity, attaching (or detaching) devices, and continuous movement into and out of ad-hoc ensembles can drive context-sensitive interactions. Yet all devices remain self-stable in useful configurations even when released in mid-air.

Much of the visualization literature focuses on assessment of visual representations with regard to their effectiveness for understanding data. In the present work, we instead focus on making data visualization experiences more enjoyable, to foster deeper engagement with data. We investigate two strategies to make visualization experiences more enjoyable and engaging: personalization, and immersion. We selected pictographs (composed of multiple data glyphs) as this representation affords creative freedom, allowing people to craft symbolic or whimsical shapes of personal significance to represent data. We present the results of a qualitative study with 12 participants crafting pictographs using a large pen-enabled device and while immersed within a VR environment. Our results indicate that personalization and immersion both have positive impact on making visualizations more enjoyable experiences.

While tablet devices are a promising platform for data visualization, supporting consistent interactions across different types of visualizations on tablets remains an open challenge. In this paper, we present multimodal interactions that function consistently across different visualizations, supporting common operations during visual data analysis. By considering standard interface elements (e.g., axes, marks) and grounding our design in a set of core concepts including operations, parameters, targets, and instruments, we systematically develop interactions applicable to different visualization types. To exemplify how the proposed interactions collectively facilitate data exploration, we employ them in a tablet-based system, InChorus that supports pen, touch, and speech input. Based on a study with 12 participants performing replication and factchecking tasks with InChorus, we discuss how participants adapted to using multimodal input and highlight considerations for future multimodal visualization systems.

Knowledge work increasingly spans multiple computing surfaces. Yet in status quo user experiences, content as well as tools, behaviors, and workflows are largely bound to the current device-running the current application, for the current user, and at the current moment in time. SurfaceFleet is a system and toolkit that uses resilient distributed programming techniques to explore cross-device interactions that are unbounded in these four dimensions of device, application, user, and time. As a reference implementation, we describe an interface built using SurfaceFleet that employs lightweight, semi-transparent UI elements known as Applets. Applets appear always-on-top of the operating system, application windows, and (conceptually) above the device itself. But all connections and synchronized data are virtualized and made resilient through the cloud. For example, a sharing Applet known as a Portfolio allows a user to drag and drop unbound Interaction Promises into a document. Such promises can then be fulfilled with content asynchronously, at a later time (or multiple times), from another device, and by the same or a different user.

Drawing boards offer a self-stable work surface that is continuously adjustable. On digital displays, such as the Microsoft Surface Studio, these properties open up a class of techniques that sense and respond to tilt adjustments. Each display posture-whether angled high, low, or somewhere in-between-affords some activities, but not others. Because what is appropriate also depends on the application and task, we explore a range of app-specific transitions between reading vs. writing (annotation), public vs. personal, shared person-space vs. task-space, and other nuances of input and feedback, contingent on display angle. Continuous responses provide interactive transitions tailored to each use-case. We show how a variety of knowledge work scenarios can use sensed display adjustments to drive context-appropriate transitions, as well as technical software details of how to best realize these concepts. A preliminary remote user study suggests that techniques must balance effort required to adjust tilt, versus the potential benefits of a sensed transition.

We present DrawMyPhoto, an interactive system that can assist a drawing novice in producing a quality drawing by automatically parsing a photograph in to a step-by-step drawing tutorial. The system utilizes image processing to produce distinct line work and shading steps from the photograph, and offers novel real-time feedback on pressure and tilt, along with grip suggestions as the user completes the tutorial. Our evaluation showed that the generated steps and real-time assistance allowed novices to produce significantly better drawings than with a more traditional grid-based approach, particularly with respect to accuracy, shading, and details. This was confirmed by domain experts who blindly rated the drawings. The participants responded well to the real-time feedback, and believed it helped them learn proper shading techniques and the order in which a drawing should be approached. We saw promising potential in the tool to boost the confidence of novices and lower the barrier to artistic creation.

Comics are an entertaining and familiar medium for presenting compelling stories about data. However, existing visualization authoring tools do not leverage this expressive medium. In this paper, we seek to incorporate elements of comics into the construction of data-driven stories about dynamic networks. We contribute DataToon, a flexible data comic storyboarding tool that blends analysis and presentation with pen and touch interactions. A storyteller can use DataToon rapidly generate visualization panels, annotate them, and position them within a canvas to produce a visually compelling narrative. In a user study, participants quickly learned to use DataToon for producing data comics.

Prior research identified that physical paper documents have many positive attributes, for example natural tangibility and inherent physical flexibility. When documents are presented on digital devices, however, they can provide unique functionality to users, such as the ability to search, view dynamic multimedia content, and make use of indexing. This work explores the fusion of physical and digital paper documents. It first presents the results of a study that probed how users perform document-intensive analytical tasks when both physical and digital versions of documents were available. The study findings then informed the design of HoloDoc, a mixed reality system that augments physical artifacts with rich interaction and dynamic virtual content. Finally, we present the interaction techniques that HoloDoc affords, and the results of a second study that assessed HoloDoc's utility when working with digital and physical copies of academic articles.

During sensemaking, people annotate insights: underlining sentences in a document or circling regions on a map. They jot down their hypotheses: drawing correlation lines on scatterplots or creating personal legends to track patterns. We present ActiveInk, a system enabling people to seamlessly transition between exploring data and externalizing their thoughts using pen and touch. ActiveInk enables the natural use of pen for active reading behaviors, while supporting analytic actions by activating any of these ink strokes. Through a qualitative study with eight participants, we contribute observations of active reading behaviors during data exploration and design principles to support sensemaking.

Many status-quo interfaces for tablets with pen + touch input capabilities force users to reach for device-centric UI widgets at fixed locations, rather than sensing and adapting to the user-centric posture. To address this problem, we propose sensing techniques that transition between various nuances of mobile and stationary use via postural awareness. These postural nuances include shifting hand grips, varying screen angle and orientation, planting the palm while writing or sketching, and detecting what direction the hands approach from. To achieve this, our system combines three sensing modalities: 1) raw capacitance touchscreen images, 2) inertial motion, and 3) electric field sensors around the screen bezel for grasp and hand proximity detection. We show how these sensors enable posture-aware pen+touch techniques that adapt interaction and morph user interface elements to suit fine-grained contexts of body-, arm-, hand-, and grip-centric frames of reference.

Externalizing one's thoughts can be helpful during data analysis, such as which one marks interesting data, notes hypotheses, and draws diagrams. In this paper, we present two exploratory studies conducted to investigate types and use of externalizations during the analysis process. We first studied how people take notes during different stages of data analysis using VoyagerNote, a visualization recommendation system augmented to support text annotations, and coupled with participants' favorite external note-taking tools (e.g., word processor, pen & paper). Externalizations manifested mostly as notes written on paper or in a word processor, with annotations atop views used almost exclusively in the initial phase of analysis. In the second study, we investigated two specific opportunities: (1) integrating digital pen input to facilitate the use of free-form externalizations and (2) providing a more explicit linking between visualizations and externalizations. We conducted the study with VoyagerInk, a visualization system that enabled free-form externalization with a digital pen as well as touch interactions to link externalizations to data. Participants created more graphical externalizations with VoyagerInk and revisited over half of their externalizations via the linking mechanism. Reflecting on the findings from these two studies, we discuss implications for the design of data analysis tools.

When editing or reviewing a document, people directly overlay ink marks on content. For instance, they underline words, or circle elements in a fgure. These overlay marks often accompany in-context annotations in the form of handwritten footnotes and marginalia. People tend to put annotations close to the content that elicited them, but have to compose with the often-limited whitespace. We introduce SpaceInk, a design space of pen+touch techniques that make room for in-context annotations by dynamically refowing documents. We identify representative techniques in this design space, spanning both new ones and existing ones. We evaluate them in a user study, with results that inform the design of a prototype system. Our system lets users concentrate on capturing feeting thoughts, streamlining the overall annotation process by enabling the fuid inverleaving of space-making gestures with freeform ink.

Prior research has demonstrated that users are increasingly employing multiple devices during daily work. Currently, devices such as keyboards, cell phones, and tablets remain largely unaware of their role within a user's workflow. As a result, transitioning between devices is tedious, often to the degree that users are discouraged from taking full advantage of the devices they have within reach. This work explores the device ecologies used in desk-centric environments and complies the insights observed into SMAC, a simplified model of attention and capture that emphasizes the role of user-device proxemics, as mediated by hand placement, gaze, and relative body orientation, as well as inter-device proxemics. SMAC illustrates the potential of harnessing the rich, proxemic diversity that exists between users and their device ecologies, while also helping to organize and synthesize the growing body of literature on distributed user interfaces. An evaluation study using SMAC demonstrated that users could easily understand the tenants of user- and inter-device proxemics and found them to be valuable within their workflows.

editorial Free Access Share on The Editor's Spotlight: TOCHI Issue 24:6 Author: Ken Hinckley Redmond, Washington Redmond, WashingtonView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 24Issue 6December 2017 Article No.: 37pp 1–4https://doi.org/10.1145/3173378Published:22 January 2018Publication History 0citation291DownloadsMetricsTotal Citations0Total Downloads291Last 12 Months27Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

editorial Free AccessThe Editor's Spotlight: TOCHI Issue 25:2 Share on Editor: Ken Hinckley Redmond, Washington Redmond, WashingtonView Profile Authors Info & Affiliations ACM Transactions on Computer-Human InteractionVolume 25Issue 2April 2018 Article No.: 7pp 1–3https://doi.org/10.1145/3196696Published:11 April 2018 1citation220DownloadsMetricsTotal Citations1Total Downloads220Last 12 Months27Last 6 weeks3 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

editorial Free Access Share on The Editor's Spotlight: TOCHI Issue 25:3 Editor: Ken Hinckley Redmond, Washington Redmond, WashingtonView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 25Issue 3June 2018 Article No.: 14pp 1–3https://doi.org/10.1145/3214352Published:08 June 2018Publication History 0citation318DownloadsMetricsTotal Citations0Total Downloads318Last 12 Months25Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteView all FormatsPDF

Modern tablets support simultaneous pen and touch input, but it remains unclear how to best leverage this capability for bimanual input when the nonpreferred hand holds the tablet. We explore Thumb + Pen interactions that support simultaneous pen and touch interaction, with both hands, in such situations. Our approach engages the thumb of the device-holding hand, such that the thumb interacts with the touch screen in an indirect manner, thereby complementing the direct input provided by the preferred hand. For instance, the thumb can determine how pen actions (articulated with the opposite hand) are interpreted. Alternatively, the pen can point at an object, while the thumb manipulates one or more of its parameters through indirect touch. Our techniques integrate concepts in a novel way that derive from marking menus, spring-loaded modes, indirect input, and multi-touch conventions. Our overall approach takes the form of a set of probes, each representing a meaningfully distinct class of application. They serve as an initial exploration of the design space at a level which will help determine the feasibility of supporting bimanual interaction in such contexts, and the viability of the Thumb + Pen techniques in so doing.

This paper sheds light on gaps and discrepancies between the experiences afforded by analog pens and their digital counterparts. Despite the long history (and recent renaissance) of digital pens, the literature still lacks a comprehensive survey of what types of marks people make and what motivates them to use ink-both analog and digital in daily life. To capture the diversity of inking behaviors and tease out the unique affordances of pen-and ink, we conducted a diary study with 26 participants from diverse backgrounds. From analysis of 493 diary entries we identified 8 analog pen-and-ink activities, and 9 affordances of pens. We contextualized and contrasted these findings using a survey with 1,633 respondents and a follow-up diary study with 30 participants, observing digital pens. Our analysis reveals gaps and research opportunities based on pen affordances not yet fully explored in the literature.

WritLarge is a freeform canvas for early-stage design on electronic whiteboards with pen+touch input. The system aims to support a higher-level flow of interaction by 'chunking' the traditionally disjoint steps of selection and action into unified selection-action phrases. This holistic goal led us to address two complementary aspects: SELECTION, for which we devise a new technique known as the Zoom-Catcher that integrates pinch-to-zoom and selection in a single gesture for fluidly selecting and acting on content; plus: ACTION, where we demonstrate how this addresses the combined issues of navigating, selecting, and manipulating content. In particular, the designer can transform select ink strokes in flexible and easily-reversible representations via semantic, structural, and temporal axes of movement that are defined as conceptual 'moves' relative to the specified content.

Prognostications.Well, 2016 has been consigned to the dustbin of history and 2017 is here whether you wanted it or not

Extravaganza-Special Issue on End-User Design for the Internet of

editorial Free Access Share on The Editor's Spotlight: TOCHI Issue 24:3 Editor: Ken Hinckley Redmond, Washington Redmond, WashingtonView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 24Issue 3June 2017 Article No.: 18epp 1–5https://doi.org/10.1145/3095801Published:13 July 2017Publication History 0citation252DownloadsMetricsTotal Citations0Total Downloads252Last 12 Months16Last 6 weeks5 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

editorial Free Access Share on The Editor's Spotlight: TOCHI Issue 24:5 Editor: Ken Hinckley Redmond, Washington Redmond, WashingtonView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 24Issue 5October 2017 Article No.: 31pp 1–4https://doi.org/10.1145/3145471Published:13 November 2017Publication History 0citation156DownloadsMetricsTotal Citations0Total Downloads156Last 12 Months11Last 6 weeks1 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

Touchscreens continue to advance including progress towards sensing fingers proximal to the display. We explore this emerging pre-touch modality via a self-capacitance touchscreen that can sense multiple fingers above a mobile device, as well as grip around the screen's edges. This capability opens up many possibilities for mobile interaction. For example, using pre-touch in an anticipatory role affords an "ad-lib interface" that fades in a different UI--appropriate to the context--as the user approaches one-handed with a thumb, two-handed with an index finger, or even with a pinch or two thumbs. Or we can interpret pre-touch in a retroactive manner that leverages the approach trajectory to discern whether the user made contact with a ballistic vs. a finely-targeted motion. Pre-touch also enables hybrid touch + hover gestures, such as selecting an icon with the thumb while bringing a second finger into range to invoke a context menu at a convenient location. Collectively these techniques illustrate how pre-touch sensing offers an intriguing new back-channel for mobile interaction.

We explore the contextual details afforded by wearable devices to support multi-user, direct-touch interaction on electronic whiteboards in a way that-unlike previous work-can be fully consistent with natural bimanual-asymmetric interaction as set forth by Guiard.

short-paper Free Access Share on Editorial: Welcome to a New Era for TOCHI Author: Ken Hinckley View Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 23Issue 1February 2016 Article No.: 1epp 1–6https://doi.org/10.1145/2882897Published:20 February 2016Publication History 0citation602DownloadsMetricsTotal Citations0Total Downloads602Last 12 Months79Last 6 weeks10 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

short-paper Free Access Share on The Editor's Spotlight: TOCHI Issue 23:1 Editor: Ken Hinckley Redmond, Washington Redmond, WashingtonView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 23Issue 1February 2016 Article No.: 1pp 1–4https://doi.org/10.1145/2882899Published:20 February 2016Publication History 0citation285DownloadsMetricsTotal Citations0Total Downloads285Last 12 Months31Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

editorial Free Access Share on The Editor's Spotlight: TOCHI Issue 23:2 Author: K. Hinckley Snoqualmie Pass, Washington Snoqualmie Pass, WashingtonView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 23Issue 2May 2016 Article No.: 7pp 1–5https://doi.org/10.1145/2904385Published:11 May 2016Publication History 0citation360DownloadsMetricsTotal Citations0Total Downloads360Last 12 Months29Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

editorial Free Access Share on The Editor's Spotlight: TOCHI Issue 23:3 Editor: Ken Hinckley Redmond, Washington Redmond, WashingtonView Profile Authors Info & Claims ACM Transactions on Computer-Human InteractionVolume 23Issue 3July 2016 Article No.: 13pp 1–5https://doi.org/10.1145/2943789Published:21 July 2016Publication History 0citation147DownloadsMetricsTotal Citations0Total Downloads147Last 12 Months15Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteeReaderPDF

My goodness-it's the last week of July already.How did this happen?Has some bizarre turnabout of time and space, presumably a relativistic distortion brought on by the extreme gravitational field of my ever-mounting TO DO pile, vaulted me far into an incomprehensible future?No matter how vigorously I attack this great mountain of editorial work with my trusty red pen, following the trail blazed by my yellow highlighter and its volatile scent, the tasks heaped thereupon exhibit a rather strange property:Each and every one instantaneously expands to fill (if not exceed) all time that I can manage to pry loose for it. . .not unlike this editorial, on which I am rather tardy at this point.Yet, one thing is certain: as usual, the summit of TOCHI affords a perspective from which one can appreciate a rather stunning view.And the cool thing is, this mountain's always getting higher; there is no top, there is no culmination to scientific curiosity and the insights afforded by the CHI community's brilliant analyses and designs.This month's new contributions, in that sense, offer a magnificent glimpse of what's still to come as we climb ever higher.Somehow, almost magically, more and more of this wonderful mountain always appears underfoot each time one suspects one is getting anywhere near the top.But trust me, I do not mind the failure of this would-be summit bid one bit.Rather than a Sisyphean task, it is a journey of incredible wonder.TOCHI itself continues to reach greater heights-and in recent events, as an institution it has grown far wiser as well, because I am happy to report that four illustrious new members have graciously accepted my invitation to serve on the Editorial Board:-Pourang P.

So I hear there has been an election recently.And not unlike the #Brexit vote that recently sundered the United Kingdom, I have to say that for most of us in the States it has been something of a shocker, no matter one's political affiliation.I had my say about Trump in Interzone #265-a magazine published in the UK, ironically enough-so I have no desire to pour any further gasoline onto that particular bonfire in these pages.Hopefully, concerns that have been voiced about what his administration may mean for unbiased scientific research ultimately prove to be unfounded.We shall see.The world marches on.As I parked and walked into my office building after picking up my battered old Volvo from the auto shop, the hiss of tires overdamp asphalt and the drone of aircraft across the battleship-gray cloud deck echoed an eerie sense of normality down from the sky.One can only hope that it signals business as usual, rather than a portent.May we live in interesting times, indeed.Scientific progress marches on as well.It is hard to believe that another full year of TOCHI, Volume 23, is now in the books.We have seen over 200 new submissions this year, as of mid-November-already more than all of 2015-and a tally that does not include the many, many revisions submitted as well.Issue 6 brings five new contributions to the annals of the journal, making 33 articles published in total this year.Each of these, of course, is now eligible for the 2016 TOCHI Best Paper Award, and we will soon start sorting through all the strong manuscripts to single out one for the award.And of course, do not forget to nominate your favorite TOCHI article.Just send us a note at tochi@acm.

The orientation and repositioning of physical artefacts (such as paper documents) to afford shared viewing of content, or to steer the attention of others to specific details, is known as micro-mobility. But the role of grasp in micro-mobility has rarely been considered, much less sensed by devices. We therefore employ capacitive grip sensing and inertial motion to explore the design space of combined grasp + micro-mobility by considering three classes of technique in the context of active reading. Single user, single device techniques support grip-influenced behaviors such as bookmarking a page with a finger, but combine this with physical embodiment to allow flipping back to a previous location. Multiple user, single device techniques, such as passing a tablet to another user or working side-by-side on a single device, add fresh nuances of expression to co-located collaboration. And single user, multiple device techniques afford facile cross-referencing of content across devices. Founded on observations of grasp and micro-mobility, these techniques open up new possibilities for both individual and collaborative interaction with electronic documents.

We explore grip and motion sensing to afford new techniques that leverage how users naturally manipulate tablet and stylus devices during pen + touch interaction. We can detect whether the user holds the pen in a writing grip or tucked between his fingers. We can distinguish bare-handed inputs, such as drag and pinch gestures produced by the nonpreferred hand, from touch gestures produced by the hand holding the pen, which necessarily impart a detectable motion signal to the stylus. We can sense which hand grips the tablet, and determine the screen's relative orientation to the pen. By selectively combining these signals and using them to complement one another, we can tailor interaction to the context, such as by ignoring unintentional touch inputs while writing, or supporting contextually-appropriate tools such as a magnifier for detailed stroke work that appears when the user pinches with the pen tucked between his fingers. These and other techniques can be used to impart new, previously unanticipated subtleties to pen + touch interaction on tablets.

We present LightRing, a wearable sensor in a ring form factor that senses the 2d location of a fingertip on any surface, independent of orientation or material. The device consists of an infrared proximity sensor for measuring finger flexion and a 1-axis gyroscope for measuring finger rotation. Notably, LightRing tracks subtle fingertip movements from the finger base without requiring instrumentation of other body parts or the environment. This keeps the normal hand function intact and allows for a socially acceptable appearance. We evaluate LightRing in a 2d pointing experiment in two scenarios: on a desk while sitting down, and on the leg while standing. Our results indicate that the device has potential to enable a variety of rich mobile input scenarios.

This paper presents a new tap and gesture hybrid method for authenticating mobile device users. The new technique augments four simple gestures - up, down, left, and right, to the dominant digit lock technique, allowing users to either tap or perform any one of the four gestures on the digit keys. It offers in total 6250000 unique four-symbol password combinations, which is substantially more than the conventional techniques. Results of a pilot study showed that the new technique was slower and more error prone than the digit lock technique. However, we believe with practice it could get faster and more accurate. Also, most users were comfortable and all of them felt more secured while using the new technique.

We present a method for composing handwritten messages on a small touchscreen device. A word is entered by drawing overlapped, screen sized letters on top of each other. The system does not require gestures or timeouts to delimit characters within a word - it automatically segments the overlapping strokes and renders the message in real-time as the user is writing. The auto-segmentation algorithm was designed for practicality; it is extremely simple, requires only public domain data for training, and runs very fast on low-power devices. Drawings may also be included with the text. Experimental data indicates the effectiveness of our system, even for novice users.

We contrast the Chameleon Lens, which uses 3D movement of a mobile device held in the nonpreferred hand to support panning and zooming, with the Pinch-Flick-Drag metaphor of directly manipulating the view using multi-touch gestures. Lens-like approaches have significant potential because they can support navigation-selection, navigation-annotation, and other such compound tasks by off-loading navigation to the nonpreferred hand while the preferred hand annotates, marks a location, or draws a path on the screen. Our experimental results show that the Chameleon Lens is significantly slower than Pinch-Flick-Drag for the navigation subtask in isolation. But our studies also reveal that for navigation between a few known targets the lens performs significantly faster, that differences between the Chameleon Lens and Pinch-Flick-Drag rapidly diminish as users gain experience, and that in the context of a compound navigation-annotation task, the lens performs as well as Pinch-Flick-Drag despite its deficit for the navigation subtask itself.

GatherReader is a prototype e-reader with both pen and multi-touch input that illustrates several interesting design trade-offs to fluidly interleave content consumption behaviors (reading and flipping through pages) with information gathering and informal organization activities geared to active reading tasks. These choices include (1) relaxed precision for casual specification of scope; (2) multiple object collection via a visual clipboard; (3) flexible workflow via deferred action; and (4) complementary use of pen+touch. Our design affords active reading by limiting the transaction costs for secondary subtasks, while keeping users in the flow of the primary task of reading itself.

GroupTogether is a system that explores cross-device interaction using two sociological constructs. First, F-formations concern the distance and relative body orientation among multiple users, which indicate when and how people position themselves as a group. Second, micro-mobility describes how people orient and tilt devices towards one another to promote fine-grained sharing during co-present collaboration. We sense these constructs using: (a) a pair of overhead Kinect depth cameras to sense small groups of people, (b) low-power 8GHz band radio modules to establish the identity, presence, and coarse-grained relative locations of devices, and (c) accelerometers to detect tilting of slate devices. The resulting system supports fluid, minimally disruptive techniques for co-located collaboration by leveraging the proxemics of people as well as the proxemics of devices.

Direct-touch interaction on mobile phones revolves around screens that compete for visual attention with users' real-world tasks and activities. This paper investigates the impact of these situational impairments on touch-screen interaction. We probe several design factors for touch-screen gestures, under various levels of environmental demands on attention, in comparison to the status-quo approach of soft buttons. We find that in the presence of environmental distractions, gestures can offer significant performance gains and reduced attentional load, while performing as well as soft buttons when the user's attention is focused on the phone. In fact, the speed and accuracy of bezel gestures did not appear to be significantly affected by environment, and some gestures could be articulated eyes-free, with one hand. Bezel-initiated gestures offered the fastest performance, and mark-based gestures were the most accurate. Bezel-initiated marks therefore may offer a promising approach for mobile touch-screen interaction that is less demanding of the user's attention.

We explore techniques for hand-held devices that leverage the multimodal combination of touch and motion. Hybrid touch + motion gestures exhibit interaction properties that combine the strengths of multi-touch with those of motion-sensing. This affords touch-enhanced motion gestures, such as one-handed zooming by holding one's thumb on the screen while tilting a device. We also consider the reverse perspective, that of motion-enhanced touch, which uses motion sensors to probe what happens underneath the surface of touch. Touching the screen induces secondary accelerations and angular velocities in the sensors. For example, our prototype uses motion sensors to distinguish gently swiping a finger on the screen from 'Sdrags with a hard onset' - to enable more expressive touch interactions.

This paper explores the interaction possibilities enabled when the barrel of a digital pen is augmented with a multi-touch sensor. We present a novel multi-touch pen (MTPen) prototype and discuss its alternate uses beyond those of a standard stylus, such as allowing new touch gestures to be performed using the index finger or thumb and detecting how users grip the device as a mechanism for mode switching. We also discuss the hardware and software implementation challenges in realizing our prototype, and showcase how one can combine different grips (tripod, relaxed tripod, sketch, wrap) and gestures (swipe and double tap) to enable new interaction techniques with the MTPen in a prototype drawing application. One specific aim is the elimination of some of the comfort problems associated with existing auxiliary controls on digital pens. Mechanical controls such as barrel buttons and barrel scroll wheels work best in only a few specific hand grips and pen rotations. Comparatively, our gestures can be successfully and comfortably performed regardless of the rotation of the pen or how the user grips it, offering greater flexibility in use. We describe a formal evaluation comparing MTPen gestures against the use of a barrel button for mode switching. This study shows that both swipe and double tap gestures are comparable in performance to commonly employed barrel buttons without its disadvantages.

We describe techniques for direct pen+touch input. We observe people's manual behaviors with physical paper and notebooks. These serve as the foundation for a prototype Microsoft Surface application, centered on note-taking and scrapbooking of materials. Based on our explorations we advocate a division of labor between pen and touch: the pen writes, touch manipulates, and the combination of pen + touch yields new tools. This articulates how our system interprets unimodal pen, unimodal touch, and multimodal pen+touch inputs, respectively. For example, the user can hold a photo and drag off with the pen to create and place a copy; hold a photo and cross it in a freeform path with the pen to slice it in two; or hold selected photos and tap one with the pen to staple them all together. Touch thus unifies object selection with mode switching of the pen, while the muscular tension of holding touch serves as the "glue" that phrases together all the inputs into a unitary multimodal gesture. This helps the UI designer to avoid encumbrances such as physical buttons, persistent modes, or widgets that detract from the user's focus on the workspace.

The Codex is a dual-screen tablet computer, about the size of a 4"x 6 day planner, with a self-supporting binding and embedded sensors. The device can be oriented in a variety of postures to support different nuances of individual work, ambient display, or collaboration with another user. In the context of a pen-operated note taking application, we demonstrate interaction techniques that support a fluid division of labor for tasks and information across the two displays while minimizing disruption to the primary experience of authoring notes.

Synchronous gestures are patterns of sensed user or users' activity, spanning a distributed system that take on a new meaning when they occur together in time. Synchronous gestures draw inspiration from real-world social rituals such as toasting by tapping two drinking glasses together. In this article, we explore several interactions based on synchronous gestures, including bumping devices together, drawing corresponding pen gestures on touch-sensitive displays, simultaneously pressing a button on multiple smart-phones, or placing one or more devices on the sensing surface of a tabletop computer. These interactions focus on wireless composition of physically colocated devices, where users perceive one another and coordinate their actions through social protocol. We demonstrate how synchronous gestures may be phrased together with surrounding interactions. Such connection-action phrases afford a rich syntax of cross-device commands, operands, and one-to-one or one-to-many associations with a flexible physical arrangement of devices. Synchronous gestures enable colocated users to combine multiple devices into a heterogeneous display environment, where the users may establish a transient network connection with other select colocated users to facilitate the pooling of input capabilities, display resources, and the digital contents of each device. For example, participants at a meeting may bring mobile devices including tablet computers, PDAs, and smart-phones, and the meeting room infrastructure may include fixed interactive displays, such as a tabletop computer. Our techniques facilitate creation of an ad hoc display environment for tasks such as viewing a large document across multiple devices, presenting information to another user, or offering files to others. The interactions necessary to establish such ad hoc display environments must be rapid and minimally demanding of attention: during face-to-face communication, a pause of even 5 sec is socially awkward and disrupts collaboration. Current devices may associate using a direct transport such as Infrared Data Association ports, or the emerging Near Field Communication standard. However, such transports can only support one-to-one associations between devices and require close physical proximity as well as a specific relative orientation to connect the devices (e.g., the devices may be linked when touching head-to-head but not side-to-side). By contrast, sociology research in proxemics (the study of how people use the “personal space” surrounding their bodies) demonstrates that people carefully select physical distance as well as relative body orientation to suit the task, mood, and social relationship with other persons. Wireless networking can free device-to-device connections from the limitations of direct transports but results in a potentially large number of candidate devices. Synchronous gestures address these problems by allowing users to express naturally a spontaneous wireless connection between specific proximal (collocated) interactive displays.

Acquiring small targets on a tablet or touch screen can be challenging. To address the problem, researchers have proposed techniques that enlarge the effective size of targets by extending targets into adjacent screen space. When applied to targets organized in clusters, however, these techniques show little effect because there is no space to grow into. Unfortunately, target clusters are common in many popular applications. We present Starburst, a space partitioning algorithm that works for target clusters. Starburst identifies areas of available screen space, grows a line from each target into the available space, and then expands that line into a clickable surface. We present the basic algorithm and extensions. We then present 2 user studies in which Starburst led to a reduction in error rate by factors of 9 and 3 compared to traditional target expansion.

Many mobile phones integrate services such as personal calendars. Given the social nature of the stored data, however, users often need to access such information as part of a phone conversation. In typical non-headset use, this re-quires users to interrupt their conversations to look at the screen.

Current pen input mainly utilizes the position of the pen tip, and occasionally, a button press. Other possible device parameters, such as rolling the pen around its longitudinal axis, are rarely used. We explore pen rolling as a supporting input modality for pen-based interaction. Through two studies, we are able to determine 1) the parameters that separate intentional pen rolling for the purpose of interaction from incidental pen rolling caused by regular writing and drawing, and 2) the parameter range within which accurate and timely intentional pen rolling interactions can occur. Building on our experimental results, we present an exploration of the design space of rolling-based interaction techniques, which showcase three scenarios where pen rolling interactions can be useful: enhanced stimulus-response compatibility in rotation tasks [7], multi-parameter input, and simplified mode selection.

Paper persists as an integral component of active reading and other knowledge-worker tasks because it provides ease of use unmatched by digital alternatives. Paper documents are light to carry, easy to annotate, rapid to navigate, flexible to manipulate, and robust to use in varied environments. Interactions with paper documents create rich webs of annotation, cross reference, and spatial organization. Unfortunately, the resulting webs are confined to the physical world of paper and, as they accumulate, become increasingly difficult to store, search, and access. XLibris [Schilit et al. 1998] and similar systems address these difficulties by simulating paper with tablet PCs. While this approach is promising, it suffers not only from limitations of current tablet computers (e.g., limited screen space) but also from loss of invaluable paper affordances. In this article, we describe PapierCraft, a gesture-based command system that allows users to manipulate digital documents using paper printouts as proxies. Using an Anoto [Anoto 2002] digital pen, users can draw command gestures on paper to tag a paragraph, e-mail a selected area, copy selections to a notepad, or create links to related documents. Upon pen synchronization, PapierCraft executes the commands and presents the results in a digital document viewer. Users can then search the tagged information and navigate the web of annotated digital documents resulting from interactions with the paper proxies. PapierCraft also supports real time interactions across mix-media, for example, letting users copy information from paper to a Tablet PC screen. This article presents the design and implementation of the PapierCraft system and describes user feedback from initial use.

We present ExperiScope, an analytical tool to help designers and experimenters explore the results of quantitative evaluations of interaction techniques. ExperiScope combines a new visualization incorporating aspects of the KLM and the three-state model with an interface helping users to rapidly cluster similar patterns of interactions. The tool makes it easy to identify and compare key patterns of use encountered during data collection. This promotes a deeper understanding of the results of a given evaluation.We illustrate the advantages of this tool by revisiting the data collected for an experiment conducted by Hinckley et al. [19] which compared different mode switching techniques. Our results show that our tool complements the previously reported results by offering insights about error behavior and the impact of mode switching on user performance.By providing a more fine-grained analysis of the data gathered during empirical evaluations, we hope that our tool will improve researchers' understanding of existing and newly developed interaction techniques.

Using a notebook to sketch designs, reflect on a topic, or capture and extend creative ideas are examples of active note taking tasks. Optimal experience for such tasks demands concentration without interruption. Yet active note taking may also require reference documents or emails from team members. InkSeine is a Tablet PC application that supports active note taking by coupling a pen-and-ink interface with an in situ search facility that flows directly from a user's ink notes (Fig. 1). InkSeine integrates four key concepts: it leverages preexisting ink to initiate a search; it provides tight coupling of search queries with application content; it persists search queries as first class objects that can be commingled with ink notes; and it enables a quick and flexible workflow where the user may freely interleave inking, searching, and gathering content. InkSeine offers these capabilities in an interface that is tailored to the unique demands of pen input, and that maintains the primacy of inking above all other tasks.

Accessing and manipulating occluded content in layered 2D drawings can be difficult. This paper characterizes a design space of techniques that facilitate access to occluded content. In addition, we introduce two new tools, Tumbler and Splatter, which represent unexplored areas of the design space. Finally, we present results of a study that contrasts these two tools against the traditional scene index used in most drawing applications. Results show that Splatter is comparable to and can be better than the scene index. Our findings allow us to understand the inherent design tradeoffs, and to identify areas for further improvement.

We present Hover Widgets, a new technique for increasing the capabilities of pen-based interfaces. Hover Widgets are implemented by using the pen movements above the display surface, in the tracking state. Short gestures while hovering, followed by a pen down, access the Hover Widgets, which can be used to activate localized interface widgets. By using the tracking state movements, Hover Widgets create a new command layer which is clearly distinct from the input layer of a pen interface. In a formal experiment Hover Widgets were found to be faster than a more traditional command activation technique, and also reduced errors due to divided attention.

Modes allow a few inputs to invoke many operations, yet if a user misclassifies or forgets the state of a system, modes can result in errors. Spring-loaded modes (quasimodes) maintain a mode while the user holds a control such as a button or key. The Springboard is an interaction technique for tablet computers that extends quasimodes to encompass multiple tool modes in a single spring-loaded control. The Springboard allows the user to continue holding down a nonpreferred-hand command button after selecting a tool from a menu as a way to repeatedly apply the same tool. We find the Springboard improves performance for both a local marking menu and for a non-local marking menu ("lagoon") at the lower left corner of the screen. Despite the round-trip costs incurred to move the pen to a tool lagoon, a keystroke-level analysis of the true cost of each technique reveals the local marking menu is not significantly faster.

We present Zone and Polygon menus, two new variants of multi-stroke marking menus that consider both the relative position and orientation of strokes. Our menus are designed to increase menu breadth over the 8 item limit of status quo orientation-based marking menus. An experiment shows that Zone and Polygon menus can successfully increase breadth by a factor of 2 or more over orientation-based marking menus, while maintaining high selection speed and accuracy. We also discuss hybrid techniques that may further increase menu breadth and performance. Our techniques offer UI designers new options for balancing menu breadth and depth against selection speed and accuracy.

Sometimes users fail to notice a change that just took place on their display. For example, the user may have accidentally deleted an icon or a remote collaborator may have changed settings in a control panel. Animated transitions can help, but they force users to wait for the animation to complete. This can be cumbersome, especially in situations where users did not need an explanation. We propose a different approach. Phosphor objects show the outcome of their transition instantly; at the same time they explain their change in retrospect. Manipulating a phosphor slider, for example, leaves an afterglow that illustrates how the knob moved. The parallelism of instant outcome and explanation supports both types of users. Users who already understood the transition can continue interacting without delay, while those who are inexperienced or may have been distracted can take time to view the effects at their own pace. We present a framework of transition designs for widgets, icons, and objects in drawing programs. We evaluate phosphor objects in two user studies and report significant performance benefits for phosphor objects.

Snapping is a widely used technique that helps users position graphical objects precisely, e.g., to align them with a grid or other graphical objects. Unfortunately, whenever users want to position a dragged object close to such an aligned location, they first need to deactivate snapping. We propose snap-and-go, a snapping technique that overcomes this limitation. By merely stopping dragged objects at aligned positions, rather than "warping" them there, snap-and-go helps users align objects, yet still allows placing dragged objects anywhere else. While this approach of inserting additional motor space renders snap-and-go slightly slower than traditional snapping, snap-and-go simplifies the user interface by eliminating the need for a deactivation option and thereby allows introducing snapping to application scenarios where traditional snapping is inapplicable. In our user studies, participants were able to align objects up to 138% (1D) and 231% (2D) faster with snap-and-go than without and snap-and-go proved robust against the presence of distracting snap targets.

We present a quantitative analysis of delimiters for pen gestures. A delimiter is "something different" in the input stream that a computer can use to determine the structure of input phrases. We study four techniques for delimiting a selection-action gesture phrase consisting of lasso selection plus marking-menu-based command activation. Pigtail is a new technique that uses a small loop to delimit lasso selection from marking (Fig. 1). Handle adds a box to the end of the lasso, from which the user makes a second stroke for marking. Timeout uses dwelling with the pen to delimit the lasso from the mark. Button uses a button press to signal when to delimit the gesture. We describe the role of delimiters in our Scriboli pen interaction testbed, and show how Pigtail supports scope selection, command activation, and direct manipulation all in a single fluid pen gesture.

Inking and gesturing are two central tasks in pen-based user interfaces. Switching between modes for entry of uninterpreted ink and entry of gestures is required by many pen-based user interfaces. Without an appropriate mode switching technique, pen-based interactions in such situations may be inefficient and cumbersome. In this paper, we investigate five techniques for switching between ink and gesture modes in pen interfaces, including a pen-pressure based mode switching technique that allows implicit mode transition. A quantitative experimental study was conducted to evaluate the performance of these techniques. The results suggest that pressing a button with the non-preferred hand offers the fastest performance, while the technique of holding the pen still is significantly slower and more prone to error than the other techniques. Pressure, while promising, did not perform as well as the non-preferred hand button with our current implementation.

Knowledge workers use paper extensively for document reviewing and note-taking due to its versatility and simplicity of use. As users annotate printed documents and gather notes, they create a rich web of annotations and cross references. Unfortunately, as paper is a static media, this web often gets trapped in the physical world. While several digital solutions such as XLibris [15] and Digital Desk [18] have been proposed, they suffer from a small display size or onerous hardware requirements.To address these limitations, we propose PapierCraft, a gesture-based interface that allows users to manipulate digital documents directly using their printouts as proxies. Using a digital pen, users can annotate a printout or draw command gestures to indicate operations such as copying a document area, pasting an area previously copied, or creating a link. Upon pen synchronization, our infrastructure executes these commands and presents the result in a customized viewer. In this paper we describe the design and implementation of the PapierCraft command system, and report on early user feedback.

Building on Buxton's foreground/background model, we discuss the importance of explicitly considering both foreground interaction and background interaction, as well as transitions between foreground and background, in the design and implementation of sensing techniques for sensor-enhanced mobile devices. Our view is that the foreground concerns deliberate user activity where the user is attending to the device, while the background is the realm of inattention or split attention, using naturally occurring user activity as an input that allows the device to infer or anticipate user needs. The five questions for sensing systems of Bellotti et al. [2002] proposed as a framework for this special issue, primarily address the foreground, but neglect critical issues with background sensing. To support our perspective, we discuss a variety of foreground and background sensing techniques that we have implemented for sensor-enhanced mobile devices, such as powering on the device when the user picks it up, sensing when the user is holding the device to his ear, automatically switching between portrait and landscape display orientations depending on how the user is holding the device, and scrolling the display using tilt. We also contribute system architecture issues, such as using the foreground/background model to handle cross-talk between multiple sensor-based interaction techniques, and theoretical perspectives, such as a classification of recognition errors based on explicitly considering transitions between the foreground and background. Based on our experiences, we propose design issues and lessons learned for foreground/background sensing systems.

Stitching is a new interaction technique that allows users to combine pen-operated mobile devices with wireless networking by using pen gestures that span multiple displays. To stitch, a user starts moving the pen on one screen, crosses over the bezel, and finishes the stroke on the screen of a nearby device. Properties of each portion of the pen stroke are observed by the participating devices, synchronized via wireless network communication, and recognized as a unitary act performed by one user, thus binding together the devices. We identify the general requirements of stitching and describe a prototype photo sharing application that uses stitching to allow users to copy images from one tablet to another that is nearby, expand an image across multiple screens, establish a persistent shared workspace, or use one tablet to present images that a user selects from another tablet. We also discuss design issues that arise from proxemics, that is, the sociological implications of users collaborating in close quarters.

This paper describes techniques that allow users to collaborate on tablet computers that employ distributed sensing techniques to establish a privileged connection between devices. Each tablet is augmented with a two-axis linear accelerometer (tilt sensor), touch sensor, proximity sensor, and light sensor. The system recognizes when users bump two tablets together by looking for spikes in each tablet's accelerometer data that are synchronized in time; bumping establishes a privileged connection between the devices. Users can face one another and bump the tops of two tablets together to establish a collaborative face-to-face workspace. The system then uses the sensors to enhance transitions between personal work and shared work. For example, a user can hold his or her hand near the top of the workspace to "shield" the display from the other user. This gesture is sensed using the proximity sensor together with the light sensor, allowing for quick "asides" into private information or to sketch an idea in a personal workspace. Picking up, putting down, or walking away with a tablet are also sensed, as is angling the tablet towards the other user. Much research in single display groupware considers shared displays and shared artifacts, but our system explores a unique form of dual display groupware for face-to-face communication and collaboration using personal display devices.

This research explores distributed sensing techniques for mobile devices using synchronous gestures. These are patterns of activity, contributed by multiple users (or one user with multiple devices), which take on a new meaning when they occur together in time, or in a specific sequence in time. To explore this new area of inquiry, this work uses tablet computers augmented with touch sensors and two-axis linear accelerometers (tilt sensors). The devices are connected via an 802.11 wireless network and synchronize their time-stamped sensor data. This paper describes a few practical examples of interaction techniques using synchronous gestures such as dynamically tiling together displays by physically bumping them together, discusses implementation issues, and speculates on further possibilities for synchronous gestures.

We propose a formal experimental paradigm designed to help evaluate scrolling interaction techniques. Such a method is needed by interaction designers to quantify scrolling performance, thereby providing a tool to evaluate and improve upon new techniques. We systematically vary the scrolling distance as well as the required tolerance of scrolling. Distance and tolerance are the parameters of Fitts' Law, which traditionally has been applied to the evaluation of pointing devices in tasks involving rapid, aimed movement to visible targets. Scrolling involves acquisition of targets well beyond the edges of the screen, yet Fitts' Law models our experimental data very wellWe apply our paradigm to the IBM ScrollPoint and the IntelliMouse Wheel. Our experimental approach reveals a crossover effect in performance versus distance, with the Wheel performing best at short distances but the ScrollPoint performing best at long distances. We also demonstrate that the performance of the Wheel can be significantly improved using an acceleration algorithm. These results show that our approach yields a practical and rigorous method for the evaluation of scrolling techniques.

Although cell phones are extremely useful, they can be annoying and distracting to owners and others nearby. We describe sensing techniques intended to help make mobile phones more polite and less distracting. For example, our phone's ringing quiets as soon as the user responds to an incoming call, and the ring mutes if the user glances at the caller ID and decides not to answer. We also eliminate the need to press a TALK button to answer an incoming call by recognizing if the user picks up the phone and listens to it.

We present experimental work that explores the factors governing symmetric bimanual interaction in a two-handed task that requires the user to track a pair of targets, one target with each hand. A symmetric bimanual task is a two-handed task in which each hand is assigned an identical role. In this context, we explore three main experimental factors. We vary the distance between the pair of targets to track: as the targets become further apart, visual diversion increases, forcing the user to divide attention between the two targets. We also vary the demands of the task by using both a slow and a fast tracking speed. Finally, we explore visual integration of sub-tasks: in one condition, the two targets to track are connected by a line segment which visually links the targets, while in the other condition there is no connecting line. Our results indicate that all three experimental factors affect the degree of parallelism, which we quantify using a new metric of bimanual parallelism. However, differences in tracking error between the two hands are affected only by the visual integration factor.

The Task Gallery is a window manager that uses interactive 3D graphics to provide direct support for task management and document comparison, lacking from many systems implementing the desktop metaphor. User tasks appear as artwork hung on the walls of a virtual art gallery, with the selected task on a stage. Multiple documents can be selected and displayed side-by-side using 3D space to provide uniform and intuitive scaling. The Task Gallery hosts any Windows application, using a novel redirection mechanism that routes input and output between the 3D environment and unmodified 2D Windows applications. User studies suggest that the Task Gallery helps with task management, is enjoyable to use, and that the 3D metaphor evokes spatial memory and cognition.

We describe sensing techniques motivated by unique aspects of human-computer interaction with handheld devices in mobile settings.Special features of mobile interaction include changing orientation and position, changing venues, the use of computing as auxiliary to ongoing, real-world activities like talking to a colleague, and the general intimacy of use for such devices.We introduce and integrate a set of sensors into a handheld device, and demonstrate several new functionalities engendered by the sensors, such as recording memos when the device is held like a cell phone, switching between portrait and landscape display modes by holding the device in the desired orientation, automatically powering up the device when the user picks it up the device to start using it, and scrolling the display using tilt.We present an informal experiment, initial usability testing results, and user reactions to these techniques.

We propose a navigation technique for browsing large documents that integrates rate-based scrolling with automatic zooming.The view automatically zooms out when the user scrolls rapidly so that the perceptual scrolling speed in screen space remains constant.As a result, the user can efficiently and smoothly navigate through a large document without becoming disoriented by extremely fast visual flow.By incorporating semantic zooming techniques, the user can smoothly access a global overview of the document during rate-based scrolling.We implemented several prototype systems, including a web browser, map viewer, image browser, and dictionary viewer.An informal usability study suggests that for a document browsing task, most subjects prefer automatic zooming and the technique exhibits approximately equal performance time to scroll bars, suggesting that automatic zooming is a helpful alternative to traditional scrolling when the zoomed out view provides appropriate visual cues.

We can touch things, and our senses tell us when our hands are touching something. But most computer input devices cannot detect when the user touches or releases the device or some portion of the device. Thus, adding touch sensors to input devices offers many possibilities for novel interaction techniques. We demonstrate the TouchTrackball and the Scrolling TouchMouse, which use unobtrusive capacitance sensors to detect contact from the users hand without requiring pressure or mechanical actuation of a switch. We further demonstrate how the capabilities of these devices can be matched to an implicit interaction technique, the On-Demand Interface, which uses the passive information captured by touch sensors to fade in or fade out portions of a display depending on what the user is doing; a second technique uses explicit, intentional interaction with touch sensors for enhanced scrolling. We present our new devices in the context of a simple tax- onomy of tactile input technologies. Finally, we discuss the properties of touch-sensing as an input channel in general.

We present experimental work which explores how the match (or mismatch) between the input space of the hands and the output space of a graphical display influences two-handed input performance. During interaction with computers, a direct correspondence between the input and output spaces is often lacking. Not only are the hands disjoint from the display space, but the reference frames of the hands may in fact be disjoint from one another if two separate input devices (e.g. two mice) are used for two-handed input. In general, we refer to the workspace and origin within which the hands operate as kinesthetic reference frames. Our goal is to better understand how an interface designer's choice of kinesthetic reference frames influences a user's ability to coordinate two-handed movements, and to explore how the answer to this question may depend on the availability of visual feedback. Understanding this issue has implications for the design of two-handed interaction techniques and input devices, as well as for the reference principle of Guiard's Kinematic Chain model of human bimanual action. Our results suggest that the Guiard reference principle is robust with respect to variances in the kinesthetic reference frames as long as appropriate visual feedback is present.

The VideoMouse is a mouse that uses a camera as its input sensor. A real-time vision algorithm determines the six degree-of-freedom mouse posture, consisting of 2D motion, tilt in the forward/back and left/right axes, rotation of the mouse about its vertical axis, and some limited height sensing. Thus, a familiar 2D device can be extended for three-dimensional manipulation, while remaining suitable for standard 2D GUI tasks. We describe techniques for mouse functionality, 3D manipulation, navigating large 2D spaces, and using the camera for lightweight scanning tasks.

We describe input devices and two-handed interaction techniques to support map navigation tasks. We discuss several design variations and user testing of two-handed navigation techniques, including puck and stylus input on a Wacom tablet, as well as a novel design incorporating a touchpad (for the nonpreferred hand) and a mouse (for the preferred hand). To support the latter technique, we introduce a new input device, the TouchMouse, which is a standard mouse augmented with a pair of one-bit touch sensors, one for the palm and one for the index finger. Finally, we propose several enhancements to Buxton's three-state model of graphical input and extend this model to encompass two-handed input transactions as well. Keywords Two-handed input, three-state model, input devices, tablets, touchpads, TouchMouse, map navigation INTRODUCTION Two-handed input is a promising technique to improve the directness and degree of manipulation afforded by desktop computers. A strong foundation of re...

We discuss a two-handed user interface designed to support three-dimesional neurosurgical visualization. By itself, this system is a “point design,” an example of an advanced user interface technique. In this work, we argue that in order to understand why interaction techniques do or do not work, and to suggest possibilities for new techniques, it is important to move beyond point design and to introduce careful scientific measurement of human behavioral principles. In particular, we argue that the common-sense viewpoint that “two hands save time by working in parallel” may not always be an effective way to think about two-handed interface design because the hands do not necessarily work in parallel (there is a structure to two-handed manipulation) and because two hands do more than just save time over one hand (two hands provide the user with more information and can structure how the user thinks about a task). To support these claims, we present an interface design developed in collaboration with neurosurgeons which has undergone extensive informal usability testing, as well as a pair of formal experimental studies which investigate behavioral aspects of two-handed virtual object manipulation. Our hope is that this discussion will help others to apply the lessons in our neurosurgery application to future two-handed user interface designs.

We present an experiment on cooperative bimanual action.Right-handed subjects manipulated a pair of physical objects, a tool and a target object, so that the tool would touch a target on the object @g. 1).For this task, there is a marked specialization of the hands.Performance is best when the left hand orients the target object and the right hand manipulates the tool, but is significantly reduced when these roles are reversed.This suggests that the right hand operates relative to the frame-of-reference of the left hand.Furthermore, when physical constraints guide the tool placement, this fundamentally changes the type of motor control required.The task is tremendously simplified for both hands, and reversing roles of the hands is no longer an important factor.Thus, specialization of the roles of the hands is significant only for skilled manipulation.

We investigate the synergy of the two hands for virtual object manipuhttion.We report results from an experiment which suggest that the two hands together provide sufficient perceptual cues to form a frame of reference which is independent of visual feedback.The same is not true for one hand moving in empty space.Our inteqmtadon is that users may not have to constantly maintain visual attention when both hands can be involved in a manipulation.Our results suggest that using two hands can provide more than just a time savings over one-handed manipulation.llvo hands together provide the user with information which one hand alone cannot.Our results also suggest that using two hands can potentially impact performance at the cognitive level by changing how users think about a task.Since the user can potentially integrate subtasks controlled by each hand without an explicit cost to switch between subtasks, this encourages exploration of the task solution space.Fhmlly, to illustrate why one might expect this to be true, we present a task analysis which helps to reason about the differences between one and two-handed interfaces.

We report results from a formal user study of interactive 3D rotation using the mouse-driven Virtual Sphere and Arcball techniques, as well as multidimensional input techniques based on magnetic orientation sensors. Multidimensional input is often assumed to allow users to work quickly, but at the cost of precision, due to the instability of the hand moving in the open air. We show that, at least for the orientation matching task used in this experiment, users can take advantage of the integrated degrees of freedom provided by multidimensional input without necessarily sacrificing precision: using multidimensional input, users completed the experimental task up to 36% faster without any statistically detectable loss of accuracy. We also report detailed observations of common usability problems when first encountering the techniques. Our observations suggest some design issues for 3D input devices. For example, the physical form-factors of the 3D input device significantly influenced us...

Alice is a rapid prototyping system used to create three dimensional graphics simulations like those seen in virtual reality applications. Alice uses an interpreted language called Python as its scripting language to implement user actions. This interactive development environment allows users to explore many more design options than is possible in a compiled language environment. The alpha version of Alice for Windows 95 is available for free over the intemet, with the beta release scheduled for August.

Article Passive real-world interface props for neurosurgical visualization Share on Authors: Ken Hinckley University of Virginia, Departments of Neurosurgery and Computer Science University of Virginia, Departments of Neurosurgery and Computer ScienceView Profile , Randy Pausch University of Virginia, University of Virginia, Computer Science University of Virginia, University of Virginia, Computer ScienceView Profile , John C. Goble University of Virginia, Departments of Neurosurgery University of Virginia, Departments of NeurosurgeryView Profile , Neal F. Kassell University of Virginia, Departments of Neurosurgery University of Virginia, Departments of NeurosurgeryView Profile Authors Info & Claims CHI '94: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsApril 1994 Pages 452–458https://doi.org/10.1145/191666.191821Online:24 April 1994Publication History 192citation2,131DownloadsMetricsTotal Citations192Total Downloads2,131Last 12 Months132Last 6 weeks12 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

We present a survey of design issues for developing effective free-space three-dimensional (3D) user interfaces. Our survey is based upon previous work in 3D interaction, our experience in developing free-space interfaces, and our informal observations of test users. We illustrate our design issues using examples drawn from instances of 3D interfaces. For example, our first issue suggests that users have difficulty understanding three-dimensional space. We offer a set of strategies which may help users to better perceive a 3D virtual environment, including the use of spatial references, relative gesture, two-handed interaction, multisensory feedback, physical constraints, and head tracking. We describe interfaces which employ these strategies. Our major contribution is the synthesis of many scattered results, observations, and examples into a common framework. This framework should serve as a guide to researchers or systems builders who may not be familiar with design issues in spatial input. Where appropriate, we also try to identify areas in free-space 3D interaction which we see as likely candidates for additional research. An extended and annotated version of the references list for this paper is available on-line through mosaic at address