International audience

Although video is extremely useful for expressing interaction, post-hoc editing makes it impractical for rapid prototyping. We present an early-stage video-based design method — “editing-in-the-camera” — where title cards guide video capture and label video clips. This method lets designers easily create video prototypes that can be discussed within the same design session, without further editing. We present VideoClipper, a mobile app that incorporates this method by transforming sequences of title cards into an interactive storyboard that designers can shoot into directly. VideoClipper offers simple special effects to better illustrate user interaction with paper prototypes, including ghosting and stop-motion animation. We also present Collaborative VideoClipper, which was created during the COVID-19 pandemic to support multi-user, multi-device rapid prototyping with remote participants. We describe evaluation results of both applications and describe our experiences in diverse educational and professional settings, including brainstorming, interviewing, video prototyping, user studies and participatory design workshops.

User interfaces typically feature tools to act on objects and rely on the ability of users to discover or learn how to interact with them. Previous work in HCI has used the Theory of Affordances to explain how users understand the possibilities for action in digital environments. A complementary theory from cognitive neuroscience, Technical Reasoning, posits that users accumulate abstract knowledge of object properties and technical principles known as mechanical knowledge, essential in tool use. Drawing from this theory, we introduce interaction knowledge as the “mechanical” knowledge of digital environments. We provide evidence of its relevance by reporting on an experiment where participants performed tasks in a digital environment with ambiguous possibilities for interaction. We analyze how interaction knowledge was transferred across two digital domains, text editing and graphical editing, and conclude that interaction knowledge models an essential type of knowledge for interacting in the digital world.

Projections of computer languages are tools that help users interact with representations that better fit their needs than plain text. We collected 62 projections from the literature and from a design workshop and found that 60% of them can be implemented using a table, a graph or a form. However, projections are often hardcoded for specific languages and situations, and in most cases only the developers of a code editor can create or adapt projections, leaving no room for appropriation by their users. We introduce lorgnette, a new framework for letting programmers augment their code editor with projections. We demonstrate five examples that use lorgnette to create projections that can be reused in new contexts. We discuss how this approach could help democratise projections and conclude with future work.

How can we let users adapt video-based meetings as easily as they rearrange furniture in a physical meeting room? We describe a design space for video conferencing systems that includes a five-step “ladder of tailorability,” from minor adjustments to live reprogramming of the interface. We then present Mirrorverse and show how it applies the principles of computational media to support live tailoring of video conferencing interfaces to accommodate highly diverse meeting situations. We present multiple use scenarios, including a virtual workshop, an online yoga class, and a stand-up team meeting to evaluate the approach and demonstrate its potential for new, remote meetings with fluid transitions across activities.

Document description languages such as LaTeX are used extensively to author scientific and technical documents, but editing them is cumbersome: code-based editors only provide generic features, while WYSIWYG interfaces only support a subset of the language. Our interviews with 11 LaTeX users highlighted their difficulties dealing with textually-encoded abstractions and with the mappings between source code and document output. To address some of these issues, we introduce Transitional Representations for document description languages, which enable the visualisation and manipulation of fragments of code in relation to their generated output. We present i-LaTeX, a LaTeX editor equipped with Transitional Representations of formulae, tables, images, and grid layouts. A 16-participant experiment shows that Transitional Representations let them complete common editing tasks significantly faster, with fewer compilations, and with a lower workload. We discuss how Transitional Representations affect editing strategies and conclude with directions for future work.

A key aspect of knowledge work is the analysis and manipulation of sets of related documents. We conducted interviews with 12 patent examiners and 12 scientists and found that all face difficulties using specialized tools for managing text from multiple documents across interconnected activities, including searching, collecting, annotating, organizing, writing and reviewing, while manually tracking their provenance. We introduce Passages, interactive objects that reify text selections and can then be manipulated, reused, and shared across multiple tools. Passages directly supports the above-listed activities as well as fluid transitions among them. Two user studies show that participants found Passages both elegant and powerful, facilitating their work practices and enabling greater reuse and novel strategies for analyzing and composing documents. We argue that Passages offers a general approach applicable to a wide variety of text-based interactions.

Suggesting multiple target candidates based on touch input is a possible option for high-accuracy target selection on small touchscreen devices. But it can become overwhelming if suggestions are triggered too often. To address this, we propose SATS, a Suggestion-based Accurate Target Selection method, where target selection is formulated as a sequential decision problem. The objective is to maximize the utility: the negative time cost for the entire target selection procedure. The SATS decision process is dictated by a policy generated using reinforcement learning. It automatically decides when to provide suggestions and when to directly select the target. Our user studies show that SATS reduced error rate and selection time over Shift [51], a magnification-based method, and MUCS, a suggestion-based alternative that optimizes the utility for the current selection. SATS also significantly reduced error rate over BayesianCommand [58], which directly selects targets based on posteriors, with only a minor increase in selection time.

The Technical Reasoning hypothesis in cognitive neuroscience posits that humans engage in physical tool use by reasoning about mechanical interactions among objects. By modeling the use of objects as tools based on their abstract properties, this theory explains how tools can be re-purposed beyond their assigned function. This paper assesses the relevance of Technical Reasoning to digital tool use. We conducted an experiment with 16 participants that forced them to re-purpose commands to complete a text layout task. We analyzed self-reported scores of creative personality and experience with text editing, and found a significant association between re-purposing performance and creativity, but not with experience. Our results suggest that while most participants engaged in Technical Reasoning to re-purpose digital tools, some experienced “functional fixedness.” This work contributes Technical Reasoning as a theoretical model for the design of digital tools.

With the increasing interest in movement sonification and expressive gesture-based interaction, it is important to understand which factors contribute to movement learning and how. We explore the effects of movement sonification and users’ musical background on motor variability in complex gesture learning. We contribute an empirical study in which musicians and non-musicians learn two gesture sequences over three days, with and without movement sonification. Results show the interlaced interaction effects of these factors and how they unfold in the three-day learning process. For gesture 1, which is fast and dynamic with a direct “action-sound” sonification, movement sonification induces higher variability for both musicians and non-musicians on day 1. While musicians reduce this variability to a similar level as no auditory feedback condition on day 2 and day 3, non-musicians remain to have significantly higher variability. Across three days, musicians also have significantly lower variability than non-musicians. For gesture 2, which is slow and smooth with an “action-music” metaphor, there are virtually no effects. Based on these findings, we recommend future studies to take into account participants’ musical background, consider longitudinal study to examine these effects on complex gestures, and use awareness when interpreting the results given a specific design of gesture and sound.

Aerial hoops are circular, hanging devices for both acrobatic exercise and artistic performance that let us explore the role of interactive sonification in physical activity. We present SonicHoop, an augmented aerial hoop that generates auditory feedback via capacitive touch sensing, thus becoming a digital musical instrument that performers can play with their bodies. We compare three sonification strategies through a structured observation study with two professional aerial hoop performers. Results show that SonicHoop fundamentally changes their perception and choreographic processes: instead of translating music into movement, they search for bodily expressions that compose music. Different sound designs affect their movement differently, and auditory feedback, regardless of type of sound, improves movement quality. We discuss opportunities for using SonicHoop as an aerial hoop training tool, as a digital musical instrument, and as a creative object; as well as using interactive sonification in other acrobatic practices to explore full-body vertical interaction.

Selecting targets accurately and quickly with eye-gaze input remains an open research question. In this paper, we introduce BayesGaze, a Bayesian approach of determining the selected target given an eye-gaze trajectory. This approach views each sampling point in an eye-gaze trajectory as a signal for selecting a target. It then uses the Bayes' theorem to calculate the posterior probability of selecting a target given a sampling point, and accumulates the posterior probabilities weighted by sampling interval to determine the selected target. The selection results are fed back to update the prior distribution of targets, which is modeled by a categorical distribution. Our investigation shows that BayesGaze improves target selection accuracy and speed over a dwell-based selection method, and the Center of Gravity Mapping (CM) method. Our research shows that both accumulating posterior and incorporating the prior are effective in improving the performance of eye-gaze based target selection.

Although Human–Computer Interaction research has developed various theories and frameworks for analyzing new and existing interactive systems, few address thegenerationof novel technological solutions, and new technologies often lack theoretical foundations. We introduceGenerative Theories of Interaction, which draw insights from empirical theories about human behavior in order to define specific concepts and actionable principles, which, in turn, serve as guidelines for analyzing, critiquing, and constructing new technological artifacts. After introducing and defining Generative Theories of Interaction, we present three detailed examples from our own work: Instrumental Interaction, Human–Computer Partnerships, and Communities & Common Objects. Each example describes the underlying scientific theory and how we derived and applied HCI-relevant concepts and principles to the design of innovative interactive technologies. Summary tables offer sample questions that help analyze existing technology with respect to a specific theory, critique both positive and negative aspects, and inspire new ideas for constructing novel interactive systems.

Hick's law is a key quantitative law in Psychology that relates reaction time to the logarithm of the number of stimulus-response alternatives in a task. Its application to HCI is controversial: Some believe that the law does not apply to HCI tasks, others regard it as the cornerstone of interface design. The law, however, is often misunderstood. We review the choice-reaction time literature and argue that: (1) Hick's law speaks against, not for, the popular principle that 'less is better'; (2) logarithmic growth of observed temporal data is not necessarily interpretable in terms of Hick's law; (3) the stimulus-response paradigm is rarely relevant to HCI tasks, where choice-reaction time can often be assumed to be constant; and (4) for user interface design, a detailed examination of the effects on choice-reaction time of psychological processes such as visual search and decision making is more fruitful than a mere reference to Hick's law.

Writing technical documents frequently requires following constraints and consistently using domain-specific terms. We interviewed 12 legal professionals and found that they all use a standard word processor, but must rely on their memory to manage dependencies and maintain consistent vocabulary within their documents. We introduce Textlets, interactive objects that reify text selections into persistent items. We show how Textlets help manage consistency and constraints within the document, including selective search and replace, word count, and alternative wording. Eight participants tested a search-and-replace Textlet as a technology probe. All successfully interacted directly with the Textlet to perform advanced tasks; and most (6/8) spontaneously generated a novel replace-all-then-correct strategy. Participants suggested additional ideas, such as supporting collaborative editing over time by embedding a Textlet into the document to flag forbidden words. We argue that Textlets serve as a generative concept for creating powerful new tools for document editing.

Knowledge management and sharing involves a variety of specialized but isolated software tools, tied together by the files that these tools use and produce. We interviewed 23 scientists and found that they all had difficulties using the file system to keep track of, re-find and maintain consistency among related but distributed information. We introduce FileWeaver, a system that automatically detects dependencies among files without explicit user action, tracks their history, and lets users interact directly with the graphs representing these dependencies and version history. Changes to a file can trigger recipes, either automatically or under user control, to keep the file consistent with its dependants. Users can merge variants of a file, e.g. different output formats, into a polymorphic file, or morph, and automate the management of these variants. By making dependencies among files explicit and visible, FileWeaver facilitates the automation of workflows by scientists and other users who rely on the file system to manage their data.

Professional designers create mood boards to explore, visualize, and communicate hard-to-express ideas. We present ImageCascade, an intelligent, collaborative ideation tool that combines individual and shared work spaces, as well as collaboration with multiple forms of intelligent agents. In the collection phase, ImageCascade offers fluid transitions between serendipitous discovery of curated images via ImageCascade, combined text- and image-based Semantic search, and intelligent AI suggestions for finding new images. For later composition and reflection, ImageCascade provides semantic labels, generated color palettes, and multiple tag clouds to help communicate the intent of the mood board. A study of nine professional designers revealed nuances in designers' preferences for designer-led, system-led, and mixed-initiative approaches that evolve throughout the design process. We discuss the challenges in creating effective human-computer partnerships for creative activities, and suggest directions for future research.

Touchstone2 offers a direct-manipulation interface for generating and examining trade-offs in experiment designs. Based on interviews with experienced researchers, we developed an interactive environment for manipulating experiment design parameters, revealing patterns in trial tables, and estimating and comparing statistical power. We also developed TSL, a declarative language that precisely represents experiment designs. In two studies, experienced HCI researchers successfully used Touchstone2 to evaluate design trade-offs and calculate how many participants are required for particular effect sizes. We discuss Touchstone2's benefits and limitations, as well as directions for future research.

We present Videostrates, a concept and a toolkit for creating real-time collaborative video editing tools. Videostrates supports both live and recorded video composition with a declarative HTML-based notation, combining both simple and sophisticated editing tools that can be used collaboratively. Videostrates is programmable and unleashes the power of the modern web platform for video manipulation. We demonstrate its potential through three use scenarios: collaborative video editing with multiple tools and devices; orchestration of multiple live streams that are recorded and broadcast to a popular streaming platform; and programmatic creation of video using WebGL and shaders for blue screen effects. These scenarios only scratch the surface of Videostrates' potential, which opens up a design space for novel collaborative video editors with fully programmable interfaces.

Professional designers and developers often struggle when transitioning between the design and implementation of an interactive system. We conducted three studies that focused on the design of custom interactions to understand the mismatches between their processes, tools, and representations. We found that current practices induce unnecessary rework and cause discrepancies between design and implementation. We identified three recurring types of breakdowns: omitting critical details, ignoring edge cases, and disregarding technical limitations. We propose four design principles to create tools that mitigate these problems: Provide multiple viewpoints , maintain a single source of truth , reveal the invisible, and support design by enaction . We applied these principles to create E NACT , a live environment for prototyping touch-based interactions. We conducted two studies to assess E NACT and to compare designer–developer collaboration with E NACT versus current tools. Results suggest that E NACT helps participants detect more edge cases, increases designers’ participation and provides new opportunities for co-creation.

The design of Fitts' historical reciprocal tapping experiment gravely confounds index of difficulty ID with target distance D: Summary statistics for the candidate Fitts model and a competing model may appear identical, and the validity of Fitts' model for some tasks can be legitimately questioned. We show that the contamination of ID by either target distance D or width W is due to the common practices of pooling and averaging data belonging to different distance-width (D,W) pairs for the same ID, and taking a geometric progression for values of D and W. We analyze a case study of the validation of Fitts' law in eye-gaze movements, where an unfortunate experimental design has misled researchers into believing that eye-gaze movements are not ballistic. We then provide simple guidelines to prevent confounds: Practitioners should carefully design the experimental conditions of (D,W), fully distinguish data acquired for different conditions, and put less emphasis on r² scores. We also recommend investigating the use of stochastic sampling for D and W.

We introduce BIGFile, a new fast file retrieval technique based on the Bayesian Information Gain framework. BIGFile provides interface shortcuts to assist the user in navigating to a desired target (file or folder). BIGFile's split interface combines a traditional list view with an adaptive area that displays shortcuts to the set of file paths estimated by our computationally efficient algorithm. Users can navigate the list as usual, or select any part of the paths in the adaptive area. A pilot study of 15 users informed the design of BIGFile, revealing the size and structure of their file systems and their file retrieval practices. Our simulations show that BIGFile outperforms Fitchett et al.'s AccessRank, a best-of-breed prediction algorithm. We conducted an experiment to compare BIGFile with ARFile (AccessRank instantiated in a split interface) and with a Finder-like list view as baseline. BIGFile was by far the most efficient technique (up to 44% faster than ARFile and 64% faster than Finder), and participants unanimously preferred the split interfaces to the Finder.

Video prototypes help capture and communicate interaction with paper prototypes in the early stages of design. However, designers sometimes find it tedious to create stop-motion videos for continuous interactions and to re-shoot clips as the design evolves. We introduce Montage, a proof-of-concept implementation of a computer-assisted process for video prototyping. Montage lets designers progressively augment video prototypes with digital sketches, facilitating the creation, reuse and exploration of dynamic interactions. Montage uses chroma keying to decouple the prototyped interface from its context of use, letting designers reuse or change them independently. We describe how Montage enhances video prototyping by combining video with digital animated sketches, encourages the exploration of different contexts of use, and supports prototyping of different interaction styles.

Remote collaboration across wall-sized displays creates a key challenge: how to support audio-video communication among users as they move in front of the display. We present CamRay, a platform that uses camera arrays embedded in wall-sized displays to capture video of users and present it on remote displays according to the users' positions. We investigate two settings: in Follow-Remote, the position of the video window follows the position of the remote user; in Follow-Local, the video window always appears in front of the local user. We report the results of a controlled experiment showing that with Follow-Remote, participants are faster, use more deictic instructions, interpret them more accurately, and use fewer words. However, some participants preferred the virtual face-to-face created by Follow-Local when checking for their partners' understanding. We conclude with design recommendations to support remote collaboration across wall-sized displays.

Multi-touch wall-sized displays afford collaborative exploration of large datasets and re-organization of digital content. However, standard touch interactions, such as dragging to move content, do not scale well to large surfaces and were not designed to support collaboration, such as passing an object around. This paper introduces CoReach, a set of collaborative gestures that combine input from multiple users in order to manipulate content, facilitate data exchange and support communication. We conducted an observational study to inform the design of CoReach, and a controlled study showing that it reduced physical fatigue and facilitated collaboration when compared with traditional multi-touch gestures. A final study assessed the value of also allowing input through a handheld tablet to manipulate content from a distance.

This paper introduces BIGnav, a new multiscale navigation technique based on Bayesian Experimental Design where the criterion is to maximize the information-theoretic concept of mutual information, also known as information gain. Rather than simply executing user navigation commands, BIGnav interprets user input to update its knowledge about the user's intended target. Then it navigates to a new view that maximizes the information gain provided by the user's expected subsequent input. We conducted a controlled experiment demonstrating that BIGnav is significantly faster than conventional pan and zoom and requires fewer commands for distant targets, especially in non-uniform information spaces. We also applied BIGnav to a realistic application and showed that users can navigate to highly probable points of interest on a map with only a few steps. We then discuss the tradeoffs of BIGnav--including efficiency vs. increased cognitive load--and its application to other interaction tasks.

Traditional graphic design tools emphasize the grid for structuring layout. Interviews with professional graphic designers revealed that they use surprisingly sophisticated structures that go beyond the grid, which we call graphical substrates. We present a framework to describe how designers establish graphical substrates based on properties extracted from concepts, content and context, and use them to compose layouts in both space and time. We developed two technology probes to explore how to embed graphical substrates into tools. Contextify lets designers tailor layouts according to each reader's intention and context; while Linkify lets designers create dynamic layouts based on relationships among content properties. We tested the probes with professional graphic designers, who all identified novel uses in their current projects. We incorporated their suggestions into StyleBlocks, a prototype that reifies CSS declarations into interactive graphical substrates. Graphical substrates offer an untapped design space for tools that can help graphic designers generate personal layout structures.

Users must manage frequent software and operating system upgrades across multiple computing devices. While current research focuses primarily on the security aspect, we investigate the user's perspective of upgrading software. Our first study (n=65) found that users delay major upgrades by an average of 80 days. We then ran a field study (n=14), beginning with in-depth observations during an operating system upgrade, followed by a four-week diary study. Very few participants prepared for upgrades (e.g., backing up files), and over half had negative reactions to the upgrade process and other changes (e.g., bugs, lost settings, unwanted features). During the upgrade process, waiting times were too long, feedback was confusing or misleading, and few had clear mental models of what was happening. Users almost never mentioned security as a concern or reason for upgrading. By contrast, interviews (n=3) with technical staff responsible for one organization's upgrades focused only on security and licensing, not user interface changes. We conclude with recommendations to improve the user's upgrade experience.

Professional interaction designers and software developers have different trainings and skills, yet they need to closely collaborate to create interactive systems. We conducted three studies to understand the mismatches between their processes, tools and representations. Based on 16 interviews, we found that current practices induce unnecessary rework and cause discrepancies between the original design and the implementation. We identified three key design breakdowns where designers omitted critical details, ignored the presence of edge cases or disregarded technical limitations. We next observed two face-to-face meetings between designers and developers. We found that early involvement of the developer helped to mitigate potential design breakdowns but new ones emerged as the project unfolded. Finally, we ran a participatory design session with two designer/developer pairs. Both pairs had difficulty representing and communicating pre-existing interactions. Creating complete interaction descriptions required iterating from individual examples to rule-based representations. We conclude with implications for designing collaborative tools that facilitate the designer's ability to express and the developer's ability to implement complex interactive systems.

Wall-sized displays support small groups of users working together on large amounts of data. Observational studies of such settings have shown that users adopt a range of collaboration styles, from loosely to closely coupled. Shared interaction techniques, in which multiple users perform a command collaboratively, have also been introduced to support co-located collaborative work. In this paper, we operationalize five collaborative situations with increasing levels of coupling, and test the effects of providing shared interaction support for a data manipulation task in each situation. The results show the benefits of shared interaction for close collaboration: it encourages collaborative manipulation, it is more efficient and preferred by users, and it reduces physical navigation and fatigue. We also identify the time costs caused by disruption and communication in loose collaboration and analyze the trade-offs between parallelization and close collaboration. These findings inform the design of shared interaction techniques to support collaboration on wall-sized displays.

Aligning and distributing graphical objects is a common, but cumbersome task. In a preliminary study (six graphic designers, six non-designers), we identified three key problems with current tools: lack of persistence, unpredictability of results, and inability to 'tweak' the layout. We created StickyLines, a tool that treats guidelines as first-class objects: Users can create precise, predictable and persistent interactive alignment and distribution relationships, and 'tweaked' positions can be maintained for subsequent interactions. We ran a [2x2] within-participant experiment to compare StickyLines with standard commands, with two levels of layout difficulty. StickyLines performed 40% faster and required 49% fewer actions than traditional alignment and distribution commands for complex layouts. In study three, six professional designers quickly adopted StickyLines and identified novel uses, including creating complex compound guidelines and using them for both spatial and semantic grouping.

This paper presents a controlled experiment assessing the accuracy when interpreting remote users showing a shared object on a large wall-sized display, either by looking at it or by looking and pointing at it. We analyze both distance and angle errors and how they are sensitive to the relative position be- tween the remote viewer and the video feed. We show that the remote user can accurately determine the target, that eye gaze alone is more accurate than combined with the hand, and that the relative position between the viewer and the video feed has little effect on accuracy. These findings can inform the design of future telepresence systems for wall-sized displays.

We revisit Alan Kay's early vision of dynamic media that blurs the distinction between documents and applications. We introduce shareable dynamic media that are malleable by users, who may appropriate them in idiosyncratic ways; shareable among users, who collaborate on multiple aspects of the media; and distributable across diverse devices and platforms. We present Webstrates, an environment for exploring shareable dynamic media. Webstrates augment web technology with real-time sharing. They turn web pages into substrates, i.e. software entities that act as applications or documents depending upon use. We illustrate Webstrates with two implemented case studies: users collaboratively author an article with functionally and visually different editors that they can personalize and extend at run-time; and they orchestrate its presentation and audience participation with multiple devices. We demonstrate the simplicity and generative power of Webstrates with three additional prototypes and evaluate it from a systems perspective.

Ultra-high resolution wall-sized displays (“ultra-walls”) are effective for presenting large datasets, but their size and resolution make traditional pointing techniques inadequate for precision pointing. We study mid-air pointing techniques that can be combined with other, domain-specific interactions. We first explore the limits of existing single-mode remote pointing techniques and demonstrate theoretically that they do not support high-precision pointing on ultra-walls. We then explore solutions to improve mid-air pointing efficiency: a tunable acceleration function and a framework for dual-precision (DP) techniques, both with precise tuning guidelines. We designed novel pointing techniques following these guidelines, several of which outperform existing techniques in controlled experiments that involve pointing difficulties never tested prior to this work. We discuss the strengths and weaknesses of our techniques to help interaction designers choose the best technique according to the task and equipment at hand. Finally, we discuss the cognitive mechanisms that affect pointing performance with these techniques.

Pointing on large displays with an indirect, relative pointing device such as a touchpad often requires clutching. This article introduces gliding, where the cursor continues to move during the clutching gestures. The effect is that of controlling the cursor as a detached object that can be pushed, with inertia and friction similar to a puck being pushed on a table. We analyze gliding from a practical and a theoretical perspective and report on two studies. The first controlled experiment establishes that gliding reduces clutching and can improve pointing performance for large distances. We introduce cursor efficiency to capture the effects of gliding on clutching. The second experiment demonstrates that participants use gliding even when an efficient acceleration function lets them perform the task without it, without degrading performance.

The advent of ultra-high resolution wall-size displays and their use for complex tasks require a more systematic analysis and deeper understanding of their advantages and drawbacks compared with desktop monitors. While previous work has mostly addressed search, visualization and sense-making tasks, we have designed an abstract classification task that involves explicit data manipulation. Based on our observations of real uses of a wall display, this task represents a large category of applications. We report on a controlled experiment that uses this task to compare physical navigation in front of a wall-size display with virtual navigation using pan-and-zoom on the desktop. Our main finding is a robust interaction effect between display type and task difficulty: while the desktop can be faster than the wall for simple tasks, the wall gains a sizable advantage as the task becomes more difficult. A follow-up study shows that other desktop techniques (overview+detail, lens) do not perform better than pan-and-zoom and are therefore slower than the wall for difficult tasks.

Rich interaction with high-resolution wall displays is not limited to remotely pointing at targets. Other relevant types of interaction include virtual navigation, text entry, and direct manipulation of control widgets. However, most techniques for remotely acquiring targets with high precision have studied remote pointing in isolation, focusing on pointing efficiency and ignoring the need to support these other types of interaction. We investigate high-precision pointing techniques capable of acquiring targets as small as 4 millimeters on a 5.5 meters wide display while leaving up to 93 % of a typical tablet device's screen space available for task-specific widgets. We compare these techniques to state-of-the-art distant pointing techniques and show that two of our techniques, a purely relative one and one that uses head orientation, perform as well or better than the best pointing-only input techniques while using a fraction of the interaction resources.

Effectively planning a large multi-track conference requires an understanding of the preferences and constraints of organizers, authors, and attendees. Traditionally, the onus of scheduling the program falls on a few dedicated organizers. Resolving conflicts becomes difficult due to the size and complexity of the schedule and the lack of insight into community members' needs and desires. Cobi presents an alternative approach to conference scheduling that engages the entire community in the planning process. Cobi comprises (a) communitysourcing applications that collect preferences, constraints, and affinity data from community members, and (b) a visual scheduling interface that combines communitysourced data and constraint-solving to enable organizers to make informed improvements to the schedule. This paper describes Cobi's scheduling tool and reports on a live deployment for planning CHI 2013, where organizers considered input from 645 authors and resolved 168 scheduling conflicts. Results show the value of integrating community input with an intelligent user interface to solve complex planning tasks.

While interaction techniques that use the temporal dimension have been used for a long time, such as multiple clicks or spring-loaded widgets, more advanced uses of rhythmic patterns have received little attention in HCI. Using such temporal structures to convey information can be particularly useful in situations where the visual channel is overloaded or even not available. In this paper we introduce Rhythmic Interaction as the use of rhythms for input. We report the results of two experiments that show that (i) rhythmic patterns can be efficiently reproduced by novice users and recognized by computer algorithms, and (ii) rhythmic patterns can be memorized as efficiently as traditional shortcuts when associating them with visual commands. Overall, these results demonstrate the potential of Rhythmic Interaction and open the way to a richer repertoire of interaction techniques.

Augmented Reality (AR) has been proved useful to guide operational tasks in professional domains by reducing the shift of attention between instructions and physical objects. Modern smartphones make it possible to use such techniques in everyday tasks, but raise new challenges for the usability of AR in this context: small screen, occlusion, operation "through a lens". We address these problems by adding real-time feedback to the AR overlay. We conducted a controlled experiment comparing AR with and without feedback, and with standard textual and graphical instructions. Results show significant benefits for mobile AR with feedback and reveals some problems with the other techniques.

This paper presents a novel middleware for developing flexible interactive multi-surface applications. Using a scenario-based approach, we identify the requirements for this type of applications. We then introduce Substance, a data-oriented framework that decouples functionality from data, and Shared Substance, a middleware implemented in Substance that provides powerful sharing abstractions. We describe our implementation of two applications with Shared Substance and discuss the insights gained from these experiments. Our finding is that the combination of a data-oriented programming model with middleware support for sharing data and functionality provides a flexible, robust solution with low viscosity at both design-time and run-time.

This article introduces runtime toolkit overloading, a novel approach to help third-party developers modify the interaction and behavior of existing software applications without access to their underlying source code. We describe the abstractions provided by this approach as well as the mechanisms for implementing them in existing environments. We describe Scotty, a prototype implementation for Mac OS X Cocoa that enables developers to modify existing applications at runtime, and we demonstrate a collection of interaction and functional transformations on existing off-the-shelf applications. We show how Scotty helps a developer make sense of unfamiliar software, even without access to its source code. We further discuss what features of future environments would facilitate this kind of runtime software development.

This paper addresses interaction in multi-surface environments and questions whether the current application-centric approaches to user interfaces are adequate in this context, and presents an alternative approach based on instrumental interaction. The paper presents the VIGO (Views, Instruments, Governors and Objects) architecture and describes a prototype implementation. It then illustrates how to apply VIGO to support distributed interaction. Finally, it demonstrates how a classical Ubicomp interaction technique, Pick-and-Drop, can be easily implemented using VIGO.

International audience

Abstract This article describes SwingStates, a Java toolkit designed to facilitate the development of graphical user interfaces and bring advanced interaction techniques to the Java platform. SwingStates is based on the use of finite‐state machines specified directly in Java to describe the behavior of interactive systems. State machines can be used to redefine the behavior of existing Swing widgets or, in combination with a new canvas widget that features a rich graphical model, to create brand new widgets. SwingStates also supports arbitrary input devices to implement novel interaction techniques based, for example, on bi‐manual or pressure‐sensitive input. We have used SwingStates in several Master's‐level classes over the past two years and have developed a benchmark approach to evaluate the toolkit in this context. The results demonstrate that SwingStates can be used by non‐expert developers with little training to successfully implement advanced interaction techniques. Copyright © 2007 John Wiley & Sons, Ltd.

Touchstone is an open-source experiment design platform designed to help establish a solid research foundation for HCI in the area of novel interaction techniques. Touchstone includes a design platform for exploring alternative designs of controlled laboratory experiments, a run platform for running subjects and a limited analysis platform for advice and access to on-line statistics packages. Designed for HCI researchers and their students, Touchstone facilitates the process of creating new experiments, as well as replicating and extending experiments in the research literature. We tested Touchstone by designing two controlled experiments. One illustrates how to create a new experiment from scratch. The other replicates and extends a previous study of multiscale pointing interaction techniques: OrthoZoom was fastest, followed by bi-manual Pan & Zoom; SDAZ and traditional Pan & Zoom were consistently slower.

A number of experimental studies based on domain-specific tasks have evaluated the efficiency of navigation techniques for searching multi-scale worlds. The discrepancies among their results call for a more generic framework similar in spirit to Fitts' reciprocal pointing task, but adapted to a task that significantly differs from pure pointing. We introduce such a framework based on an abstract task and evaluate how four multi-scale navigation techniques perform in one particular multi-scale world configuration. Experimental findings indicate that, in this context, pan & zoom combined with an overview is the most efficient technique of all four, and that focus + context techniques perform better than classical pan & zoom. We relate these findings to more realistic situations, discuss their applicability, and how the framework can be used to cover a broad range of situations.

Using a stylus on a tablet computer to acquire small targets can be challenging. In this paper we present pointing lenses -- interaction techniques that help users acquire and select targets by presenting them with an enlarged visual and interaction area. We present and study three pointing lenses for pen-based systems and find that our proposed Pressure-Activated Lens is the top overall performer in terms of speed, accuracy and user preference. In addition, our experimental results not only show that participants find all pointing lenses beneficial for targets smaller than 5 pixels, but they also suggest that this benefit may extend to larger targets as well.

Structured graphics models such as Scalable Vector Graphics (SVG) enable designers to create visually rich graphics for user interfaces. Unfortunately current programming tools make it difficult to implement advanced interaction techniques for these interfaces. This paper presents the Hierarchical State Machine Toolkit (HsmTk), a toolkit targeting the development of rich interactions. The key aspect of the toolkit is to consider interactions as first-class objects and to specify them with hierarchical state machines. This approach makes the resulting behaviors self-contained, easy to reuse and easy to modify. Interactions can be attached to graphical elements without knowing their detailed structure, supporting the parallel refinement of the graphics and the interaction.

The current GUI is like a flight simulator whose camera points fixedly at right angle to the document, thus preventing users from looking ahead while navigating. We argue that perspective viewing of usual planar documents can help navigation. We analyze the scale implosion problem that arises with tilted cameras and we report the data of a formal experiment on document navigation with perspective views.

This article describes SwingStates, a library that adds state machines to the Java Swing user interface toolkit. Unlike traditional approaches, which use callbacks or listeners to define interaction, state machines provide a powerful control structure and localize all of the interaction code in one place. SwingStates takes advantage of Java's inner classes, providing programmers with a natural syntax and making it easier to follow and debug the resulting code. SwingStates tightly integrates state machines, the Java language and the Swing toolkit. It reduces the potential for an explosion of states by allowing multiple state machines to work together. We show how to use SwingStates to add new interaction techniques to existing Swing widgets, to program a powerful new Canvas widget and to control high-level dialogues.

Although the power of personal computers has increased 1000-fold over the past 20 years, user interfaces remain essentially the same. Innovations in HCI research, particularly novel interaction techniques, are rarely incorporated into products. In this paper I argue that the only way to significantly improve user interfaces is to shift the research focus from designing interfaces to designing interaction. This requires powerful interaction models, a better understanding of both the sensory-motor details of interaction and a broader view of interaction in the context of use. It also requires novel interaction architectures that address reinterpretability, resilience and scalability.

Using a new taxonomy of pointing tasks which includes view pointing beside traditional cursor pointing, we introduce the concept of multi-scale pointing. Analyzing the impact of view size, we demonstrate theoretically and experimentally that (1) the time needed to reach a remotely located target in a multi-scale interface still obeys Fitts' law and (2) the bandwidth of the interaction (i.e., the inverse of Fitts' law slope) is proportional to view size, a relationship bounded by an early ceiling effect. We discuss these results with special reference to navigation in miniaturized and enlarged interfaces.

We introduce semantic pointing, a novel interaction technique that improves target acquisition in graphical user interfaces (GUIs). Semantic pointing uses two independent sizes for each potential target presented to the user: one size in motor space adapted to its importance for the manipulation, and one size in visual space adapted to the amount of information it conveys. This decoupling between visual and motor size is achieved by changing the control-to-display ratio according to cursor distance to nearby targets. We present a controlled experiment supporting our hypothesis that the performance of semantic pointing is given by Fitts' index of difficulty in motor rather than visual space. We apply semantic pointing to the redesign of traditional GUI widgets by taking advantage of the independent manipulation of motor and visual widget sizes.

We describe a new method for use in the process of co-designing technologies with users called technology probes. Technology probes are simple, flexible, adaptable technologies with three interdisciplinary goals: the social science goal of understanding the needs and desires of users in a real-world setting, the engineering goal of field-testing the technology, and the design goal of inspiring users and researchers to think about new technologies. We present the results of designing and deploying two technology probes, the messageProbe and the videoProbe, with diverse families in France, Sweden, and the U.S. We conclude with our plans for creating new technologies for and with families based on our experiences.

McGuffin and Balakrishnan (M&B) have recently reported evidence that target expansion during a reaching movement reduces pointing time even if the expansion occurs as late as in the last 10% of the distance to be covered by the cursor. While M&B massed their static and expanding targets in separate blocks of trials, thus making expansion predictable for participants, we replicated their experiment with one new condition in which the target could unpredictably expand, shrink, or stay unchanged. Our results show that target expansion occurring as late as in M&B's experiment enhances pointing performance in the absence of expectation. We discuss these findings in terms of the basic human processes that underlie target-acquisition movements, and we address the implications for user interface design by introducing a revised design for the Mac OS X Dock.

This note presents everal techniques to improve window management with overlapping windows: tabbed windows, turning and peeling back windows, and snapping and zipping windows.

This paper presents three design principles to support the development of large-scale applications and take advantage of recent research in new interaction techniques: Reification turns concepts into first class objects, polymorphism permits commands to be applied to objects of different types, and reuse makes both user input and system output accessible for later use. We show that the power of these principles lies in their combination. Reification creates new objects that can be acted upon by a small set of polymorphic commands, creating more opportunities for reuse. The result is a simpler yet more powerful interface.

This article introduces a new interaction model called Instrumental Interaction that extends and generalizes the principles of direct manipulation. It covers existing interaction styles, including traditional WIMP interfaces, as well as new interaction styles such as two-handed input and augmented reality. It defines a design space for new interaction techniques and a set of properties for comparing them. Instrumental Interaction describes graphical user interfaces in terms of domain objects and interaction instruments. Interaction between users and domain objects is mediated by interaction instruments, similar to the tools and instruments we use in the real world to interact with physical objects. The article presents the model, applies it to describe and compare a number of interaction techniques, and shows how it was used to create a new interface for searching and replacing text.

We have developed an interface for editing and simulating Coloured Petri Nets based on toolglasses, marking menus and bi-manual interaction, in order to understand how novel interaction techniques could be supported by a new generation of user interface toolkits.The architecture of CPN2000 is based on three components: the Document Structure stores all the persistent data in the system; the Display Structure represents the contents of the screen and implements rendering and hit detection algorithms; and the Input Structure uses "instruments" to manage interaction.The rendering engine is based on OpenGL and a number of techniques have been developed to take advantage of 3D accelerated graphics for a 2D application.Performance data show that high frame rates have been achieved with off-theshelf hardware even with a non-optimized redisplay.This work paves the way towards a post-WIMP UI toolkit.

Fitts pointing model has proven extremely useful for understanding basic selection in WIMP user interfaces. Yet todays interfaces involve more complex navigation within electronic environments. As navigation amounts to a form of multi-scale pointing, Fitts model can be applied to these more complex tasks. We report the results of a preliminary pointing experiment that shows that users can handle higher levels of task difficulty with two-scale rather than traditional one-scale pointing control. Also, in tasks with very high-precision hand movements, performance is higher with a stylus than with a mouse.

DIVA supports exploratory data analysis of multimedia streams, enabling users to visualize, explore and evaluate patterns in data that change over time. The underlying stream algebra provides the mathematical basis for operating on diverse kinds of streams. The streamer visualization technique provides a smooth transition between spatial and temporal views of the data. Mapping source and presentation streams into a two-dimensional space provides users with a direct manipulation, nontemporal interface for viewing and editing streams. DIVA was developed to help us analyze both qualitative and quantitative data collected in our research with French air traffic controllers, including video of controllers at work, audio records of telephone, radio and other conversations, output from tools such as RADAR, and coded logs based on our observations. Although our emphasis is on exploratory data analysis, DIVA's stream architecture should prove useful for a wide variety of multimedia applications.

International audience

ENO is an audio server designed to make it easy for applications in the Unix environment to incorporate non-speech audio cues. At the physical level, ENO manages a shared resource, namely the audio hardware. At the logical level, it manages a sound space that is shared by various client applications. Instead of dealing with sound in terms of its physical description (i.e., sampled sounds), ENO allows sounds to be presented and controlled in terms of higher-level descriptions of sources, interactions, attributes, and sound space. Using this structure, ENO can facilitate the creation of consistent, rich systems of audio cues. In this paper, we discuss the justification, design, and implementation of ENO.

Computer supported cooperative work (CSCW) is a rapidly growing field. Real-time groupware systems are addressed that allow a group of users to edit a shared document. The architecture and concurrency control algorithm used in this system are described. The algorithm is based on the semantics of the application and can be used by the developers of other groupware systems. The authors begin by introducing the notion of a purely replicated architecture and then present GroupDesign, a shared drawing tool implemented with this architecture. They then present the main parts of the algorithm that implement the distribution. The algorithm gives the best response time for the interface and reduces the number of undo and redo operations when conflicts occur.>

This article explores real-time groupware systems from the perspective of both the users and the designer. This exploration is carried out through the description of GroupDesign, a real-time multi-user drawing tool that we have developed. From the perspective of the users, we present a number of functionalities that we feel necessary in any real-time groupware system: Graphic & Audio Echo, Localization, Identification, Age, and History. From the perspective of the designer, we demonstrate the possibility of creating a multi-user application from a single-user one, and we introduce the notion of purely replicated architecture.