Swarm UIs provide assistance to support users in their tasks and are increasingly explored in HCI. This paper studies the extent to which this assistance impacts users’ sense of agency. A reduced sense of agency can lead to non-use of the interface or a diminishing sense of responsibility regarding the consequences of users’ actions. We conduct three experiments studying the impact of three factors on the sense of agency: the level of assistance, the task difficulty, and the predictability of modules. Our nine assistance levels vary in system autonomy and module coordination (proxy vs. no proxy). We find that higher assistance reduces users’ sense of agency, and this effect is not impacted by task difficulty. Predictability only impacts the least assistive interaction techniques. Our results will foster users’ acceptance, responsibility, and use of swarm UIs.

Scrolling is ubiquitous in our daily computing experience. We explore how single-handed microgestures can be used for scrolling. Based on an analysis of the basic components necessary for scrolling, we selected 3 microgestures: Tap, Hold and Drag. Considering both rate and position controls, we designed 4 microgesture-based scrolling techniques adapted to these 3 microgestures. We contrasted these 4 techniques in a laboratory experiment with 24 participants who performed 2 tasks: a reciprocal selection task, where participants scrolled the view to reach and select a target; and a counting task, where participants scrolled the view to count image occurrences. Our results suggest that the technique based on Drag microgestures with rate control is the most effective for scrolling operations, regardless of the task. This work demonstrates that microgestures, with their advantages for frequent everyday tasks, offer a promising approach to continuous and efficient scrolling control.

With the rapid development of microgesture research, numerous recognition devices with different sensors (e.g. force-sensitive, IMU), form factors (e.g. glove, ring) and recognition algorithms (e.g. ad-hoc, random forest) have become available. With this increasing number and diversity of recognition devices, it is important to be able to easily switch between devices and/or modify the set of recognized microgestures. To address these challenges, we propose µPoly, a first microgesture toolkit based on the µGlyph microgesture notation. µPoly takes as input the µGlyph descriptions of elementary microgestures. These inputs are combined to recognize complex microgestures described using µGlyph. µPoly therefore offers a standard for microgesture events. The design of µPoly is based on properties taken from the toolkit literature. To highlight these properties: we present (1) how practitioners are using the toolkit in four scenarios, and report on user testing and (2) how applications are being developed with µPoly.

Microgestures, i.e . fast and subtle finger movements, have shown a high potential for ubiquitous interaction. However, work to-date either focuses on grasp contexts (holding an object) or on the free-hand context (no held object). These two contexts influence the microgestures feasibility. Researchers have created sets of microgesture feasible across the entire taxonomy of everyday grasps, called transferable microgestures. However, those sets include a limited number of microgestures as compared to those for the free-hand context, for which microgestures are distinguished according to fine characteristics such as the part of the finger being touched or the number of fingers used. We provide knowledge and methods for identifying and recognizing microgestures that can transfer across contexts. First, we report a study on ergonomics factors that influence the feasibility of a microgesture in a given context. Then, we propose a conceptual model serving as a tool to determine the feasibility of a microgesture in a given context without the need for time-consuming user studies. As expected, not all microgestures were transferable to all considered contexts. Thus, we then expose two different ways of defining a set of microgestures transferable between free-hand and grasping contexts. Finally, we report a user study on recognition factors of a transferable microgesture set. • Provides the results of a study on the feasibility of transferable microgestures. • Defines a set of rules for predictively determining the feasibility of microgestures. • Explores design of transferable microgesture sets for free-hand and grasp contexts. • Empirically demonstrates that different phalanxes can be used in grasp contexts.

We explore the use of vibrotactile haptic cues for representing microgestures. We built a four-axes haptic device for providing vibrotactile cues mapped to all four fingers. We also designed six patterns, inspired by six most commonly studied microgestures. The patterns can be played independently on each axis of the device. We ran an experiment with 36 participants testing three different device locations (fingers, back of the hand, and forearm) for pattern and axis recognition. For all three device locations, participants interpreted the patterns with similar accuracy. We also found that they were better at distinguishing the axes when the device is placed on the fingers. Hand and Forearm device locations remain suitable alternatives but involve a greater trade-off between recognition rate and expressiveness. We report the recognition rates obtained for the different patterns, axes and their combinations per device location. These results per device location are important, as constraints of various kinds, such as hardware, context of use and user activities, influence device location. We discuss this choice of device location by improving literature microgesture-based scenarios with haptic feedback or feedforward.

An interactive application’s overall pace of interaction is a combination of the user’s pace and the system’s pace, and if the system’s pace is mismatched to the user’s pace (e.g., timeouts or animations are too fast or slow for the user), usability and user experience can be impaired. Through a series of four studies, we investigated whether users prefer systems where the system’s pace better matches their own pace. All of the studies used common drag-and-drop interactions with hierarchical folder widgets, in which a folder would expand when the cursor hovered over it for a timeout period. If the system pace in these interactions is too fast (i.e., the timeout is too short), then the user’s performance and subjective experience is likely to be impaired because of unintended expansions; and if the system pace is too slow (i.e., the timeout is too long), then performance and experience could be impaired by unnecessary delay before folders expand. The first experiment was designed to validate the premise that fast-paced users prefer a fast system pace to a slow one (and the inverse for slow-paced users), and results confirmed this premise. The second study used the first experiment’s data to look for measures of user pace that could enable automatic adaptation of system pace, and also examined whether participants adjusted their pace towards that of the system. The study found reliable measures of user pace and showed that participants do entrain to the system’s pace. The third and fourth studies examined whether users would prefer a system that adapted its pace to the user over a system that used a static baseline pace. Results indicated that a majority of fast-paced users preferred the adaptive interface, but that slow-paced users generally preferred the static baseline interface. We discuss several design implications, including opportunities for systems to improve user experience for fast users by automatically adapting system pace to user pace.

Hand microgestures are promising for mobile interaction with wearable devices. However, they will not be adopted if practitioners cannot communicate to users the microgestures associated with the commands of their applications. This requires unambiguous representations that simultaneously show the multiple microgestures available to control an application. Using a systematic approach, we evaluate how these representations should be designed and contrast 4 conditions depending on the microgestures (tap-swipe and tap-hold) and fingers (index and index-middle) considered. Based on the results, we design a simultaneous representation of microgestures for a given set of 14 application commands. We then evaluate the usability of the representation for novice users and the suitability of the representation for small screens compared with a baseline. Finally, we formulate 8 recommendations based on the results of all the experiments. In particular, redundant graphical and textual representations of microgestures should only be displayed for novice users.

In the active field of hand microgestures, microgesture descriptions are typically expressed informally and are accompanied by images, leading to ambiguities and contradictions. An important step in moving the field forward is a rigorous basis for precisely describing, comparing, and analyzing microgestures. Towards this goal, we propose µGlyph, a hybrid notation based on a vocabulary of events inspired by finger biomechanics. First, we investigate the expressiveness of µGlyph by building a database of 118 microgestures extracted from the literature. Second, we experimentally explore the usability of µGlyph. Participants correctly read and wrote µGlyph descriptions 90% of the time, as compared to 46% for conventional descriptions. Third we present tools that promote µGlyph usage, including a visual editor with LaTeX export. We finally describe how µGlyph can guide research on designing, developing, and evaluating microgesture interaction. Results demonstrate the strong potential of µGlyph to establish a common ground for microgesture research.

Mid-air pointing is widely used for 3D selection in Mixed Reality but leads to arm fatigue. In a first exploratory experiment we study a two-phase design and compare modalities for each phase: mid-air gestures, eye-gaze and microgestures. Results suggest that eye-gaze and microgestures are good candidates to reduce fatigue and improve interaction speed. We therefore propose two 3D selection techniques: Look&MidAir and Look&Micro. Both techniques include a first phase during which users control a cone directed along their eye-gaze. Using the flexion of their non-dominant hand index finger, users pre-select the objects intersecting this cone. If several objects are pre-selected, a disambiguation phase is performed using direct mid-air touch for Look&MidAir or thumb to finger microgestures for Look&Micro. In a second study, we compare both techniques to the standard raycasting technique. Results show that Look&MidAir and Look&Micro perform similarly. However they are 55% faster, perceived easier to use and are less tiring than the baseline. We discuss how the two techniques could be combined for greater flexibility and for object manipulation after selection.

The representations of microgestures are essentials for researchers presenting their results through academic papers and system designers proposing tutorials to novice users. However, those representations remain disparate and inconsistent. As a first attempt to investigate how to best graphically represent microgestures, we created 21 designs, each depicting static and dynamic versions of 4 commonly used microgestures (tap, swipe, flex and hold). We first studied these designs in a quantitative online experiment with 45 participants. We then conducted a qualitative laboratory experiment in Augmented Reality with 16 participants. Based on the results, we provide design guidelines on which elements of a microgesture should be represented and how. In particular, it is recommended to represent the actuator and the trajectory of a microgesture. Also, although preferred by users, dynamic representations are not considered better than their static counterparts for depicting a microgesture and do not necessarily result in a better user recognition.

We present a set of 8 thumb-to-finger microgestures (TTF µGestures) that can be used as an additional modality to enrich touch interaction in eyes-free situations. TTF µGestures possess characteristics especially suited for people with visual impairment (PVI). They have never been studied specifically for PVI to improve accessibility of touchscreen devices. We studied a set of 33 common TTF µGestures to determine which are feasible and usable without seeing while the index is touching a surface. We found that the constrained position of the hand and the absence of vision prevent participants from being able to efficiently target a specific phalanx.

The overall pace of interaction combines the user’s pace and the system’s pace, and a pace mismatch could impair user preferences (e.g., animations or timeouts that are too fast or slow for the user). Motivated by studies of speech rate convergence, we conducted an experiment to examine whether user preferences for system pace are correlated with user pace. Subjects first completed a series of trials to determine their user pace. They then completed a series of hierarchical drag-and-drop trials in which folders automatically expanded when the cursor hovered for longer than a controlled timeout. Results showed that preferences for timeout values correlated with user pace – slow-paced users preferred long timeouts, and fast-paced users preferred short timeouts. Results indicate potential benefits in moving away from fixed or customisable settings for system pace. Instead, systems could improve preferences by automatically adapting their pace to converge towards that of the user.

We present M[eye]cro an interaction technique to select on-screen objects and navigate menus through the synergistic use of eye-gaze and thumb-to-finger microgestures. Thumb-to-finger microgestures are gestures performed with the thumb of a hand onto the fingers of the same hand. The active body of research on microgestures highlights expected properties including speed, availability and eye-free interaction. Such properties make microgestures a good candidate for multitasking. However, while praised, the state-of-the-art hypothesis stating that microgestures could be beneficial for multitasking has never been quantitatively verified. We study and compare M[eye]cro to a baseline, i.e., a technique based on physical controllers, in a cockpit-based context. This context allows us to design a controlled experiment involving multitasking with low- and high-priority tasks in parallel. Our results show that performances of the two techniques are similar when participants only perform the selection task. However, M[eye]cro tends to yield better time performance when participants additionally need to treat high-priority tasks in parallel. Results also show that M[eye]cro induces less fatigue and is mostly preferred.

The HCI community has a strong and growing interest in shape-changing interfaces (SCIs) that can offer dynamic affordance. In this context, there is an increasing need for HCI researchers and designers to form close relationships with disciplines such as robotics and material science in order to be able to truly harness the state-of-the-art in morphing technologies. To help these synergies arise, we presentMorphino: a card-based toolkit to inspire shape-changing interface designs. Our cards bring together a collection of morphing mechanisms already established in the multidisciplinary literature and illustrate them through familiar examples from nature. We begin by detailing the design of the cards, based on a review of shape-change in nature; then, report on a series of design sessions conducted to demonstrate their usefulness in generating new ideas and in helping end-users gain a better understanding of the possibilities for shape-changing materials.

Touch interactions with current mobile devices have limited expressiveness. Augmenting devices with additional degrees of freedom can add power to the interaction, and several augmentations have been proposed and tested. However, there is still little known about the effects of learning multiple sets of augmented interactions that are mapped to different applications. To better understand whether multiple command mappings can interfere with one another, or affect transfer and retention, we developed a prototype with three pushbuttons on a smartphone case that can be used to provide augmented input to the system. The buttons can be chorded to provide seven possible shortcuts or transient mode switches. We mapped these buttons to three different sets of actions, and carried out a study to see if multiple mappings affect learning and performance, transfer, and retention. Our results show that all of the mappings were quickly learned and there was no reduction in performance with multiple mappings. Transfer to a more realistic task was successful, although with a slight reduction in accuracy. Retention after one week was initially poor, but expert performance was quickly restored. Our work provides new information about the design and use of chorded buttons for augmenting input in mobile interactions.

Touchscreens are the predominant medium for interactions with digital services; however, their current fixed form factor narrows the scope for rich physical interactions by limiting interaction possibilities to a single, planar surface. In this paper we introduce the concept of PickCells, a fully re-configurable device concept composed of cells, that breaks the mould of rigid screens and explores a modular system that affords rich sets of tangible interactions and novel across-device relationships. Through a series of co-design activities -- involving HCI experts and potential end-users of such systems -- we synthesised a design space aimed at inspiring future research, giving researchers and designers a framework in which to explore modular screen interactions. The design space we propose unifies existing works on modular touch surfaces under a general framework and broadens horizons by opening up unexplored spaces providing new interaction possibilities. In this paper, we present the PickCells concept, a design space of modular touch surfaces, and propose a toolkit for quick scenario prototyping.

Current haptic feedback techniques on handheld devices are applied to the finger pad or the palm of the user. These state-of-the-art approaches are coarse-grained and tend to be intrusive, rather than subtle. In contrast, we present a new feedback technique that applies stimuli around the periphery of the finger pulp, demonstrating how this can provide rich, nuanced haptic information. We use a reconfigurable haptic device employing a ferromagnetic marble for back-of-the device handheld use, which, for the first time, probes, without instrumenting the user, the periphery of the distal phalanx with localised stimulation. We present the design-space afforded by this new technique and evaluate the human-factors of finger-peripheral touch interaction in a controlled user-study. We report results with marbles of different diameters, speeds and a combination of poking, lateral vibration and patterns; present the resulting design guidelines for finger-periphery haptic feedback; and, illustrate its potential with use case scenarios.

In this paper, we present BEXHI, a new mechanical structure for prototyping expandable and bendable handheld devices. Many research projects have pushed bendable surfaces from prototypes to commercially viable devices. In the meantime, expandable devices have become a topic of interest letting one foresee that such devices are on the horizon. With BEXHI, we provide a structure to explore the combined capabilities of these devices. The structure consists of multiple interweaved units allowing non-porous expandable surfaces to bend. Through an instanciation, we illustrate and discuss that the BEXHI structure allows for the exploration of the combination of both bend and expansion interaction spaces.

Incorporating touchscreen interaction into cockpit flight systems offers several potential advantages to aircraft manufacturers, airlines, and pilots. However, vibration and turbulence are challenges to reliable interaction. We examine the design space for braced touch interaction, which allows users to mechanically stabilise selections by bracing multiple fingers on the touchscreen before completing selection. Our goal is to enable fast and accurate target selection during high levels of vibration, without impeding interaction performance when vibration is absent. Three variant methods of braced touch are evaluated, using doubletap, dwell, or a force threshold in combination with heuristic selection criteria to discriminate intentional selection from concurrent braced contacts. We carried out an experiment to test the performance of these methods in both abstract selection tasks and more realistic flight tasks. The study results confirm that bracing improves performance during vibration, and show that doubletap was the best of the tested methods.

Atomic interactions in touch interfaces, like tap, drag, and flick, are well understood in terms of interaction design, but less is known about their physical performance characteristics. We carried out a study to gather baseline data about finger pitch and roll orientation during atomic touch input actions. Our results show differences in orientation and range for different fingers, hands, and actions, and we analyse the effect of tablet angle. Our data provides designers and researchers with a new resource to better understand what interactions are possible in different settings ( e.g. when using the left or right hand), to design novel interaction techniques that use orientation as input (e.g. using finger tilt as an implicit mode), and to determine whether new sensing techniques are feasible (e.g. using fingerprints for identifying specific finger touches).

Touch interactions are now ubiquitous, but few tools are available to help designers quickly prototype touch interfaces and predict their performance. For rapid prototyping, most applications only support visual design. For predictive modelling, tools such as CogTool generate performance predictions but do not represent touch actions natively and do not allow exploration of different usage contexts. To combine the benefits of rapid visual design tools with underlying predictive models, we developed the Storyboard Empirical Modelling tool (StEM) for exploring and predicting user performance with touch interfaces. StEM provides performance models for mainstream touch actions, based on a large corpus of realistic data. We evaluated StEM in an experiment and compared its predictions to empirical times for several scenarios. The study showed that our predictions are accurate (within 7% of empirical values on average), and that StEM correctly predicted differences between alternative designs. Our tool provides new capabilities for exploring and predicting touch performance, even in the early stages of design.

Text selection on touch devices can be a difficult task for users. Letters and words are often too small to select directly, and the enhanced interaction techniques provided by the OS -- magnifiers, selection handles, and methods for selecting at the character, word, or sentence level -- often lead to as many usability problems as they solve. The introduction of force-sensitive touchscreens has added another enhancement to text selection (using force for different selection modes); however, these modes are difficult to discover and many users continue to struggle with accurate selection. In this paper we report on an investigation of the design of touch-based and force-based text selection mechanisms, and describe two novel text-selection techniques that provide improved discoverability, enhanced visual feedback, and a higher performance ceiling for experienced users. Two evaluations show that one design successfully combined support for novices and experts, was never worse than the standard iOS technique, and was preferred by participants.

HCI researchers lack low latency and robust systems to support the design and development of interaction techniques using finger identification. We developed a low cost prototype using piezo based vibration sensors attached to each finger. By combining the events from an input device with the information from the vibration sensors we demonstrate how to achieve low latency and robust finger identification. Our prototype was evaluated in a controlled experiment, using two keyboards and a touchpad, showing recognition rates of 98.2% for the keyboard and, for the touchpad, 99.7% for single touch and 94.7% for two simultaneous touches. These results were confirmed in an additional laboratory style experiment with ecologically valid tasks. Last we present new interactions techniques made possible using this technology.

The development of robust methods to identify which finger is causing each touch point, called "finger identification," will open up a new input space where interaction designers can associate system actions to different fingers. However, relatively little is known about the performance of specific fingers as single touch points or when used together in a "chord." We present empirical results for accuracy, throughput, and subjective preference gathered in five experiments with 48 participants exploring all 10 fingers and 7 two-finger chords. Based on these results, we develop design guidelines for reasonable target sizes for specific fingers and two-finger chords, and a relative ranking of the suitability of fingers and two-finger chords for common multi-touch tasks. Our work contributes new knowledge regarding specific finger and chord performance and can inform the design of future interaction techniques and interfaces utilizing finger identification.