Swarm UIs provide assistance to support users in their tasks and are increasingly explored in HCI. This paper studies the extent to which this assistance impacts users’ sense of agency. A reduced sense of agency can lead to non-use of the interface or a diminishing sense of responsibility regarding the consequences of users’ actions. We conduct three experiments studying the impact of three factors on the sense of agency: the level of assistance, the task difficulty, and the predictability of modules. Our nine assistance levels vary in system autonomy and module coordination (proxy vs. no proxy). We find that higher assistance reduces users’ sense of agency, and this effect is not impacted by task difficulty. Predictability only impacts the least assistive interaction techniques. Our results will foster users’ acceptance, responsibility, and use of swarm UIs.

Shape-changing interfaces (SCIs) dynamically alter their form, an inherent characteristic that introduces fragility into their design. As a result, users’ perceptions of an interface’s fragility or its potential to move or break may influence their interaction, however the extent of this effect is unclear. To address this gap, we conducted a qualitative study (N = 18) using video stimuli showcasing 20 existing SCIs. Through thematic analysis, we identified key factors impacting perceived fragility and formalized these into a framework. We then conducted a second study (N = 36) for which we fabricated SCIs that varied across selected fragility-related dimensions. We recorded user interactions and compared how the selected dimensions shaped manipulation of the objects and how they were considered by users. Together, these studies provide a structured foundational understanding of perceived fragility in SCIs and offer insights to enhance perceived robustness and inform future SCI development.

Mobile users have fingernails of different lengths. This paper measures the impact of fingernail length on the use of tactile mobile phones. We first conducted interviews with participants wearing long fingernails. They reported difficulties and non-satisfactory coping strategies to hold their phone securely and acquire targets accurately. We then conducted three experiments comparing different lengths of fingernails (0 mm, 5 mm, and 10 mm). Our results quantify the drop in comfort and efficiency. We measured the range of incidental pitch angle on the surface, the comfortable and useful area of the thumb, and the target acquisition efficiency. 10 mm fingernails consistently decrease by 57% the range of the finger pitch angle, by 36% the comfortable area of the thumb, and by 24% the throughput when acquiring targets. This paper contributes guidelines for future inclusive devices and techniques to also support users with long fingernails.

We explore the design of a watch that can deliver notifications through shape changes, with a specific focus on changes in curvature at the back of the watch face. We explain our design choices and the challenges we faced while creating such a watch. We conducted an experimental study to determine the absolute detection threshold (ADT) of this novel form of feedback. We compared the ADT of two different watches, both of which have a back face that can change its curvature and make contact with the wearer's wrist to notify them. These two watches exhibit different shapes when inflated with high air pressure. To determine the ADT, we conducted a standard two-down, one-up adaptive staircase procedure. Our findings show that an ADT of 3.86 psi is required to inflate the back surface for detection by participants. Overall, our qualitative findings indicate that participants enjoyed this novel type of feedback and could feel different sensations with each watch.

Soft (compliant) curvature-changing UIs provide haptic feedback through changes in softness and curvature. Different softness can impact the deformation of UIs when worn and touched, and thus impact the users’ perception of the curvature. To investigate how softness impacts users’ perception of curvature, we measured participants’ curvature perception accuracy and precision in different softness conditions. We found that participants perceived the curviest surfaces with similar precision in all different softness conditions. Participants lost half the precision of the rigid material when touching the flattest surfaces with the softest material. Participants perceived all curvatures with similar accuracy in all softness conditions. The results of our experiment lay the foundation for soft curvature perception and provide guidelines for the future design of curvature- and softness-changing UIs.

Modular Tangible User Interfaces (TUIs) –i.e., UIs made of small-scale physical modules– offer novel opportunities for tangible interaction thanks to their highly customizable form factor. Such modular TUIs were proposed with different shape of modules and bonding strength between them. The problem we address in this paper is the lack of knowledge of how bonding strength and shape of the modules impact usability. We present the first study exploring the impact of bonding strength and module shape on subjective user ratings when interacting with a magnetic modular prototype. We assessed three levels of bonding strength (low, mid, high) and two shapes (cubes and rounded cubes) in a controlled user study. Participants performed eight common manipulations found in the literature for (non-)modular TUIs. Experimental results showed that (1) cubic modules are overall easier and more satisfying to manipulate, except for precision and bending tasks, (2) low strength impairs UI solidity, but high strength impairs precision tasks with cubic modules.

Shape-changing User Interfaces (UIs) explore the ability of a UI to change its physical shape to support multiple interaction modalities for users’ input and/or system’s output. An approach currently studied to implement such interfaces at a high resolution is based on mm-sized, round, and self-actuated modules. The problem we tackle in this paper is to find the range of usable sizes of such modules, to better inform the trade-off between usability and technological feasibility. We assessed four sliders in a controlled user study: a standard slider and three sliders made of mock-up rounded modules of 1 mm, 2.5 mm, and 5 mm. Experimental results show that (1) 5 mm modules significantly impair performance for the pursuit task and subjective perception for both tasks, (2) performance increases when the size of modules decreases, but (3) users reportedly enjoyed the haptic feedback provided by 1 mm to 2.5 mm modules. These results provide deeper understanding on the impact of the size of modules on performance and subjective perception to inform current technological development of physical user interfaces made of small robotic modules.

Shape-changing User Interfaces attract growing interest in Human-Computer Interaction. Modular robotics offer a great opportunity for their implementation. However, the current theoretical and technical advances of modular robotics are fragmented and little centered on the user. To unify existing work and center future research on the user, we perform a systematic literature review enabling us to build a unifying space for the design of modular shape-changing user interfaces.Our aim is to bridge the gap between HCI and robotics. We relate properties of different domains and identify inconsistencies to structure the design space. Towards this aim, we conduct a thorough cross-disciplinary survey to propose: 1) a set of design properties at the scale of the interface (macro-scale) and at the scale of the modules (micro-scale) and 2) the impact of these properties on each other. This paper can be used to describe and compare existing modular shape-changing UIs and generate new design ideas by building upon knowledge from robotics and HCI.

The HCI community has a strong and growing interest in shape-changing interfaces (SCIs) that can offer dynamic affordance. In this context, there is an increasing need for HCI researchers and designers to form close relationships with disciplines such as robotics and material science in order to be able to truly harness the state-of-the-art in morphing technologies. To help these synergies arise, we presentMorphino: a card-based toolkit to inspire shape-changing interface designs. Our cards bring together a collection of morphing mechanisms already established in the multidisciplinary literature and illustrate them through familiar examples from nature. We begin by detailing the design of the cards, based on a review of shape-change in nature; then, report on a series of design sessions conducted to demonstrate their usefulness in generating new ideas and in helping end-users gain a better understanding of the possibilities for shape-changing materials.

One-handed Back-of-Device (BoD) interaction proved to be desired and sometimes unavoidable with a mobile touchscreen device, for both preferred and non-preferred hands. Although users? two hands are asymmetric, the impact of this asymmetry on the performance of mobile interaction has been little studied so far. Research on one-handed BoD interaction mostly focused on the preferred hand, even though users cannot avoid in real life to handle their phone with their non-preferred hand. To better design one-handed BoD interaction tailored for each hand, the identification and measure of the impact of their asymmetry are critical. In this paper, we study the impact on the performance of the asymmetry between the preferred and the non-preferred hands when interacting with one hand in the back of a mobile touch surface. Empirical data indicates that users' preferred hand performs better than the non-preferred hand in target acquisition tasks, for both time (+10%) and accuracy (+20%). In contrast, for steering tasks, we found little difference in performance between users' preferred and non-preferred hands. These results are useful for the HCI community to design mobile interaction techniques tailored for each hand only when it is necessary. We present implications for research and design directly based on the findings of the study, in particular, to reduce the impact of the asymmetry between hands and improve the performance of both hands for target acquisition.

We investigate the design of a shape-changing dial, i.e. a dial that can change its circumference and height to adapt to different contexts of interaction. We first explore how users grasp 3D printed dials of different heights and circumferences in order to inform the form factor of shape-changing dials. We then design a prototype, ExpanDial, inspired from morphing origami. We then use our prototype as a probe within design sessions and use the participants' feedback to devise a set of applications that can benefit from such reconfigurable devices. We also used the design sessions to better understand what kind of interaction and manipulation could be harnessed from such device.

Touchscreens are the predominant medium for interactions with digital services; however, their current fixed form factor narrows the scope for rich physical interactions by limiting interaction possibilities to a single, planar surface. In this paper we introduce the concept of PickCells, a fully re-configurable device concept composed of cells, that breaks the mould of rigid screens and explores a modular system that affords rich sets of tangible interactions and novel across-device relationships. Through a series of co-design activities -- involving HCI experts and potential end-users of such systems -- we synthesised a design space aimed at inspiring future research, giving researchers and designers a framework in which to explore modular screen interactions. The design space we propose unifies existing works on modular touch surfaces under a general framework and broadens horizons by opening up unexplored spaces providing new interaction possibilities. In this paper, we present the PickCells concept, a design space of modular touch surfaces, and propose a toolkit for quick scenario prototyping.

We present WiBend, a system that recognizes bending gestures as the input modalities for interacting on non-instrumented and deformable surfaces using WiFi signals. WiBend takes advantage of off-the-shelf 802.11 (Wi-Fi) devices and Channel State Information (CSI) measurements of packet transmissions when the user is placed and interacting between a Wi-Fi transmitter and a receiver. We have performed extensive user experiments in an instrumented laboratory to obtain data for training the HMM models and for evaluating the precision of WiBend. During the experiments, participants performed 12 distinct bending gestures with three surface sizes, two bending speeds and two different directions. The performance evaluation results show that WiBend can distinguish between 12 bending gestures with a precision of 84% on average.

Uncertainty is common when working with data and becomes more important as processing big data gains attention. However, no standard tangible interface element exists for inputting uncertain data. In this article, we extend the input space of two traditional TUIs: dial and slider. We present five designs that are based on dials and sliders and support uncertain input. We conduct focus group interviews to evaluate the designs. The interviews allow us to extend existing design requirements for parameter control UIs to support uncertain input.

Sliders are widely used on mobile devices. Envisioning mobile devices that can dynamically deform to raise tangible controls from the screen surface, tangible sliders offer the benefit of eyes-free interaction. However, reaching for distant values with one hand is problematic: users namely need to change their handgrip, which is not comfortable. To overcome this problem, this paper sets out to experimentally study an extendable tangible slider to support one-handed clutching. The tangible slider's knob extends to maintain the thumb's movement within its comfortable area. We first built a low-fidelity prototype made of a knob long enough to allow clutching. This low-fidelity prototype significantly improves performance when reaching distant targets, as compared to a standard tangible slider. We then built a higher-fidelity prototype, introducing actuation and allowing for a shorter knob. When used for clutching, the knob moves back towards the users' thumb. Experimental results show that the motion of the actuated knob does not interrupt eyes-free interaction during manipulation. In comparison, a graphical extendable slider performed 0.9s slower due to the required visual attention. However, the results suggest that the motion of the actuated knob affects performance, as the higher-fidelity prototype performed 0.6s slower than the low-fidelity prototype.

Physical controls are widely used by professionals such as sound engineers or aircraft pilots. In particular knobs and sliders are the most prevalent in such interfaces. They have advantages over touchscreen GUIs, especially when users require quick and eyes-free control. However, their interfaces (e.g., mixing consoles) are often bulky and crowded. To improve this, we present the results of a formative study with professionals who use physical controllers. Based on their feedback, we propose design requirements for future interfaces for parameters control. We then introduce the design of our KnobSlider that combines the advantages of a knob and a slider in one unique shape-changing device. A qualitative study with professionals shows how KnobSlider supports the design requirements, and inspired new interactions and applications.

Users interact with many reconfigurable objects in daily life. These objects embed reconfigurations and shape- changing features that users are familiar with. For this reason, everyday reconfigurable objects have informed the design and taxonomy of shape changing UI. However, they have never been explored systematically. In this paper, we present a data set of 82 everyday reconfigurable objects that we collected in a workshop. We discuss how they can inspire the design of reconfigurable interfaces. We particularly focus on taxonomies of reconfigurable interfaces. Taxonomies have been suggested to help design and communication among researchers, however despite their extensive use, taxonomies are rarely evaluated. This paper analyses two established taxonomies - Rasmussen's and Roudaut's - using daily reconfigurable objects. We show relationships between the taxonomies and area for improvements. We propose Morphees+, a refined taxonomy based on Roudaut's Shape Resolution Taxonomy.

This research forms part of a wider body of work focused around involving emergent users---those just beginning to get access to mobile devices---in the development and refinement of far-future technologies. In this paper we present an evaluation of a new type of deformable slider with emergent users, designed to investigate whether shape-changing interfaces provide any benefit over touchscreens for this type of user. Our trials, which took place in two contexts and three disparate regions, revealed that while there was a clear correlation between performance and technology exposure, emergent users had similar ability with both touchscreen and deformable controls.

Dynamic symbolic in-air hand gestures are an increasingly popular means of interaction with smart environments. However, novices need to know what commands are available and which gesture to execute in order to trigger these commands. We propose to adapt OctoPocus, a 2D gesture guiding system, to the case of 3D. The OctoPocus3D guidance system displays a set of 3D gestures as 3D pipes and allows users to understand how the system processes gesture input. Several feedback and feedforward visual alternatives are proposed in the literature. However, their impact on guidance remains to be evaluated. We report the results of two user experiments that aim at designing OctoPocus3D by exploring these alternatives. The results show that a concurrent feedback, which visually simplifies the 3D scene during the execution of the gesture, increases the recognition rate, but only during the first two repetitions. After the first two repetitions, users achieve the same recognition rate with a terminal feedback (after the execution of the gesture), a concurrent feedback, both or neither. With respect to feedforward, the overall stability of the 3D scene explored through the origin of the pipes during the execution of the gestures does not influence the recognition rate or the execution time. Finally, the results also show that displaying upcoming portions of the gestures allows 8% faster completion times than displaying the complete remaining portions. This indicates that preventing visual clutter of the 3D scene prevails over gesture anticipation.

In this paper we present the concept of Emergeables -- mobile surfaces that can deform or 'morph' to provide fully-actuated, tangible controls. Our goal in this work is to provide the flexibility of graphical touchscreens, coupled with the affordance and tactile benefits offered by physical widgets. In contrast to previous research in the area of deformable displays, our work focuses on continuous controls (e.g., dials or sliders), and strives for fully-dynamic positioning, providing versatile widgets that can change shape and location depending on the user's needs. We describe the design and implementation of two prototype emergeables built to demonstrate the concept, and present an in-depth evaluation that compares both with a touchscreen alternative. The results show the strong potential of emergeables for on-demand, eyes-free control of continuous parameters, particularly when comparing the accuracy and usability of a high-resolution emergeable to a standard GUI approach. We conclude with a discussion of the level of resolution that is necessary for future emergeables, and suggest how high-resolution versions might be achieved.

We address the problem of mobile distal selection of physical objects when pointing at them in augmented environments. We focus on the disambiguation step needed when several objects are selected with a rough pointing gesture. A usual disambiguation technique forces the users to switch their focus from the physical world to a list displayed on a handheld device's screen. In this paper, we explore the balance between change of users' focus and performance. We present two novel interaction techniques allowing the users to maintain their focus in the physical world. Both use a cycling mechanism, respectively performed with a wrist rolling gesture for P2Roll or with a finger sliding gesture for P2Slide. A user experiment showed that keeping users' focus in the physical world outperforms techniques that require the users to switch their focus to a digital representation distant from the physical objects, when disambiguating up to 8 objects.

Only intrusive and expensive ways of precisely expressing emotions has been proposed, which are not likely to appear soon in everyday Ubicomp environments. In this paper, we study to which extent we can identify the emotion a user is explicitly expressing through 2D and 3D gestures. Indeed users already often manipulate mobile devices with touch screen and accelerometers. We conducted a field study where we asked participants to explicitly express their emotion through gestures and to report their affective states. We contribute by (1) showing a high number of significant correlations in 3D motion descriptors of gestures and in the arousal dimension; (2) defining a space of affective gestures. We identify (3) groups of descriptors that structure the space and are related to arousal. Finally, we provide with (4) a preliminary model of arousal and we identify (5) interesting patterns in particular classes of gestures. Such results are useful for Ubicomp application designers in order to envision the use of gestures as a cheap and non-intrusive affective modality.

Mixed interactive systems seek to smoothly merge physical and digital worlds. In this paper we focus on mixed objects that take part in the interaction. Based on our Mixed Interaction Model, we introduce a new characterization space of the physical and digital properties of a mixed object from an intrinsic viewpoint without taking into account the context of use of the object. The resulting enriched Mixed Interaction Model aims at balancing physical and digital properties in the design process of mixed objects. The model extends and generalizes previous studies on the design of mixed systems and covers existing approaches of mixed systems including tangible user interfaces, augmented reality and augmented virtuality. A mixed system called ORBIS that we developed is used to illustrate the discussion: we highlight how the model informs the design alternatives of ORBIS.

With the current trend, we can anticipate that future mobile phones will have ever-increasing computational power and be able to embed several captors/effectors including cameras, GPS, orientation sensors, tactile surfaces and vibro-tactile display. Such powerful mobile platforms enable us to deploy mixed reality systems. Many studies on mobile mixed reality focus on games. In this paper, we describe the deployment and a user study of a mixed reality location-based mobile treasure hunt, Snap2Play[1], using technologies such as place recognition, accelerometers and GPS tracking for enhancing the interaction with the game and therefore the game playability. The game that we deployed and tested is running on an off-the-shelf camera phone.

Mixed reality systems seek to smoothly link the physical and data processing (digital) environments. Although mixed reality systems are becoming more prevalent, we still do not have a clear understanding of this interaction paradigm. Addressing this problem, this article introduces a new interaction model called Mixed Interaction model. It adopts a unified point of view on mixed reality systems by considering the interaction modalities and forms of multimodality that are involved for defining mixed environments. This article presents the model and its foundations. We then study its unifying and descriptive power by comparing it with existing classification schemes. We finally focus on the generative and evaluative power of the Mixed Interaction model by applying it to design and compare alternative interaction techniques in the context of RAZZLE, a mobile mixed reality game for which the goal of the mobile player is to collect digital jigsaw pieces localized in space.