Foldable phones naturally support displaying two applications side by side on adjacent screens. However, one-handed interaction with this dual-screen configuration faces two challenges: first, users must change grip to reach each screen; second, menus displayed on each screen take valuable screen real estate and can be difficult to reach. To address this, we investigated how the hinge of foldable phones can enrich tactile interaction and provide access to hinge menus, unifying interaction across both screens. In a first experiment, we examined how users hold a foldable phone at different fold angles and identified the thumb-accessible screen areas, validating that a hinge-based grip enhances reachability. In two subsequent experiments, we evaluated the feasibility and performance of hinge gestures, defined as touch inputs (tap, tap-tap or swipe) executed fully or partially on the hinge. Building on these findings, we designed hinge menus that combine hinge gestures for menu activation and item selection. Our final experiment identifies different hinge menus that outperform linear menus on adjacent screens. Our findings provide practical guidelines that can immediately inform and improve interaction on current foldable phones.

Computer-mediated collaboration often relies on symmetrical interactions between users, where all the collaborators use identical devices. However, in some cases, either due to constraints (e.g. users in different environments) or by choice (e.g. using devices with different properties), users engage in asymmetrical interactions. Addressing such asymmetries in heterogeneous systems can be difficult as there has been no systematic analysis of how to define them, or their impact on collaboration. In this paper, we characterize the asymmetries that can arise between users’ interactions within collaborative heterogeneous systems. To this end, we conduct a systematic literature review of asymmetric collaborative systems, coding their properties, including the interaction spaces, their input and output modalities, and shared feedback. We then define the dimensions of asymmetry that emerge from this review. We discuss their impact on collaboration and outline a set of challenges and opportunities for future research.

Recent studies in situated augmented reality have explored interactive urban physical 3D models providing a physical anchor and context to data analysis. See-through Head-Mounted Displays (HMDs) enhance these models by displaying virtual data around the buildings. However, the optimal positioning of multiple information relative to the physical referents is crucial yet has barely been explored. In this paper, we focus on arranging multiple data windows around a district model and present a design space describing the window layouts and linking techniques between the virtual data and its physical referent. We conduct three studies to evaluate these design dimensions for two fundamental visual search tasks: finding a data window for a given physical referent, and vice-versa. Our results reveal novel findings on the best combinations of window layout and link in terms of performance, mental workload and user preference. Finally, we propose a set of design guidelines for presenting situated immersive data around a physical urban model.

We extend grasp models to encompass one-handed interaction with arbitrary shaped touchscreen devices. Current models focus on how objects are stably held by external forces. However, with touchscreen devices, we postulate that users do a trade-off between holding securely and exploring interactively. To verify this, we first conducted a qualitative study which asked participants to grasp 3D printed objects while considering its different interactivity. Results of the study confirm our hypothesis and reveal obvious change in postures. To further verify this trade-off and design interactions, we developed a simulation software capable of computing the stability of a grasp and its reachability. We conducted the second study based on the observed predominant grasps to validate our software with a glove. Results also confirm a consistent trade-off between stability and reachability. We conclude by discussing how this research can help designing computational tools focusing on hand-held interactions with arbitrary shaped touchscreen devices.

Self-localization and wayfinding are challenging tasks for people with visual impairments (PVIs), severely impacting independent mobility. Visual “You-are-here” (YAH) maps are useful for assisting local wayfinding of sighted users. They are used to self-localize and display points of interest, landmarks and routes in the surroundings. However, these maps are not always available and rarely accessible to PVIs. Relying on an iterative participatory design process with eight end-users with visual impairments, we created a proof of concept of a mobile audio-vibratory YAH map. Our design is based on either a tablet or a smartphone to ensure a small and portable solution. A user study with ten PVIs showed that the audio-vibratory YAH map that we designed provides the user with a good understanding of the surroundings and wayfinding cues. Surprisingly, the results show that the audio-vibratory YAH map prototype was as usable as the control condition (audio-tactile YAH map with a tactile overlay), with similar user satisfaction and cognitive load. A follow-up field study with two participants showed the effectiveness of the prototype for assisting in crossroad understanding. To conclude, our innovative design of a mobile audio-vibratory YAH map can overcome the portability and printing issues associated with tactile overlays and can be an appropriate solution for assisting the pedestrian navigation of PVIs.

Embedding data into the physical environment using augmented reality (AR) is a practical approach for data visualization as it offers a large and flexible display space on or around the physical referent, i.e., the physical object to which the data is related. Yet, current interaction in such context is often performed using cumbersome dedicated devices, tiring mid-air gestures, or awkward on-body input. In this article, we investigate the use of the physical referent itself as a support for input interaction with an embedded space-time cube (STC) representation. Hence, we first identify the most promising mappings between the physical features of the referent (edges, faces, corners) and the STC dimensions. Then, we design three data selection techniques using the physical referent and compare them to mid-air gestures when performing selection tasks on the STC. Overall, our work demonstrates that using the physical referent to support input interaction with embedded data representations is an efficient and comfortable approach for data selection in standing and sitting situations.

Automatic layout optimisation allows users to arrange augmented reality content in the real-world environment without the need for tedious manual interactions. This optimisation is often based on modelling the intended content placement as constraints, defined as cost functions. Then, applying a cost minimization algorithm leads to a desirable placement. However, such an approach is limited by the lack of user control over the optimisation results. In this paper we explore the concept of user-driven constraints for augmented reality layout optimisation. With our approach users can define and set up their own constraints directly within the real-world environment. We first present a design space composed of three dimensions: the constraints, the regions of interest and the constraint parameters. Then we explore which input gestures can be employed to define the user-driven constraints of our design space through a user elicitation study. Using the results of the study, we propose a holistic system design and implementation demonstrating our user-driven constraints, which we evaluate in a final user study where participants had to create several constraints at the same time to arrange a set of virtual contents.

The understanding of multi-level spatial topologies is a difficult and frequent challenge in people with visual impairments daily life, impacting their independent mobility. Using the tools of the “maker” movement, and following an iterative co-design process with Orientation and Mobility instructors, we created an innovative tool (3D printed interactive model of a train station) for teaching complex spatial knowledge. Then, we did a comparative study with end users between the 3D interactive model that we designed and two 2D interactive tactile maps representing the same location. Our results show that the 3D interactive model is useful and usable, provides better satisfaction and is preferred to 2D tactile maps. In addition, complex spatial notions are better understood with the 3D model. Altogether, these results suggest that the “maker movement” may empower special education teachers with adapted and innovative tools.

Additive manufacturing of silicone is increasingly being explored to complement the traditional molding fabrication technique for Soft Pneumatic Actuators (SPAs). However, the mechanical behavior of SPAs is defined by their 3D form, which leads to prioritizing the SPAs mechanical properties over their aspect. In this paper, we propose a novel SPA fabrication method where the mechanical properties of a silicone part are defined during the fabrication phase rather than the 3D modeling phase, leading to the object's mechanical properties being independent of the object's aspect. This novel SPA fabrication method, named Local Layer Splitting (LLS), consists of local modifications of the printing layer height to integrate stiffness variation, thus generating controlled mechanical deformation when pressured. We discovered that silicone printing layer height impacts the final stiffness of the material, and it could be used to program bending deformation to actuators during printing. We first characterize the effect of the layer height parameters on 3D-printed silicone stiffness with tensile tests. Then, we present a custom slicer we developed to generate G-codes with local layer height variations depending on the x and y positions. We then characterize the bending and force achievable by SPAs made with the LLS process and find that they match those of state-of-the-art SPAs. Finally, we present and discuss how the LLS method impacts the SPAs design by shifting the bending behavior integration from the SPAs 3D conception to their fabrication phase.

Pneumatically operated soft actuators are increasingly researched due to their fabrication simplicity, actuation capabilities, and low production cost. Depending on the Soft Pneumatic Actuator (SPA) objective, its design can be modified to reach new bending angles or increase its actuation strength. However, increasing the abilities of Soft Pneumatic Actuators (SPAs) requires increasing the complexity of their air cavities or using multiple materials with different mechanical stiffness. Both solutions complexify the fabrication of SPAs, reducing their primary benefits of manufacturing simplicity and low production cost. This paper presents a novel additive manufacturing fabrication process incorporating multiple mechanical stiffnesses using a single bicomponent soft material. This process aims to integrate multiple bending angles with multi-channel SPAs without increasing their manufacturing complexity. Our process uses a dynamic modification of the bi-component silicone mix ratio to generate the desired mechanical properties of the material. Modifying the mix ratio allows us to control the material's cure time and mechanical properties, such as its final stiffness. We found that using a single 30 shore-A bi-component silicone, we could achieve several stiffness values with different reticulation times and levels of stickiness. Using these shore ranges and our fabrication process, we built several SPAs. We explored how the printing orientation of the SPAs modifies its bending actuation using our fabrication process to illustrate the capabilities of our approach.

Advanced technologies are increasingly enabling the creation of interactive devices with non-rectangular form-factors but it is currently unclear what alternative form-factors are desirable for end-users. We contribute an understanding of the interplay between the rationale for the form factors of such devices and their interactive content through think-aloud design sessions in which participants could mold devices as they wished using clay. We analysed their qualitative reflections on how the shapes affected interaction. Using thematic analysis, we identified shape features desirable on handheld freeform devices and discuss the particularity of three themes central to the choice of form factors: freeform dexterity, shape features discoverability and shape adaptability (to the task and context). In a second study following the same experimental set-up, we focused on the trade off between dexterity and discoverability and the relation to the concept of affordance. Our work reveals the shape features that impact the most the choice of grasps on freeform devices from which we derive design guidelines for the design of such devices.

See-through Head-Mounted Displays (HMDs) offer interesting opportunities to augment the interaction space around screens, especially around horizontal tabletops. In such context, HMDs can display surrounding vertical virtual windows to complement the tabletop content with data displayed in close vicinity. However, the effects of such combination on the visual acquisition of targets in the resulting combined display space have scarcely been explored. In this paper we conduct a study to explore visual acquisitions in such contexts, with a specific focus on the analysis of visual transitions between the horizontal tabletop display and the vertical virtual displays (in front and on the side of the tabletop). To further study the possible visual perception of the tabletop content out of the HMD and its impact on visual interaction, we distinguished two solutions for displaying information on the horizontal tabletop: using the see-through HMD to display virtual content over the tabletop surface (virtual overlay), i.e. the content is only visible inside the HMD’s FoV, or using the tabletop itself (tabletop screen). 12 participants performed visual acquisition tasks involving the horizontal and vertical displays. We measured the time to perform the task, the head movements, the portions of the displays visible in the HMD’s field of view, the physical fatigue and the user’s preference. Our results show that it is faster to acquire virtual targets in the front display than on the side. Results reveal that the use of the virtual overlay on the tabletop slows down the visual acquisition compared to the use of the tabletop screen, showing that users exploit the visual perception of the tabletop content on the peripheral visual space. We were also able to quantify when and to which extent targets on the tabletop can be acquired without being visible within the HMD's field of view when using the tabletop screen, i.e. by looking under the HMD. These results lead to design recommendations for more efficient, comfortable and integrated interfaces combining tabletop and surrounding vertical virtual displays.

The lockdown period related to the COVID-19 pandemic has had a strong impact on the educational system in general, but more particularly on the special education system. Indeed, in the case of people with visual impairments, the regular tools relying heavily on images and videos were no longer usable. This specific situation highlighted an urgent need to develop tools that are accessible and that can provide solutions for remote teaching with people with VI. However, there is little work on the difficulties that this population encounters when they learn remotely as well as on the current practices of special education teachers. Such a lack of understanding limits the development of remote teaching systems that are adapted. In this paper, we conducted an online survey regarding the practices of 59 professionals giving lessons to pupils with VI, followed by a series of focus groups with special education workers facing teaching issues during the lockdown period. We followed an iterative design process where we designed successive low-fidelity prototypes to drive successive focus groups. We contribute with an analysis of the issues faced by special education teachers in this situation, and a concept to drive the future development of a tool for remote graphic-based teaching with pupils with VI.

Immersive Analytics is a quickly evolving field that unites several areas such as visualisation, immersive environments, and human-computer interaction to support human data analysis with emerging technologies. This research has thrived over the past years with multiple workshops, seminars, and a growing body of publications, spanning several conferences. Given the rapid advancement of interaction technologies and novel application domains, this paper aims toward a broader research agenda to enable widespread adoption. We present 17 key research challenges developed over multiple sessions by a diverse group of 24 international experts, initiated from a virtual scientific workshop at ACM CHI 2020. These challenges aim to coordinate future work by providing a systematic roadmap of current directions and impending hurdles to facilitate productive and effective applications for Immersive Analytics.

International audience

Inefficient menu interfaces lead to system and application commands being tedious to execute in Immersive Environments. HoloBar is a novel approach to ease the interaction with multi-level menus in immersive environments: with HoloBar, the hierarchical menu splits between the field of view (FoV) of the Head Mounted Display and the smartphone (SP). Command execution is based on around-the-FoV interaction with the SP, and touch input on the SP display. The HoloBar offers a unique combination of features, namely rapid mid-air activation, implicit selection of top-level items and preview of second-level items on the SP, ensuring rapid access to commands. In a first study we validate its activation method, which consists in bringing the SP within an activation distance from the FoV. In a second study, we compare the HoloBar to two alternatives, including the standard HoloLens menu. Results show that the HoloBar shortens each step of a multi-level menu interaction (menu activation, top-level item selection, second-level item selection and validation), with a high success rate. A follow-up study confirms that these results remain valid when compared with the two validation mechanisms of HoloLens (Air-Tap and clicker).

Raised-line graphics are tactile documents made for people with visual impairments (VI). Their exploration relies on a complex two-handed behavior. To better understand the cognitive processes underlying this exploration, we proposed a new method based on “tactile fixations”. A tactile fixation occurs when a finger is stationary within a specific spatial and temporal window. It is known that stationary fingers play an active role when exploring tactile graphics, but they have never been defined or studied before. In this study, we first defined the concept of tactile fixation, then we conducted a behavioral study with ten participants with VI in order to assess the role of tactile fixations under different conditions. The results show that tactile fixations vary according to different factors such as the graphic type as well as the involved hand and the aim of the exploration.

The design of immersive interaction for mixed reality based on head-mounted displays (HMDs), hereafter referred to as Mixed Reality (MR), is still a tedious task which can hinder the advent of such devices. Indeed, the effects of the interface design on task performance are difficult to anticipate during the design phase: the spatial layout of virtual objects and the interaction techniques used to select those objects can have an impact on task completion time. Besides, testing such interfaces with users in controlled experiments requires considerable time and efforts. To overcome this problem, predictive models, such as the Keystroke-Level Model (KLM), can be used to predict the time required to complete an interactive task at an early stage of the design process. However, so far these models have not been properly extended to address the specific interaction techniques of MR environments. In this paper we propose an extension of the KLM model to interaction performed in MR. First, we propose new operators and experimentally determine the unit times for each of them with a HoloLens v1. Then, we perform experiments based on realistic interaction scenarios to consolidate our model. These experiments confirm the validity of our extension of KLM to predict interaction time in mixed reality environments..

Multiple embedded sensors enable smartwatch apps to amass large amounts of interrelated time-series data simultaneously, such as heart rate, oxygen levels or steps walked. Visualizing multiple interlinked datasets is possible on smartphones but remains challenging on small smartwatch displays. We propose a new technique, the Space-Filling Line Graph (SF-LG), that preserves the key visual properties of time-series graphs while making available space on the display to augment such graphs with additional information. Results from our first study (N=30) suggest that, while SF-LG makes available additional space on the small display, it also enables effective (i.e. quick and accurate) comprehension of key line graph tasks. We next implement a greedy algorithm to embed auxiliary information in the most suitable regions on the display. In a second study (N=27), we find that participants are efficient at locating and linking interrelated content using SF-LG in comparison to two baselines approaches. We conclude with guidelines for smartwatch space maximization for visual displays.

Pervasive interfaces can present relevant information anywhere in our environment, and they are thus challenged by the non rectilinearity of the display surface (e.g. circular table) and by the presence of objects that can partially occlude the interface (e.g. a book or cup on the table). To tackle this problem, we propose a novel solution based on two core contributions: the decomposition of the interface into deformable graphical units, called Dynamic Decals, and the control of their position and behaviour by a constraint-based approach. Our approach dynamically deforms the interface when needed while minimizing the impact on its visibility and layout properties. To do so, we extend previous work on implicit deformations to propose and experimentally validate functions defining different decal shapes and new deformers modeling decal deformations when they collide. Then, we interactively optimize the decal placements according to the interface geometry and their interrelations. Relations are modeled as constraints and the interface evolution results from an easy and efficient to solve minimization problem. Our approach is validated by a user study showing that, compared to two baselines, Dynamic decals is an aesthetically pleasant interface that preserves visibility, layout and aesthetic properties.

Addressing the needs of visually impaired people is of continued interest in Human Computer Interaction (HCI) research. Yet, one of the major challenges facing researchers in this field continues to be how to design adequate quantitative empirical evaluation for these users in HCI. In this paper, we analyse a corpus of 178 papers on technologies designed for people with visual impairments, published since 1988, and including at least one quantitative empirical evaluation (243 evaluations in total). To inform future research in this area, we provide an overview, historic trends and a unified terminology to design and report quantitative empirical evaluations. We identify open issues and propose a set of guidelines to address them. Our analysis aims to facilitate and stimulate future research on this topic.

Despite being included in mainstream schools, visually impaired children still face barriers to social engagement and participation. Games could potentially help, but games that cater for both visually impaired and sighted players are scarce. We used a co-design approach to design and evaluate a robot-based educational game that could be inclusive of both visually impaired and sighted children in the context of mainstream education. We ran a focus group discussion with visual impairment educators to understand barriers to inclusive play. And then a series of co-design workshops to engage visually impaired and sighted children and educators in learning about robot technology and exploring its potential to support inclusive play experiences. We present design guidelines and an evaluation workshop of a game prototype, demonstrating group dynamics conducive to collaborative learning experiences, including shared goal setting/execution, closely coupled division of labour, and interaction symmetry.

Visual graphics are widely spread in digital media and are useful in many contexts of daily life. However, access to this type of graphical information remains a challenging task for people with visual impairments (VI). In this study, we designed and evaluated an on-hand vibrotactile interface that enables users with VI to explore digital graphics presented on tablets. We first conducted a set of exploratory tests with both people with VI and blindfolded (BF) people to investigate several design factors. We then conducted a comparative experiment to verify that on-hand vibrotactile cues (indicating direction and progression) can enhance the non-visual exploration of digital graphics. The results based on 12 participants with VI and 12 BF participants confirmed the usability of the technique and revealed that the visual status of the users does not impact graphics identification and comparison tasks.

With upcoming breakthroughs in free-form display technologies, new user interface design challenges have emerged. Here, we investigate a question, which has been widely explored on traditional GUIs but unexplored on non-rectangular interfaces: what are the user strategies in terms of visual search when information is not presented in a traditional rectangular layout? To achieve this, we present two complementary studies investigating eye movements in different visual search tasks. Our results unveil which areas are seen first according to different visual structures. By doing so we address the question of where to place relevant content for the UI designers of non-rectangular displays.

This paper presents Rolling-Menu, a technique for selecting toolbar items, based on the use of roll gestures with a multidimensional device, the Roly-Poly Mouse (RPM). Rolling-Menu reduces object-command transition, resulting in a better integration between command selection and direct manipulation of application objects. Selecting a toolbar item with Rolling-Menu requires rolling RPM in a predefined direction corresponding to the item. We propose a design space of Rolling-Menu that includes different roll mapping and validation modes. A first user's study, with a simple toolbar containing up to 14 items, establishes that the best version of Rolling-Menu takes, on average, up to 29% less time than the Mouse to select a toolbar item. Moreover accuracy of the selection with Rolling-Menu is above 90%. Both the validation mode and the mapping between roll direction and toolbar items influence the performance of Rolling-Menus. A second study compares the three best versions of Rolling-Menu with the Mouse to select an item in two types of multidimensional toolbars: a toolbar containing dropdown lists, and a grid toolbar. Results confirm the advantage of Rolling-Menu over a Mouse.

Raised-line diagrams are widely used by visually impaired (VI) people to read maps, drawings or graphs. While previous work has identified general exploration strategies for raised-line drawings, we have limited knowledge on how this exploration is performed in detail and how it extends to other types of diagrams such as maps or graphs, frequently used in specialized schools. Such information can be crucial for the design of accessible interfaces on touchscreens. We conducted a study in which participants were asked to explore five types of raised-line diagrams (common drawings, perspective drawings, mathematical graphs, neighborhood maps, and geographical maps) while tracking both hands fingers. Relying on a first set of results, we proposed a set of design guidelines for touch interfaces.

The rapid evolution of multi-display environments (MDEs) has created a vacuum in need of novel input devices to optimize interaction in MDEs. In this paper, we propose TDome, a novel touch-enabled 6DOF input and output device to facilitate interactions in MDEs. TDome offers a private display as output, and multiple degrees of freedom as input by combining touch gestures on the display with physical rotation, roll and translation manipulations of the device. TDome allows versatile interactions that address major MDE tasks, which we illustrate through various proof-of-concept implementations: detect surrounding displays, select one display, transfer data across displays, reach distant displays and perform private interactions. We explore TDome's usability and suitability for MDEs through three user studies. First we explore combined physical+touch gestures from which we discard uncomfortable combinations. We experimentally validate their feasibility and come up with a set of 71 combined gestures that are comfortable and ensure a high success rate, i.e. that can be easily performed and efficiently detected. Finally, we collect user feedback to identify natural mappings between gestures and MDE interactions.

Graphical user interfaces are composed of varying elements (text, images, etc.) whose visual arrangement has been relatively well established in the context of rectangular interfaces. The advent of non-rectangular displays questions this knowledge. In this paper we study how traditional content layouts can be adapted to fit different non-rectangular displays. We performed a first qualitative study where graphic designers fitted text and images into different non-rectangular displays. From the analysis of their output we generalize and adapt ten composition principles that have been proposed in the literature for rectangular displays. We evaluate the revised principles through a paired comparison questionnaire where 57 participants compared pairs of layouts. Using the Bradley-Terry-Luce model to analyze our data we show that some results contradict current conventions on visual design for rectangular displays. We then extracted the most interesting cases and conducted a follow up study with additional shapes to investigate how the principles generalize. From these results we propose a set of guidelines for designing visual content for non-rectangular displays.

Numerous interaction devices are designed through device composition. However, there is no conceptual support for this process and designers are left out to explore the space of combinations in an ad-hoc manner. In this paper we propose a design space for device composition, DECO, which focuses on the physical aspects of the composition. This design space is built around two main axes, namely Physical arrangement, which describes how elements are physically combined, and Physical manipulation, which describes how users manipulate each element. We first classify existing devices using our design space and then compare four of them to illustrate their similarities and differences. Using DECO, we design a new compound device: RPM2. This device is based on the combination of a regular mouse with the Roly-Poly Mouse. We describe in detail the user-centered iterative design process that leads to the final prototype. Finally we propose a set of design guidelines to assist the use of DECO for device composition.

Maps are essential in everyday life, but inherently inaccessible to visually impaired users. They must be transcribed to non-editable tactile graphics, or rendered on very expensive shape changing displays. To tackle these issues, we developed a tangible tabletop interface that enables visually impaired users to build tangible maps on their own, using a new type of physical icon called Tangible Reels. Tangible Reels are composed of a sucker pad that ensures stability, with a retractable reel that renders digital lines tangible. In order to construct a map, audio instructions guide the user to precisely place Tangible Reels onto the table and create links between them. During subsequent exploration, the device provides the names of the points and lines that the user touches. A pre-study confirmed that Tangible Reels are stable and easy to manipulate, and that visually impaired users can understand maps that are built with them. A follow-up experiment validated that the designed system, including non-visual interactions, enables visually impaired participants to quickly build and explore maps of various complexities.

Emerging technologies allow for the creation of non-rectangular displays with unlimited constraints in shape. However, the introduction of such displays radically deviates from the prevailing tradition of placing content on rectangular screens and raises fundamental design questions. Among these is the foremost question of how to legibly present text. We address this fundamental concern through a multi-part exploration that includes: (1) a focus-group study from which we collected free-form display scenarios and extracted display shape properties; (2) a framework that identifies different mappings of text onto a non-rectangular shape and formulates hypotheses concerning legibility for different display shape properties; and (3) a series of quantitative text legibility studies to assess our hypotheses. Or results agree with and extend upon other findings in the existing literature on text legibility, but they also uncover unique instances in which different rules need to be applied for non-rectangular displays. These results also provide guidelines for the design of visual interfaces.

Tactile raised-line maps are paper maps widely used by visually impaired people. We designed a mobile technique, based on hand tracking and a smartwatch, in order to leverage pervasive access to virtual maps. We use the smartwatch to render localized text-to-speech and vibratory feedback during hand exploration, but also to provide filtering functions activated by swipe gestures. We conducted a first study to compare the usability of a raised-line map with three virtual maps (plain, with filter, with filter and grid). The results show that virtual maps are usable, and that adding a filter, or a filter and a grid, significantly speeds up data exploration and selection. The results of a following case study showed that visually impaired users were able to achieve a complex task with the device, i.e. finding spatial correlations between two sets of data.

While several techniques offer more than one detailed view in Overview+Detail (O+D) interfaces, the optimal number of detailed views has not been investigated. But the answer is not trivial: using a single detailed view offers a larger display size but only allows a sequential exploration of the overview; using several detailed views reduces the size of each view but allows a parallel exploration of the overview. In this paper we investigate the benefits of splitting the detailed view in O+D interfaces for working with very large graphs. We implemented an O+D interface where the overview is displayed on a large screen while 1, 2 or 4 split views are displayed on a tactile tablet. We experimentally evaluated the effect of the number of split views according to the number of nodes to connect. Using 4 split views is better than 1 and 2 for working on more than 2 nodes.

We present the design and evaluation of the Roly-Poly Mouse (RPM), a rolling input device that combines the advantages of the mouse (position displacement) and of 3D devices (roll and rotation) to unify 2D and 3D interaction. Our first study explores RPM gesture amplitude and stability for different upper shapes (Hemispherical, Convex) and hand postures. 8 roll directions can be performed precisely and their amplitude is larger on Hemispherical RPM. As minor rolls affect translation, we propose a roll correction algorithm to support stable 2D pointing with RPM. We propose the use of compound gestures for 3D pointing and docking, and evaluate them against a commercial 3D device, the SpaceMouse. Our studies reveal that RPM performs 31% faster than the SpaceMouse for 3D pointing and equivalently for 3D rotation. Finally, we present a proof-of-concept integrated RPM prototype along with discussion on the various technical challenges to overcome to build a final integrated version of RPM.

Distributed display environments (DDEs) allow use of various specialized devices but challenge designers to provide a clean flow of data across multiple displays. Upcoming consumer-ready head-worn displays (HWDs) can play a central role in unifying the interaction experience in such ecosystems. In this paper, we report on the design and development of Gluey, a user interface that acts as a 'glue' to facilitate seamless input transitions and data movement across displays. Based on requirements we refine for such an interface, Gluey leverages inherent headworn display attributes such as field-of-view tracking and an always-available canvas to redirect input and migrate content across multiple displays, while minimizing device switching costs. We implemented a functional prototype integrating Gluey's numerous interaction possibilities. From our experience in this integration and from user evaluation results, we identify the open challenges in using HWDs to unify the interaction experience in DDEs.

We propose the use of Hand-to-Face input, a method to interact with head-worn displays (HWDs) that involves contact with the face. We explore Hand-to-Face interaction to find suitable techniques for common mobile tasks. We evaluate this form of interaction with document navigation tasks and examine its social acceptability. In a first study, users identify the cheek and forehead as predominant areas for interaction and agree on gestures for tasks involving continuous input, such as document navigation. These results guide the design of several Hand-to-Face navigation techniques and reveal that gestures performed on the cheek are more efficient and less tiring than interactions directly on the HWD. Initial results on the social acceptability of Hand-to-Face input allow us to further refine our design choices, and reveal unforeseen results: some gestures are considered culturally inappropriate and gender plays a role in selection of specific Hand-to-Face interactions. From our overall results, we provide a set of guidelines for developing effective Hand-to-Face interaction techniques.

As public displays integrate 3D content, Overview+Detail (O+D) interfaces on mobile devices will allow for a personal 3D exploration of the public display. In this paper we study the properties of mobile-based interaction with O+D interfaces on 3D public displays. We evaluate three types of existing interaction techniques for the 3D translation of the Detail view: touchscreen input, mid-air movement of the mobile device (Mid-Air Phone) and mid-air movement of the hand around the device (Mid-Air Hand). In a first experiment, we compare the performance and user preference of these three types of techniques with previous training. In a second experiment, we study how well the two mid-air techniques perform with no training or human help to imitate usual conditions in public context. Results reveal that Mid-Air Phone and Hand perform best with training. However, without training or human help Mid-Air Phone is more intuitive and performs better on the first trial. Interestingly, on both experiments users preferred Mid-Air Hand. We conclude with a discussion on the use of mobile devices to interact with public O+D interfaces.

We present a two-part exploration on mobile true-3D displays, i.e. displaying volumetric 3D content in mid-air. We first identify and study the parameters of a mobile true-3D projection, in terms of the projection's distance to the phone, angle to the phone, display volume and position within the display. We identify suitable parameters and constraints, which we propose as requirements for developing mobile true-3D systems. We build on the first outcomes to explore methods for coordinating the display configurations of the mobile true-3D setup. We explore the resulting design space through two applications: 3D map navigation and 3D interior design. We discuss the implications of our results for the future design of mobile true-3D displays.

We present Bezel-Tap Gestures, a novel family of interaction techniques for immediate interaction on handheld tablets regardless of whether the device is alive or in sleep mode. The technique rests on the close succession of two input events: first a bezel tap, whose detection by accelerometers will awake an idle tablet almost instantly, then a screen contact. Field studies confirmed that the probability of this input sequence occurring by chance is very low, excluding the accidental activation concern. One experiment examined the optimal size of the vocabulary of commands for all four regions of the bezel (top, bottom, left, right). Another experiment evaluated two variants of the technique which both allow two-level selection in a hierarchy of commands, the initial bezel tap being followed by either two screen taps or a screen slide. The data suggests that Bezel-Tap Gestures may serve to design large vocabularies of micro-interactions with a sleeping tablet.

In this paper, we explore the use of movement qualities as interaction modality. The notion of movement qualities is widely used in dance practice and can be understood as how the movement is performed, independently of its specific trajectory in space. We implemented our approach in the context of an artistic installation called A light touch. This installation invites the participant to interact with a moving light spot reacting to the hand movement qualities. We conducted a user experiment that showed that such an interaction based on movement qualities tends to enhance the user experience favouring explorative and expressive usage.

The CARE properties (Complementarity, Assignment, Redundancy and Equivalence) define various forms that multimodal input interaction can take. While Equivalence and Assignment express the availability and respective absence of choice between multiple input modalities for performing a given task, Complementarity and Redundancy describe relationships between modalities and require fusion mechanisms. In this paper we present a summary of the works we have carried using the CARE properties for conceiving and implementing multimodal interaction, as well as a new approach using WoZ components. We present different technical solutions for implementing the Complementarity and Redundancy of modalities with a focus on the temporal aspects of the fusion. Starting from a monolithic fusion mechanism, we then explain our component-based approach and the composition components (i.e., Redundancy and Complementarity components). As a new contribution for exploring design solutions before implementing an adequate fusion mechanism as well as for tuning the temporal aspects of the performed fusion, we introduce Wizard of Oz (WoZ) fusion components. We illustrate the composition components as well as the implemented tools exploiting them using several multimodal systems including a multimodal slide viewer and a multimodal map navigator.

In this paper, we illustrate the OpenInterface Interaction Development Environment (OIDE) that addresses the design and development of multimodal interfaces. Multimodal interaction software development presents a particular challenge because of the ever increasing number of novel interaction devices and modalities that can used for a given interactive application. To demonstrate our graphical OIDE and its underlying approach, we present a multimodal slideshow implemented with our tool.

In this paper we address the problem of the development of multimodal interfaces. We describe a three-dimensional characterization space for software components along with its implementation in a component-based platform for rapidly developing multimodal interfaces. By graphically assembling components, the designer/developer describes the transformation chain from physical devices to tasks and vice-versa. In this context, the key point is to identify generic components that can be reused for different multimodal applications. Nevertheless for flexibility purposes, a mixed approach that enables the designer to use both generic components and tailored components is required. As a consequence, our characterization space includes one axis dedicated to the reusability aspect of a component. The two other axes of our characterization space, respectively depict the role of the component in the data-flow from devices to tasks and the level of specification of the component. We illustrate our three dimensional characterization space as well as the implemented tool based on it using a multimodal map navigator.

The development and the evaluation of multimodal interactive systems on mobile phones remains a difficult task. In this paper we address this problem by describing a component-based approach, called ACICARE, for developing and evaluating multimodal interfaces on mobile phones. ACICARE is dedicated to the overall iterative design process of mobile multimodal interfaces, which consists of cycles of designing, prototyping and evaluation. ACICARE is based on two complementary tools that are combined: ICARE and ACIDU. ICARE is a component-based platform for rapidly developing multimodal interfaces. We adapted the ICARE components to run on mobile phones and we connected them to ACIDU, a probe that gathers customer's usage on mobile phones. By reusing and assembling components, ACICARE enables the rapid development of multimodal interfaces as well as the automatic capture of multimodal usage for in-field evaluations. We illustrate ACICARE using our contact manager system, a multimodal system running on the SPV c500 mobile phone.