Uncertainty caused by instrument tracking errors affects critical tasks such as surgery assisted by Augmented Reality (AR) guidance. This work investigates whether visualizing such uncertainty can improve task performance, trust, and confidence. We present four visualization techniques: Cone, Circle, Gauge, and Color. A two-part study evaluated these techniques on a surgical drilling task, first with 24 non-professional participants and then with 4 professional surgeons. Results indicate that uncertainty visualization improved drilling accuracy by 24% but increased task time by 76%. It also enhanced user confidence and trust in the system, with Cone and Circle as the most preferred visualizations. Based on our findings, we discuss design recommendations for integrating uncertainty visualization into AR-based surgical systems. This work paves the way for a higher success rate in surgical procedures.

Scrolling is ubiquitous in our daily computing experience. We explore how single-handed microgestures can be used for scrolling. Based on an analysis of the basic components necessary for scrolling, we selected 3 microgestures: Tap, Hold and Drag. Considering both rate and position controls, we designed 4 microgesture-based scrolling techniques adapted to these 3 microgestures. We contrasted these 4 techniques in a laboratory experiment with 24 participants who performed 2 tasks: a reciprocal selection task, where participants scrolled the view to reach and select a target; and a counting task, where participants scrolled the view to count image occurrences. Our results suggest that the technique based on Drag microgestures with rate control is the most effective for scrolling operations, regardless of the task. This work demonstrates that microgestures, with their advantages for frequent everyday tasks, offer a promising approach to continuous and efficient scrolling control.

With the rapid development of microgesture research, numerous recognition devices with different sensors (e.g. force-sensitive, IMU), form factors (e.g. glove, ring) and recognition algorithms (e.g. ad-hoc, random forest) have become available. With this increasing number and diversity of recognition devices, it is important to be able to easily switch between devices and/or modify the set of recognized microgestures. To address these challenges, we propose µPoly, a first microgesture toolkit based on the µGlyph microgesture notation. µPoly takes as input the µGlyph descriptions of elementary microgestures. These inputs are combined to recognize complex microgestures described using µGlyph. µPoly therefore offers a standard for microgesture events. The design of µPoly is based on properties taken from the toolkit literature. To highlight these properties: we present (1) how practitioners are using the toolkit in four scenarios, and report on user testing and (2) how applications are being developed with µPoly.

International audience

International audience

Microgestures, i.e . fast and subtle finger movements, have shown a high potential for ubiquitous interaction. However, work to-date either focuses on grasp contexts (holding an object) or on the free-hand context (no held object). These two contexts influence the microgestures feasibility. Researchers have created sets of microgesture feasible across the entire taxonomy of everyday grasps, called transferable microgestures. However, those sets include a limited number of microgestures as compared to those for the free-hand context, for which microgestures are distinguished according to fine characteristics such as the part of the finger being touched or the number of fingers used. We provide knowledge and methods for identifying and recognizing microgestures that can transfer across contexts. First, we report a study on ergonomics factors that influence the feasibility of a microgesture in a given context. Then, we propose a conceptual model serving as a tool to determine the feasibility of a microgesture in a given context without the need for time-consuming user studies. As expected, not all microgestures were transferable to all considered contexts. Thus, we then expose two different ways of defining a set of microgestures transferable between free-hand and grasping contexts. Finally, we report a user study on recognition factors of a transferable microgesture set. • Provides the results of a study on the feasibility of transferable microgestures. • Defines a set of rules for predictively determining the feasibility of microgestures. • Explores design of transferable microgesture sets for free-hand and grasp contexts. • Empirically demonstrates that different phalanxes can be used in grasp contexts.

We explore the use of vibrotactile haptic cues for representing microgestures. We built a four-axes haptic device for providing vibrotactile cues mapped to all four fingers. We also designed six patterns, inspired by six most commonly studied microgestures. The patterns can be played independently on each axis of the device. We ran an experiment with 36 participants testing three different device locations (fingers, back of the hand, and forearm) for pattern and axis recognition. For all three device locations, participants interpreted the patterns with similar accuracy. We also found that they were better at distinguishing the axes when the device is placed on the fingers. Hand and Forearm device locations remain suitable alternatives but involve a greater trade-off between recognition rate and expressiveness. We report the recognition rates obtained for the different patterns, axes and their combinations per device location. These results per device location are important, as constraints of various kinds, such as hardware, context of use and user activities, influence device location. We discuss this choice of device location by improving literature microgesture-based scenarios with haptic feedback or feedforward.

Hand microgestures are promising for mobile interaction with wearable devices. However, they will not be adopted if practitioners cannot communicate to users the microgestures associated with the commands of their applications. This requires unambiguous representations that simultaneously show the multiple microgestures available to control an application. Using a systematic approach, we evaluate how these representations should be designed and contrast 4 conditions depending on the microgestures (tap-swipe and tap-hold) and fingers (index and index-middle) considered. Based on the results, we design a simultaneous representation of microgestures for a given set of 14 application commands. We then evaluate the usability of the representation for novice users and the suitability of the representation for small screens compared with a baseline. Finally, we formulate 8 recommendations based on the results of all the experiments. In particular, redundant graphical and textual representations of microgestures should only be displayed for novice users.

In the active field of hand microgestures, microgesture descriptions are typically expressed informally and are accompanied by images, leading to ambiguities and contradictions. An important step in moving the field forward is a rigorous basis for precisely describing, comparing, and analyzing microgestures. Towards this goal, we propose µGlyph, a hybrid notation based on a vocabulary of events inspired by finger biomechanics. First, we investigate the expressiveness of µGlyph by building a database of 118 microgestures extracted from the literature. Second, we experimentally explore the usability of µGlyph. Participants correctly read and wrote µGlyph descriptions 90% of the time, as compared to 46% for conventional descriptions. Third we present tools that promote µGlyph usage, including a visual editor with LaTeX export. We finally describe how µGlyph can guide research on designing, developing, and evaluating microgesture interaction. Results demonstrate the strong potential of µGlyph to establish a common ground for microgesture research.

Mid-air pointing is widely used for 3D selection in Mixed Reality but leads to arm fatigue. In a first exploratory experiment we study a two-phase design and compare modalities for each phase: mid-air gestures, eye-gaze and microgestures. Results suggest that eye-gaze and microgestures are good candidates to reduce fatigue and improve interaction speed. We therefore propose two 3D selection techniques: Look&MidAir and Look&Micro. Both techniques include a first phase during which users control a cone directed along their eye-gaze. Using the flexion of their non-dominant hand index finger, users pre-select the objects intersecting this cone. If several objects are pre-selected, a disambiguation phase is performed using direct mid-air touch for Look&MidAir or thumb to finger microgestures for Look&Micro. In a second study, we compare both techniques to the standard raycasting technique. Results show that Look&MidAir and Look&Micro perform similarly. However they are 55% faster, perceived easier to use and are less tiring than the baseline. We discuss how the two techniques could be combined for greater flexibility and for object manipulation after selection.

The representations of microgestures are essentials for researchers presenting their results through academic papers and system designers proposing tutorials to novice users. However, those representations remain disparate and inconsistent. As a first attempt to investigate how to best graphically represent microgestures, we created 21 designs, each depicting static and dynamic versions of 4 commonly used microgestures (tap, swipe, flex and hold). We first studied these designs in a quantitative online experiment with 45 participants. We then conducted a qualitative laboratory experiment in Augmented Reality with 16 participants. Based on the results, we provide design guidelines on which elements of a microgesture should be represented and how. In particular, it is recommended to represent the actuator and the trajectory of a microgesture. Also, although preferred by users, dynamic representations are not considered better than their static counterparts for depicting a microgesture and do not necessarily result in a better user recognition.

We present a set of 8 thumb-to-finger microgestures (TTF µGestures) that can be used as an additional modality to enrich touch interaction in eyes-free situations. TTF µGestures possess characteristics especially suited for people with visual impairment (PVI). They have never been studied specifically for PVI to improve accessibility of touchscreen devices. We studied a set of 33 common TTF µGestures to determine which are feasible and usable without seeing while the index is touching a surface. We found that the constrained position of the hand and the absence of vision prevent participants from being able to efficiently target a specific phalanx.

Extending a standard desktop workstation (i.e. a screen, a mouse, a keyboard) with virtual scenes displayed on an Augmented Reality Head-Mounted Display (AR HMD) offers many identified advantages including limited physical space requirements, very large and flexible display spaces, and 3D stereoscopic views. While the technologies become more mainstream, the remaining open question is how to interact with such hybrid workstations that combine 2D views displayed on a physical monitor and 3D views displayed on a HoloLens. For a selection task, we compared mouse-based interaction (standard for 2D desktop workstations) and direct touch interaction in mid-air (standard for 3D AR) while considering different positions of the 3D scene according to a physical monitor. To extend mouse-based selection to 3D views, we experimentally explored different interaction metaphors where the mouse cursor moves either on a horizontal or a vertical plane in a 3D virtual scene. To check for ecological validity of our results, we conducted an additional study focusing on interaction with a 2D/3D Gapminder dataset visualization. The results show 1) that the mouse-based interaction, as compared to direct touch interaction in mid-air, is easy and efficient, 2) that using a vertical plane placed in front of the 3D virtual scene to mimic the double screen metaphor outperforms other interaction techniques and 3) that flexibility is required to allow users to choose the selection techniques and to position the 3D virtual scene relative to the physical monitor. Based on these results, we derive interaction design guidelines for hybrid workstations.

We present M[eye]cro an interaction technique to select on-screen objects and navigate menus through the synergistic use of eye-gaze and thumb-to-finger microgestures. Thumb-to-finger microgestures are gestures performed with the thumb of a hand onto the fingers of the same hand. The active body of research on microgestures highlights expected properties including speed, availability and eye-free interaction. Such properties make microgestures a good candidate for multitasking. However, while praised, the state-of-the-art hypothesis stating that microgestures could be beneficial for multitasking has never been quantitatively verified. We study and compare M[eye]cro to a baseline, i.e., a technique based on physical controllers, in a cockpit-based context. This context allows us to design a controlled experiment involving multitasking with low- and high-priority tasks in parallel. Our results show that performances of the two techniques are similar when participants only perform the selection task. However, M[eye]cro tends to yield better time performance when participants additionally need to treat high-priority tasks in parallel. Results also show that M[eye]cro induces less fatigue and is mostly preferred.

Current Mixed Reality (MR) Head-Mounted Displays (HMDs) offer a limited Field Of View (FOV) of the mixed environment. Turning the head is thus necessary to visually perceive the virtual objects that are placed within the real world. However, turning the head also means loosing the initial visual context. This limitation is critical in contexts like augmented surgery where surgeons need to visually focus on the operative field. To address this limitation we propose to bring virtual objects/widgets back to the users' FOV instead of forcing the users to turn their head. We carry an initial investigation to demonstrate the approach by designing and evaluating three new menu techniques to first bring the menu back to the users' FOV before selecting an item. Results show that our three menu techniques are 1.5s faster on average than the baseline head-motion menu technique and are largely preferred by participants.

Target expansion techniques facilitate pointing by enlarging the effective sizes of targets. As opposed to the numerous studies on target expansion solely focusing on optimizing pointing, we study the compound task of pointing at a Point of Interest (POI) and then interacting with the POI menu in handheld Augmented Reality (AR). A POI menu in AR has a fixed position because it contains relevant information about its location in the real world. We present two techniques that make the cursor jump to the closest opened POI menu after pointing at a POI. Our experimental results show that 1) for selecting a POI the expansion techniques are 31 % faster than the baseline screen-centered crosshair pointing technique, 2) the expansion techniques with/without a jumping cursor to the closest opened POI menu offer similar performances and 3) Touch relative pointing is preferred by participants because it minimizes physical movements.

This paper contributes a first comparative study of three techniques for selecting 3D objects anchored to the table in tabletop Augmented Reality (AR). The impetus for this study is that touch interaction makes more sense when the targeted objects are anchored to the table. We experimentally compare touch and a mixed (touch+mid-air) techniques with the common direct mid-air technique. The touch and mixed techniques involve a decomposition of the 3D task into a 2D task by touch on the table followed by a 1D task by touch or mid-air interaction. Results show that: (1) The touch and mixed techniques present completion times similar to the mid-air technique and are more accurate than the mid-air technique; (2) The mixed technique defines a good compromise between accuracy of touch interaction and speed of mid-air interaction.

Touchscreens are the predominant medium for interactions with digital services; however, their current fixed form factor narrows the scope for rich physical interactions by limiting interaction possibilities to a single, planar surface. In this paper we introduce the concept of PickCells, a fully re-configurable device concept composed of cells, that breaks the mould of rigid screens and explores a modular system that affords rich sets of tangible interactions and novel across-device relationships. Through a series of co-design activities -- involving HCI experts and potential end-users of such systems -- we synthesised a design space aimed at inspiring future research, giving researchers and designers a framework in which to explore modular screen interactions. The design space we propose unifies existing works on modular touch surfaces under a general framework and broadens horizons by opening up unexplored spaces providing new interaction possibilities. In this paper, we present the PickCells concept, a design space of modular touch surfaces, and propose a toolkit for quick scenario prototyping.

We present WiBend, a system that recognizes bending gestures as the input modalities for interacting on non-instrumented and deformable surfaces using WiFi signals. WiBend takes advantage of off-the-shelf 802.11 (Wi-Fi) devices and Channel State Information (CSI) measurements of packet transmissions when the user is placed and interacting between a Wi-Fi transmitter and a receiver. We have performed extensive user experiments in an instrumented laboratory to obtain data for training the HMM models and for evaluating the precision of WiBend. During the experiments, participants performed 12 distinct bending gestures with three surface sizes, two bending speeds and two different directions. The performance evaluation results show that WiBend can distinguish between 12 bending gestures with a precision of 84% on average.

Navigating Augmented Reality (AR) environments with a handheld device often requires users to access digital contents (i.e. Points of Interests - POIs) associated with physical objects outside the field of view of the device's camera. Halo3D is a technique that displays the location of off-screen POIs as halos (arcs) along the edges of the screen. Halo3D reduces clutter by aggregating POIs but has not been evaluated. The results of a first experiment show that an enhanced version of Halo3D was 18% faster than the focus+context technique AroundPlot* for pointing at a POI, and perceived as 34% less intrusive than the arrow-based technique Arrow2D. The results of a second experiment in more realistic settings reveal that two variants of Halo3D that show the spatial distribution of POIs in clusters (1) enable an effective understanding of the off-screen environment and (2) require less effort than AroundPlot* to find POIs in the environment.

We experimentally compare the performance and usability of tablet-based and see-through head-mounted display (HMD)-based interaction techniques for selecting 3D virtual objects projected on a table. This study is a first step toward a better understanding of the advantages and limitations of these interaction techniques, with the perspective of improving interaction with augmented maps. To this end, we evaluate the performance of 3 interaction techniques in selecting 3D virtual objects in sparse and dense environments: (1) the direct touch interaction with a HMD; (2) the ray-casting interaction with a HMD; and (3) the touch interaction on a tablet. Our results show that the two techniques using a HMD are faster, less physically tiring and preferred by the participants over the tablet. The HMD-based interaction techniques perform equally well but the direct touch technique seems to be less impacted by small targets and occlusion.

Sliders are widely used on mobile devices. Envisioning mobile devices that can dynamically deform to raise tangible controls from the screen surface, tangible sliders offer the benefit of eyes-free interaction. However, reaching for distant values with one hand is problematic: users namely need to change their handgrip, which is not comfortable. To overcome this problem, this paper sets out to experimentally study an extendable tangible slider to support one-handed clutching. The tangible slider's knob extends to maintain the thumb's movement within its comfortable area. We first built a low-fidelity prototype made of a knob long enough to allow clutching. This low-fidelity prototype significantly improves performance when reaching distant targets, as compared to a standard tangible slider. We then built a higher-fidelity prototype, introducing actuation and allowing for a shorter knob. When used for clutching, the knob moves back towards the users' thumb. Experimental results show that the motion of the actuated knob does not interrupt eyes-free interaction during manipulation. In comparison, a graphical extendable slider performed 0.9s slower due to the required visual attention. However, the results suggest that the motion of the actuated knob affects performance, as the higher-fidelity prototype performed 0.6s slower than the low-fidelity prototype.

We present EXHI-bit, a mechanical structure for prototyping unique shape-changing interfaces that can be easily built in a fabrication laboratory. EXHI-bit surfaces consist of inter-weaving units that slide in two dimensions. This assembly enables the creation of unique expandable handheld surfaces with continuous transitions while maintaining the surface flat, rigid, and non-porous. EXHI-bit surfaces can be combined to create 2D and 3D multi-surface objects. In this paper, we demonstrate the versatility and generality of EXHI-bit with user-deformed and self-actuated 1D, 2D, and 3D prototypes employed in an architectural urban planning scenario. We also present vision on the use of expandable tablets in our everyday life from 10 users after having interacted with an EXHI-bit tablet.

Dynamic symbolic in-air hand gestures are an increasingly popular means of interaction with smart environments. However, novices need to know what commands are available and which gesture to execute in order to trigger these commands. We propose to adapt OctoPocus, a 2D gesture guiding system, to the case of 3D. The OctoPocus3D guidance system displays a set of 3D gestures as 3D pipes and allows users to understand how the system processes gesture input. Several feedback and feedforward visual alternatives are proposed in the literature. However, their impact on guidance remains to be evaluated. We report the results of two user experiments that aim at designing OctoPocus3D by exploring these alternatives. The results show that a concurrent feedback, which visually simplifies the 3D scene during the execution of the gesture, increases the recognition rate, but only during the first two repetitions. After the first two repetitions, users achieve the same recognition rate with a terminal feedback (after the execution of the gesture), a concurrent feedback, both or neither. With respect to feedforward, the overall stability of the 3D scene explored through the origin of the pipes during the execution of the gestures does not influence the recognition rate or the execution time. Finally, the results also show that displaying upcoming portions of the gestures allows 8% faster completion times than displaying the complete remaining portions. This indicates that preventing visual clutter of the 3D scene prevails over gesture anticipation.

To enhance pointing tasks, target expansion techniques allocate larger activation areas to targets. We distinguish two basic elements of a target expansion technique: the expansion algorithm and the visual aid on the effective expanded targets. We present a systematic analysis of the relevance of the visual aid provided by (1) existing target expansion techniques and (2) Expansion Lens. The latter is a new continuous technique for acquiring targets. Expansion Lens namely, uses a round area centered on the cursor: the lens. The users can see in the lens the target expanded area boundaries that the lens is hovering over. Expansion Lens serves as a magic lens revealing the underlying expansion algorithm. The design rationale of Expansion Lens is based on a systematic analysis of the relevance of the visual aid according to the three goal-oriented phases of a pointing task namely the starting, transfer and validation phases. Expansion Lens optimizes (1) the transfer phase by providing a simple-shaped visual aid centered on the cursor, and (2) the validation phase regarding error rates, by displaying the target expanded area boundaries. The results of our controlled experiment comparing Expansion Lens with four existing target expansion techniques show that Expansion Lens highlights a good trade-off for performance by being the less-error prone technique and the second fastest technique. The experimental data for each phase of the pointing task also confirm our design approach based on the relevance of the visual aid according to the phase of the pointing task.

In this paper we present the concept of Emergeables -- mobile surfaces that can deform or 'morph' to provide fully-actuated, tangible controls. Our goal in this work is to provide the flexibility of graphical touchscreens, coupled with the affordance and tactile benefits offered by physical widgets. In contrast to previous research in the area of deformable displays, our work focuses on continuous controls (e.g., dials or sliders), and strives for fully-dynamic positioning, providing versatile widgets that can change shape and location depending on the user's needs. We describe the design and implementation of two prototype emergeables built to demonstrate the concept, and present an in-depth evaluation that compares both with a touchscreen alternative. The results show the strong potential of emergeables for on-demand, eyes-free control of continuous parameters, particularly when comparing the accuracy and usability of a high-resolution emergeable to a standard GUI approach. We conclude with a discussion of the level of resolution that is necessary for future emergeables, and suggest how high-resolution versions might be achieved.

We explore the combination of touch modalities with pressure-based modalities for multiscale navigation in bifocal views. We investigate a two-hand mobile configuration in which: 1) The dominant hand is kept free for precise touch interaction at any scale of a bifocal view, and 2) The non-dominant hand is used for holding the device in landscape mode, keeping the thumb free for pressure input for navigation at the context scale. The pressure sensor is fixed to the front bezel. Our investigation of pressure-based modalities involves two design options: control (continuous or discrete) and inertia (with or without). The pressure-based modalities are compared to touch-only modalities: the well-known drag-flick and drag-drop modalities. The results show that continuous pressure-based modality without inertia is 1) the fastest one along with the drag-drop touch modality 2) is preferred by the users and 3) importantly minimizes screen occlusion during a phase that requires navigating a large part of the information space.

Target expansion techniques facilitate the pointing task by enlarging the effective sizes of targets. When the target expansion is applied to both the motor and visual spaces, the visual feedforward mechanism is key: Indeed it provides a visual aid to the user on the effective expanded targets prior to the execution or completion of the pointing task, enabling the user to take full advantage of the target expansion technique. Focusing on feedforward mechanisms, we introduce a design space that allows us to describe, classify and design target expansion techniques. To do so we first introduce and characterize the concept of atomic feedforward mechanism along three design axes. We then describe a target expansion technique as a combination of atomic feedforward mechanisms using a matrix-based notation. We provide an analytical exploration of the design space by classifying existing techniques and by designing six new techniques. We also provide a first experimental exploration of the design space in the context of distant pointing. The experimental protocol includes an innovative target layout for handling non-centroidal target expansion. The results show that feedforward dynamicity increases movement time and decreases subjective usability, while explicit expansion observability efficiently supports error prevention for distant pointing.

On a mobile device, the intuitive Focus+Context layout of a detailed view (focus) and perspective/distorted panels on either side (context) is particularly suitable for maximizing the utilization of the limited available display area. Interacting with such a bifocal view requires both fast access to data in the context view and high precision interaction with data in the detailed focus view. We introduce combined modalities that solve this problem by combining the well-known flick-drag gesture-based precise modality with modalities for fast access to data in the context view. The modalities for fast access to data in the context view include direct touch in the context view as well as navigation based on drag gestures, on tilting the device, on side-pressure inputs or by spatially moving the device (dynamic peephole). Results of a comparison experiment of the combined modalities show that the performance can be analyzed according to a 3-phase model of the task: a focus-targeting phase, a transition phase (modality switch) and a cursor-pointing phase. Moreover modalities of the focus-targeting phase based on a discrete mode of navigation control (direct access, pressure sensors as discrete navigation controller) require a long transition phase: this is mainly due to disorientation induced by the loss of control in movements. This effect is significantly more pronounced than the articulatory time for changing the position of the fingers between the two modalities ("homing" time).

Addressing the challenges of distant pointing, we present the feedforward static targeting assistance technique VTE: Voronoi-based Target Expansion. VTE statically displays all the activation areas by dividing the total screen space into areas such that there is only one target inside each area, also called Voronoi tessellation. The key benefit of VTE is in providing the user with an immediate understanding of the targets' activation boundaries before the pointing task even begins: VTE then provides static targeting assistance for both phases of a pointing task, the ballistic motion and the corrective phase. With the goal of making the environment visually uncluttered, we present a first user study to explore the visual parameters of VTE that affect the performance of the technique. In a second user study focusing on static versus dynamic assistance, we compare VTE with Bubble Ray, a dynamic Voronoi-based targeting assistance technique for distant pointing. Results show that VTE significantly outperforms the dynamic assistance technique and is preferred by users both for ray-casting pointing and relative pointing with a hand-controlled cursor.

This paper presents a study of a method for estimating the position and orientation of a photo-shoot in indoor environments for augmented reality applications. Our proposed localization method is based on registered frame data of virtualized reality models, which are photos with known photo-shoot positions and orientations, and depth data. Because registered frame data are secondary product of modeling process, additional works are not necessary to create registered frame data especially for the localization. In the method, a photo taken by a mobile camera is compared to registered frame data for the localization. Since registered frame data are linked with photo-shoot position, orientation, and depth data, 3D coordinates of each pixel on the photo of registered frame data is available. We conducted experiments with employing five techniques of the estimation for comparative evaluations.

We address the problem of mobile distal selection of physical objects when pointing at them in augmented environments. We focus on the disambiguation step needed when several objects are selected with a rough pointing gesture. A usual disambiguation technique forces the users to switch their focus from the physical world to a list displayed on a handheld device's screen. In this paper, we explore the balance between change of users' focus and performance. We present two novel interaction techniques allowing the users to maintain their focus in the physical world. Both use a cycling mechanism, respectively performed with a wrist rolling gesture for P2Roll or with a finger sliding gesture for P2Slide. A user experiment showed that keeping users' focus in the physical world outperforms techniques that require the users to switch their focus to a digital representation distant from the physical objects, when disambiguating up to 8 objects.

In the context of pervasive environments, multimodal interaction plays a pivotal role because multimodality provides flexibility and naturalness for interaction. The challenge of multimodal interfaces in pervasive environments is then to build reliable and autonomic processing systems able to analyze and understand multiple interaction modalities and reconfigure itself in real-time. Addressing this issue, we have developed an autonomic framework called DynaMo (Dynamic multiMOdality) for the development and runtime management of multimodal interaction in pervasive environments. DynaMo is composed by a specification language dedicated to the multimodality domain and a runtime machine that instantiates these specifications. In this paper, we present the overall architecture of our solution DynaMo that is based on partial interaction models, and how these models are completed at runtime to build multimodal interfaces adapted to the local execution environment.

Heterogeneity and dynamicity of pervasive environments require the construction of flexible multimodal interfaces at run time. In this paper, we present how we use an autonomic approach to build and maintain adaptable input multimodal interfaces in smart building environments. We have developed an autonomic solution relying on partial interaction models specified by interaction designers and developers. The role of the autonomic manager is to build complete interaction techniques based on runtime conditions and in conformity with the predicted models. The sole purpose here is to combine and complete partial models in order to obtain an appropriate multimodal interface. We illustrate our autonomic solution by considering a running example based on an existing application and several input devices.

This paper presents the design and evaluation of the Wavelet menu and its implementation on the iPhone. The Wavelet menu consists of a concentric hierarchical Marking menu using simple gestures. The novice mode, i.e. when the menu is displayed, is well adapted to the limited screen space of handheld devices because the representation of the menu hierarchy is inverted, the deeper submenu being always displayed at the center of the screen. The visual design is based on a stacking metaphor to reinforce the perception of the hierarchy and to help users to quickly understand how the technique works. The menu also supports submenu previsualization, a key property to navigate efficiently in a hierarchy of commands. The quantitative evaluation shows that the Wavelet menu provides an intuitive way for supporting efficient gesture-based navigation. The expert mode, i.e. gesture without waiting for the menu to pop-up, is another key property of the Wavelet menu: By providing stroke shortcuts, the Wavelet favors the selection of frequent commands in expert mode and makes eyes-free selection possible. A user experiment shows that participants are able to select commands, eyes-free, while walking.

Fusion engines are fundamental components of multimodal inter-active systems, to interpret input streams whose meaning can vary according to the context, task, user and time. Other surveys have considered multimodal interactive systems; we focus more closely on the design, specification, construction and evaluation of fusion engines. We first introduce some terminology and set out the major challenges that fusion engines propose to solve. A history of past work in the field of fusion engines is then presented using the BRETAM model. These approaches to fusion are then classified. The classification considers the types of application, the fusion principles and the temporal aspects. Finally, the challenges for future work in the field of fusion engines are set out. These include software frameworks, quantitative evaluation, machine learning and adaptation.

The CARE properties (Complementarity, Assignment, Redundancy and Equivalence) define various forms that multimodal input interaction can take. While Equivalence and Assignment express the availability and respective absence of choice between multiple input modalities for performing a given task, Complementarity and Redundancy describe relationships between modalities and require fusion mechanisms. In this paper we present a summary of the works we have carried using the CARE properties for conceiving and implementing multimodal interaction, as well as a new approach using WoZ components. We present different technical solutions for implementing the Complementarity and Redundancy of modalities with a focus on the temporal aspects of the fusion. Starting from a monolithic fusion mechanism, we then explain our component-based approach and the composition components (i.e., Redundancy and Complementarity components). As a new contribution for exploring design solutions before implementing an adequate fusion mechanism as well as for tuning the temporal aspects of the performed fusion, we introduce Wizard of Oz (WoZ) fusion components. We illustrate the composition components as well as the implemented tools exploiting them using several multimodal systems including a multimodal slide viewer and a multimodal map navigator.

Exploration and navigation in multimedia data hierarchies (e.g., photos, music) are frequent tasks on mobile devices. However, visualization and interaction are impoverished due to the limited size of the screen and the lack of precise input devices. As a result, menus on mobile devices do not provide efficient navigation as compared to many innovative menu techniques proposed for Desktop platforms. In this paper, we present Wavelet, the adaptation of the Wave menu for the navigation in multimedia data on iPhone. Its layout, based on an inverted representation of the hierarchy, is particularly well adapted to mobile devices. Indeed, it guarantees that submenus are always displayed on the screen and it supports efficient navigation by providing previsualization of the submenus.

This paper presents Flower menu, a new type of Marking menu that does not only support straight, but also curved gestures for any of the 8 usual orientations. Flower menus make it possible to put many commands at each menu level and thus to create as large a hierarchy as needed for common applications. Indeed our informal analysis of menu breadth in popular applications shows that a quarter of them have more than 16 items. Flower menus can easily contain 20 items and even more (theoretical maximum of 56 items). Flower menus also support within groups as well as hierarchical groups. They can thus favor breadth organization (within groups) or depth organization (hierarchical groups): as a result, the designers can lay out items in a very flexible way in order to reveal meaningful item groupings. We also investigate the learning performance of the expert mode of Flower menus. A user experiment is presented that compares linear menus (baseline condition), Flower menus and Polygon menus, a variant of Marking menus that supports a breadth of 16 items. Our experiment shows that Flower menus are more efficient than both Polygon and Linear menus for memorizing command activation in expert mode.

Mixed interactive systems seek to smoothly merge physical and digital worlds. In this paper we focus on mixed objects that take part in the interaction. Based on our Mixed Interaction Model, we introduce a new characterization space of the physical and digital properties of a mixed object from an intrinsic viewpoint without taking into account the context of use of the object. The resulting enriched Mixed Interaction Model aims at balancing physical and digital properties in the design process of mixed objects. The model extends and generalizes previous studies on the design of mixed systems and covers existing approaches of mixed systems including tangible user interfaces, augmented reality and augmented virtuality. A mixed system called ORBIS that we developed is used to illustrate the discussion: we highlight how the model informs the design alternatives of ORBIS.

In this paper, we illustrate the OpenInterface Interaction Development Environment (OIDE) that addresses the design and development of multimodal interfaces. Multimodal interaction software development presents a particular challenge because of the ever increasing number of novel interaction devices and modalities that can used for a given interactive application. To demonstrate our graphical OIDE and its underlying approach, we present a multimodal slideshow implemented with our tool.

In this paper we address the problem of the development of multimodal interfaces. We describe a three-dimensional characterization space for software components along with its implementation in a component-based platform for rapidly developing multimodal interfaces. By graphically assembling components, the designer/developer describes the transformation chain from physical devices to tasks and vice-versa. In this context, the key point is to identify generic components that can be reused for different multimodal applications. Nevertheless for flexibility purposes, a mixed approach that enables the designer to use both generic components and tailored components is required. As a consequence, our characterization space includes one axis dedicated to the reusability aspect of a component. The two other axes of our characterization space, respectively depict the role of the component in the data-flow from devices to tasks and the level of specification of the component. We illustrate our three dimensional characterization space as well as the implemented tool based on it using a multimodal map navigator.

With the current trend, we can anticipate that future mobile phones will have ever-increasing computational power and be able to embed several captors/effectors including cameras, GPS, orientation sensors, tactile surfaces and vibro-tactile display. Such powerful mobile platforms enable us to deploy mixed reality systems. Many studies on mobile mixed reality focus on games. In this paper, we describe the deployment and a user study of a mixed reality location-based mobile treasure hunt, Snap2Play[1], using technologies such as place recognition, accelerometers and GPS tracking for enhancing the interaction with the game and therefore the game playability. The game that we deployed and tested is running on an off-the-shelf camera phone.

In the context of natural multimodal dialogue systems, we address the challenging issue of the definition of cooperative answers in an appropriate multimodal form. Highlighting the intertwined relation of multimodal outputs with the content, we focus on the Dialogic strategy component, a component that defines from the set of possible contents to answer a user's request, the content to be presented to the user and its multimodal presentation. The content selection and the presentation allocation managed by the Dialogic strategy component are based on various constraints such as the availability of a modality and the user's preferences. Considering three generic types of dialogue strategies and their corresponding handled types of information as well as three generic types of presentation tasks, we present a first implementation of the Dialogic strategy component based on rules. By providing a graphical interface to configure the component by editing the rules, we show how the component can be easily modified by ergonomists at design time for exploring different solutions. In further work we envision letting the user modify the component at runtime.

The development of natural multimodal dialogue systems remains a very difficult task. The flexibility and naturalness they offer result in an increased complexity that current software tools do not address appropriately. One challenging issue we address here is the generation of cooperative responses in an appropriate multimodal form, highlighting the intertwined relation of content and presentation. We identify a key component, the dialogic strategy component, as a mediator between the natural dialogue management and the multimodal presentation. This component selects the semantic information content to be presented according to various presentation constraints. Constraints include inherent characteristics of modalities, the availability of a modality as well as preferences of the user. Thus the cooperative behaviour of the system could be adapted as could its multimodal behaviour. In this paper, we present the dialogic strategy component and an associated platform to quickly develop output multimodal cooperative responses in order to explore different dialogic strategies.

In this paper we describe NAVRNA, an interactive system that enables biologists or researchers in bioinformatics to visualize, explore and edit RNA molecules. The key characteristics of NAVRNA are (1) to exploit multiple display surfaces (2) to enable the manipulation of both the 2D view of RNA called secondary structure, as well as the 3D view of RNA called tertiary structure while maintaining consistency between the two views, (3) to enable co-located synchronous collaborative manipulation of the RNA structures and (4) to provide two-handed interaction techniques for navigating and editing RNA structures and in particular a two-handed technique for bending the structure.

Mixed reality systems seek to smoothly link the physical and data processing (digital) environments. Although mixed reality systems are becoming more prevalent, we still do not have a clear understanding of this interaction paradigm. Addressing this problem, this article introduces a new interaction model called Mixed Interaction model. It adopts a unified point of view on mixed reality systems by considering the interaction modalities and forms of multimodality that are involved for defining mixed environments. This article presents the model and its foundations. We then study its unifying and descriptive power by comparing it with existing classification schemes. We finally focus on the generative and evaluative power of the Mixed Interaction model by applying it to design and compare alternative interaction techniques in the context of RAZZLE, a mobile mixed reality game for which the goal of the mobile player is to collect digital jigsaw pieces localized in space.

Ribonucleic Acid (RNA) is an important molecule which performs a wide range of functions in biological systems. We present a method for visualizing, exploring and editing RNA molecules. The method relies on the visualization of the 3D tertiary structure of RNA as well as on the automatic extraction and visualization of the 2D secondary structure. The 2D representation is used to navigate in the tertiary structure. The method has been implemented in a multi-surface interactive system, NAVRNA, that allows collaborative analysis of RNA.

The development and the evaluation of multimodal interactive systems on mobile phones remains a difficult task. In this paper we address this problem by describing a component-based approach, called ACICARE, for developing and evaluating multimodal interfaces on mobile phones. ACICARE is dedicated to the overall iterative design process of mobile multimodal interfaces, which consists of cycles of designing, prototyping and evaluation. ACICARE is based on two complementary tools that are combined: ICARE and ACIDU. ICARE is a component-based platform for rapidly developing multimodal interfaces. We adapted the ICARE components to run on mobile phones and we connected them to ACIDU, a probe that gathers customer's usage on mobile phones. By reusing and assembling components, ACICARE enables the rapid development of multimodal interfaces as well as the automatic capture of multimodal usage for in-field evaluations. We illustrate ACICARE using our contact manager system, a multimodal system running on the SPV c500 mobile phone.

In this paper we present a method for automatically testing interactive multimodal systems. The proposed approach was originally dedicated to synchronous programming which is mainly used for real-time systems. Nevertheless, the behaviour of real-time systems, consisting of cycles starting by reading an external input and ending by issuing an output, is to a certain extent similar to the one of interactive systems. So considered, this paper investigates the use of the Lutess testing environment dedicated to synchronous software for automatically testing multimodal interactive (not necessarily real-time) systems. More precisely, we focus on test data generation based on operational profiles. The main benefit of this approach is that it increases the ability to automatically test an interactive system over long input sequences, according to various use profiles.

Although several real multimodal systems have been built, their development still remains a difficult task. In this paper we address this problem of development of multimodal interfaces by describing a component-based approach, called ICARE, for rapidly developing multimodal interfaces. ICARE stands for Interaction-CARE (Complementarity Assignment Redundancy Equivalence). Our component-based approach relies on two types of software components. Firstly ICARE elementary components include Device components and Interaction Language components that enable us to develop pure modalities. The second type of components, called Composition components, define combined usages of modalities. Reusing and assembling ICARE components enable rapid development of multimodal interfaces. We have developed several multimodal systems using ICARE and we illustrate the discussion using one of them: the FACET simulator of the Rafale French military plane cockpit.

In this paper we present ASUR++, a notation for describing, and reasoning about the design of, mobile interactive computer systems that combine physical and digital objects and information: mobile mixed systems. ASUR++ helps a designer to specify the key characteristics of such systems and to focus on the relationship between physical objects and actions and digital information exchanges. Following a brief introduction to the notation, we illustrate its potential usefulness via examples based on the design of an augmented museum gallery. We conclude with a consideration of the integration of ASUR++ into the system development process and its augmentation via associated methods and tools.

In this paper we present the Clover architectural model, a new conceptual architectural model for groupware. Our model results from the combination of the layer approach of Dewan's generic architecture with the functional decomposition of the Clover design model. The Clover design model defines three classes of services that a groupware application may support, namely, production, communication and coordination services. The three classes of services can be found in each functional layer of our model. Our model is illustrated with a working system, the CoVitesse system, its software being organized according to our Clover architectural model.

Our work focuses on the design of interaction techniques for large information spaces. Our goal is not to define yet another visualization technique but to provide insights for the design of such techniques. Our design approach is based on ergonomic criteria that arose from a study of how the user perceives and manipulates a large information space. We then provide design rules that should help the designer in devising an interaction technique that verifies the ergonomic criteria. After the design of the interaction technique, the next step is software design. We establish links between our design rules/ergonomic criteria and the software architecture model. By applying our PAC-Amodeus model, we show how the software architecture model helps to either verify or assess the ergonomic criteria. We therefore adopt a predictive evaluation approach to the design of interaction techniques for large information spaces. We illustrate our design approach and results through our VITESSE system.

This article nqmts our tiection on softsvam amhiteetm modelling for multi-usersystems (or groupware).Fi~w introduce t.k notion of software architecture and make explieit the design steps that most sofl-wam designm in HCI tend to blend in a fizzy way.Building on general concepts and pmetiee fmm main stream soikm engineering, we then present a comparative analysis of the most signifhnt architecture models developed for singleand multi-user systems.We close with the pmentation d PAC*, a new arehitectuml fmmwodc for modelling and designing the sdsvare amhitectm of multi-user systems.PAC* is a motivated combination of existing anAiteetuml models seleeted for the complementarily of their "good properties".These include operational heuristics such as rules for deriving agents in accordance to the task model or criteria for reasoning about replication as well as properties such as support for style heterogeneity, portability, and reusability.

Article Interactive information retrieval systems: from user centered interface design to software design Share on Authors: P. Mulhem Laboratoire CLIPS-IMAG, Grenoble, France Laboratoire CLIPS-IMAG, Grenoble, FranceView Profile , L. Nigay Laboratoire CLIPS-IMAG, Grenoble, France Laboratoire CLIPS-IMAG, Grenoble, FranceView Profile Authors Info & Claims SIGIR '96: Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrievalAugust 1996 Pages 326–334https://doi.org/10.1145/243199.243280Online:18 August 1996Publication History 7citation1,125DownloadsMetricsTotal Citations7Total Downloads1,125Last 12 Months6Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Certain design projects raise difficult user-interface problems that are not easily amenable to designers' intuition or rapid prototyping due to their novelty, conceptual complexity, and the difficulty of conducting appropriate user studies. Interpersonal access control in computer-mediated communication (CMC) systems is just such a problem. We describe a collection of systematic theory-based analyses of a system prototype that inherited its control mechanism from two preexisting systems. We demonstrate that the collective use of system and user modeling techniques provides insight into this complex design problem and enables us to examine the implications of design decisions for users and implementation. The analyses identify a number of weaknesses in the prototype and are used to propose ways of making substantive refinements to improve its simplicity and appropriateness for two tasks: altering one's accessibility and distinguishing between who can make what kinds of connections. We conclude with a discussion of some critical issues that are relevant for CMC systems, and reflect on the process of applying formal human-computer interaction (HCI) techniques in informal, exploratory design contexts.

Multimodal interactive systems support multiple interaction techniques such as the synergistic use of speech and direct manipulation. The flexibility they offer results in an increased complexity that current software tools do not address appropriately. One of the emerging technical problems in multimodal interaction is concerned with the fusion of information produced through distinct interaction techniques. In this article, we present a generic fusion engine that can be embedded in a multi-agent architecture modelling technique. We demonstrate the fruitful symbiosis of our fusion mechanism with PAC-Amodeus, our agentbased conceptual model, and illustrate the applicability of the approach with the implementation of an effective interactive system: MATIS, a Multimodal Airline Travel Information System.