Foldable phones naturally support displaying two applications side by side on adjacent screens. However, one-handed interaction with this dual-screen configuration faces two challenges: first, users must change grip to reach each screen; second, menus displayed on each screen take valuable screen real estate and can be difficult to reach. To address this, we investigated how the hinge of foldable phones can enrich tactile interaction and provide access to hinge menus, unifying interaction across both screens. In a first experiment, we examined how users hold a foldable phone at different fold angles and identified the thumb-accessible screen areas, validating that a hinge-based grip enhances reachability. In two subsequent experiments, we evaluated the feasibility and performance of hinge gestures, defined as touch inputs (tap, tap-tap or swipe) executed fully or partially on the hinge. Building on these findings, we designed hinge menus that combine hinge gestures for menu activation and item selection. Our final experiment identifies different hinge menus that outperform linear menus on adjacent screens. Our findings provide practical guidelines that can immediately inform and improve interaction on current foldable phones.

Computer-mediated collaboration often relies on symmetrical interactions between users, where all the collaborators use identical devices. However, in some cases, either due to constraints (e.g. users in different environments) or by choice (e.g. using devices with different properties), users engage in asymmetrical interactions. Addressing such asymmetries in heterogeneous systems can be difficult as there has been no systematic analysis of how to define them, or their impact on collaboration. In this paper, we characterize the asymmetries that can arise between users’ interactions within collaborative heterogeneous systems. To this end, we conduct a systematic literature review of asymmetric collaborative systems, coding their properties, including the interaction spaces, their input and output modalities, and shared feedback. We then define the dimensions of asymmetry that emerge from this review. We discuss their impact on collaboration and outline a set of challenges and opportunities for future research.

Recent studies in situated augmented reality have explored interactive urban physical 3D models providing a physical anchor and context to data analysis. See-through Head-Mounted Displays (HMDs) enhance these models by displaying virtual data around the buildings. However, the optimal positioning of multiple information relative to the physical referents is crucial yet has barely been explored. In this paper, we focus on arranging multiple data windows around a district model and present a design space describing the window layouts and linking techniques between the virtual data and its physical referent. We conduct three studies to evaluate these design dimensions for two fundamental visual search tasks: finding a data window for a given physical referent, and vice-versa. Our results reveal novel findings on the best combinations of window layout and link in terms of performance, mental workload and user preference. Finally, we propose a set of design guidelines for presenting situated immersive data around a physical urban model.

Embedding data into the physical environment using augmented reality (AR) is a practical approach for data visualization as it offers a large and flexible display space on or around the physical referent, i.e., the physical object to which the data is related. Yet, current interaction in such context is often performed using cumbersome dedicated devices, tiring mid-air gestures, or awkward on-body input. In this article, we investigate the use of the physical referent itself as a support for input interaction with an embedded space-time cube (STC) representation. Hence, we first identify the most promising mappings between the physical features of the referent (edges, faces, corners) and the STC dimensions. Then, we design three data selection techniques using the physical referent and compare them to mid-air gestures when performing selection tasks on the STC. Overall, our work demonstrates that using the physical referent to support input interaction with embedded data representations is an efficient and comfortable approach for data selection in standing and sitting situations.

See-through Head-Mounted Displays (HMDs) offer interesting opportunities to augment the interaction space around screens, especially around horizontal tabletops. In such context, HMDs can display surrounding vertical virtual windows to complement the tabletop content with data displayed in close vicinity. However, the effects of such combination on the visual acquisition of targets in the resulting combined display space have scarcely been explored. In this paper we conduct a study to explore visual acquisitions in such contexts, with a specific focus on the analysis of visual transitions between the horizontal tabletop display and the vertical virtual displays (in front and on the side of the tabletop). To further study the possible visual perception of the tabletop content out of the HMD and its impact on visual interaction, we distinguished two solutions for displaying information on the horizontal tabletop: using the see-through HMD to display virtual content over the tabletop surface (virtual overlay), i.e. the content is only visible inside the HMD’s FoV, or using the tabletop itself (tabletop screen). 12 participants performed visual acquisition tasks involving the horizontal and vertical displays. We measured the time to perform the task, the head movements, the portions of the displays visible in the HMD’s field of view, the physical fatigue and the user’s preference. Our results show that it is faster to acquire virtual targets in the front display than on the side. Results reveal that the use of the virtual overlay on the tabletop slows down the visual acquisition compared to the use of the tabletop screen, showing that users exploit the visual perception of the tabletop content on the peripheral visual space. We were also able to quantify when and to which extent targets on the tabletop can be acquired without being visible within the HMD's field of view when using the tabletop screen, i.e. by looking under the HMD. These results lead to design recommendations for more efficient, comfortable and integrated interfaces combining tabletop and surrounding vertical virtual displays.

International audience

Inefficient menu interfaces lead to system and application commands being tedious to execute in Immersive Environments. HoloBar is a novel approach to ease the interaction with multi-level menus in immersive environments: with HoloBar, the hierarchical menu splits between the field of view (FoV) of the Head Mounted Display and the smartphone (SP). Command execution is based on around-the-FoV interaction with the SP, and touch input on the SP display. The HoloBar offers a unique combination of features, namely rapid mid-air activation, implicit selection of top-level items and preview of second-level items on the SP, ensuring rapid access to commands. In a first study we validate its activation method, which consists in bringing the SP within an activation distance from the FoV. In a second study, we compare the HoloBar to two alternatives, including the standard HoloLens menu. Results show that the HoloBar shortens each step of a multi-level menu interaction (menu activation, top-level item selection, second-level item selection and validation), with a high success rate. A follow-up study confirms that these results remain valid when compared with the two validation mechanisms of HoloLens (Air-Tap and clicker).

The design of immersive interaction for mixed reality based on head-mounted displays (HMDs), hereafter referred to as Mixed Reality (MR), is still a tedious task which can hinder the advent of such devices. Indeed, the effects of the interface design on task performance are difficult to anticipate during the design phase: the spatial layout of virtual objects and the interaction techniques used to select those objects can have an impact on task completion time. Besides, testing such interfaces with users in controlled experiments requires considerable time and efforts. To overcome this problem, predictive models, such as the Keystroke-Level Model (KLM), can be used to predict the time required to complete an interactive task at an early stage of the design process. However, so far these models have not been properly extended to address the specific interaction techniques of MR environments. In this paper we propose an extension of the KLM model to interaction performed in MR. First, we propose new operators and experimentally determine the unit times for each of them with a HoloLens v1. Then, we perform experiments based on realistic interaction scenarios to consolidate our model. These experiments confirm the validity of our extension of KLM to predict interaction time in mixed reality environments..

In this paper we propose a conceptual view on the modularization of multi-device interactive systems. Our view is supported by the re-visitation and adaptation of some software engineering concepts. We provide the definition of interaction module and its interface and discuss how these concepts contribute to the design of such systems.

This paper presents Rolling-Menu, a technique for selecting toolbar items, based on the use of roll gestures with a multidimensional device, the Roly-Poly Mouse (RPM). Rolling-Menu reduces object-command transition, resulting in a better integration between command selection and direct manipulation of application objects. Selecting a toolbar item with Rolling-Menu requires rolling RPM in a predefined direction corresponding to the item. We propose a design space of Rolling-Menu that includes different roll mapping and validation modes. A first user's study, with a simple toolbar containing up to 14 items, establishes that the best version of Rolling-Menu takes, on average, up to 29% less time than the Mouse to select a toolbar item. Moreover accuracy of the selection with Rolling-Menu is above 90%. Both the validation mode and the mapping between roll direction and toolbar items influence the performance of Rolling-Menus. A second study compares the three best versions of Rolling-Menu with the Mouse to select an item in two types of multidimensional toolbars: a toolbar containing dropdown lists, and a grid toolbar. Results confirm the advantage of Rolling-Menu over a Mouse.

The rapid evolution of multi-display environments (MDEs) has created a vacuum in need of novel input devices to optimize interaction in MDEs. In this paper, we propose TDome, a novel touch-enabled 6DOF input and output device to facilitate interactions in MDEs. TDome offers a private display as output, and multiple degrees of freedom as input by combining touch gestures on the display with physical rotation, roll and translation manipulations of the device. TDome allows versatile interactions that address major MDE tasks, which we illustrate through various proof-of-concept implementations: detect surrounding displays, select one display, transfer data across displays, reach distant displays and perform private interactions. We explore TDome's usability and suitability for MDEs through three user studies. First we explore combined physical+touch gestures from which we discard uncomfortable combinations. We experimentally validate their feasibility and come up with a set of 71 combined gestures that are comfortable and ensure a high success rate, i.e. that can be easily performed and efficiently detected. Finally, we collect user feedback to identify natural mappings between gestures and MDE interactions.

Numerous interaction devices are designed through device composition. However, there is no conceptual support for this process and designers are left out to explore the space of combinations in an ad-hoc manner. In this paper we propose a design space for device composition, DECO, which focuses on the physical aspects of the composition. This design space is built around two main axes, namely Physical arrangement, which describes how elements are physically combined, and Physical manipulation, which describes how users manipulate each element. We first classify existing devices using our design space and then compare four of them to illustrate their similarities and differences. Using DECO, we design a new compound device: RPM2. This device is based on the combination of a regular mouse with the Roly-Poly Mouse. We describe in detail the user-centered iterative design process that leads to the final prototype. Finally we propose a set of design guidelines to assist the use of DECO for device composition.

While several techniques offer more than one detailed view in Overview+Detail (O+D) interfaces, the optimal number of detailed views has not been investigated. But the answer is not trivial: using a single detailed view offers a larger display size but only allows a sequential exploration of the overview; using several detailed views reduces the size of each view but allows a parallel exploration of the overview. In this paper we investigate the benefits of splitting the detailed view in O+D interfaces for working with very large graphs. We implemented an O+D interface where the overview is displayed on a large screen while 1, 2 or 4 split views are displayed on a tactile tablet. We experimentally evaluated the effect of the number of split views according to the number of nodes to connect. Using 4 split views is better than 1 and 2 for working on more than 2 nodes.

We present the design and evaluation of the Roly-Poly Mouse (RPM), a rolling input device that combines the advantages of the mouse (position displacement) and of 3D devices (roll and rotation) to unify 2D and 3D interaction. Our first study explores RPM gesture amplitude and stability for different upper shapes (Hemispherical, Convex) and hand postures. 8 roll directions can be performed precisely and their amplitude is larger on Hemispherical RPM. As minor rolls affect translation, we propose a roll correction algorithm to support stable 2D pointing with RPM. We propose the use of compound gestures for 3D pointing and docking, and evaluate them against a commercial 3D device, the SpaceMouse. Our studies reveal that RPM performs 31% faster than the SpaceMouse for 3D pointing and equivalently for 3D rotation. Finally, we present a proof-of-concept integrated RPM prototype along with discussion on the various technical challenges to overcome to build a final integrated version of RPM.

As public displays integrate 3D content, Overview+Detail (O+D) interfaces on mobile devices will allow for a personal 3D exploration of the public display. In this paper we study the properties of mobile-based interaction with O+D interfaces on 3D public displays. We evaluate three types of existing interaction techniques for the 3D translation of the Detail view: touchscreen input, mid-air movement of the mobile device (Mid-Air Phone) and mid-air movement of the hand around the device (Mid-Air Hand). In a first experiment, we compare the performance and user preference of these three types of techniques with previous training. In a second experiment, we study how well the two mid-air techniques perform with no training or human help to imitate usual conditions in public context. Results reveal that Mid-Air Phone and Hand perform best with training. However, without training or human help Mid-Air Phone is more intuitive and performs better on the first trial. Interestingly, on both experiments users preferred Mid-Air Hand. We conclude with a discussion on the use of mobile devices to interact with public O+D interfaces.

In this paper we discuss metaphor modeling and interpretation in human computer interaction. We aim at evaluating metaphors in Tangible User Interfaces by matching entities and operations between the two domains of the metaphor: the physical interaction domain and the digital domain. A case study in tangible interaction is analyzed and a structured definition of interaction metaphor is derived and evaluated.

Development of mixed reality systems is almost always following an ad-hoc process. The development cycle often turns out to be highly expensive and time consuming. This paper presents a new prototyping approach: a combination of model-based design and simulation based prototyping.

This IUI'04 workshop is an opportunity to identify and articulate the key research challenges for the design and engineering of mixed reality systems. By clarifying and systematizing these challenges, we can improve our own understanding of the current state of the field, stimulate research activity, especially collaboration, and help establish the design of MR systems as a distinct research area.

In this paper we present ASUR++, a notation for describing, and reasoning about the design of, mobile interactive computer systems that combine physical and digital objects and information: mobile mixed systems. ASUR++ helps a designer to specify the key characteristics of such systems and to focus on the relationship between physical objects and actions and digital information exchanges. Following a brief introduction to the notation, we illustrate its potential usefulness via examples based on the design of an augmented museum gallery. We conclude with a consideration of the integration of ASUR++ into the system development process and its augmentation via associated methods and tools.