Due to the proliferation of smart wearables, it is now the case that designers can explore novel ways that devices can be used in combination by end-users. In this paper, we explore the gestural input enabled by the combination of smart earbuds coupled with a proximal smartwatch. We identify a consensus set of gestures and a taxonomy of the types of gestures participants create through an elicitation study. In a follow-on study conducted on Amazon's Mechanical Turk, we explore the social acceptability of gestures enabled by watch+earbud gesture capture. While elicited gestures continue to be simple, discrete, in-context actions, we find that elicited input is frequently abstract, varies in size and duration, and is split almost equally between on-body, proximal, and more distant actions. Together, our results provide guidelines for on-body, near-ear, and in-air input using earbuds and a smartwatch to support gesture capture.

While many smartphones now include a waterfall (or curved edge) display, this edge is rarely exploited for input. In this paper, we introduce EdgeMark, a thumb-operated, one-handed, gestural menu to provide rapid and subtle access to commands. In a first study, we assess the movement range of the user during uni-manual input to define a spatial range for our EdgeMark menu. In a second study, we compare three potential EdgeMark designs to a marking menu variant to assess accuracy and performance. Our findings indicate that EdgeMark menus can serve as a reliable, effective mechanism for subtle, one-hand command invocation in modern, waterfall-edged smartphones.

In mobile touchscreebn interaction, an important challenge is to find solutions to balance the size of individual widgets against the number of widgets needed during interaction. In this work, to address display space limitations, we explore the design of invisible off-screen toolbars (ether-toolbars) that leverage computer vision to expand application features by placing widgets adjacent to the display screen. We show how simple computer vision algorithms can be combined with a natural human ability to estimate physical placing to support highly accurate targeting. Our ether-toolbar design promises targeting accuracy approximating on-screen widget accuracy while significantly expanding the interaction space of mobile devices. Through two experiments, we examine off-screen content placement metaphors and off-screen precision of participants accessing these toolbars. From the data of the second experiment, we provide a basic model that reflects how users perceive mobile surroundings for ether-widgets and validate it. We also demonstrate a prototype system consisting of an inexpensive 3D printed mount for a mirror that supports ether-toolbar implementations. Finally, we discuss the implications of our work and potential design extensions that can increase the usability and the utility of ether-toolbars.

We present an interaction technique combining tactile actions and midair pointing to access out-of-reach content on large displays without the need to walk across the display. Users can start through a touch gesture on the display surface and finish midair by pointing to push content away or inversely to retrieve a content. The technique takes advantage of well-known semantics of pointing in human-to-human interaction. These, coupled with the semantics of proximal relations and deictic proxemics make the proposed technique very powerful as it leverages on well-understood human-human interaction modalities. Experimental results show this technique to outperform direct tactile interaction on dragging tasks. From our experience we derive four guidelines for interaction with large-scale displays.

Pervasive computing is a vision that has been an inspiring long-term target for many years now. Interaction techniques that allow one user to efficiently control many screens, or that allow several users to collaborate on one distant screen, are still hot topics, and are often considered as two different questions. Standard approaches require a strong coupling between the physical location of input device, and users. We propose to consider these two questions through the same basic concept, that uncouples physical location and user input, using a mid-air approach. We present the concept of mimetic interaction spaces (MIS), a dynamic user-definition of an imaginary input space thanks to an iconic gesture, that can be used to define mid-air interaction techniques. We describe a participative design user-study, that shows this technique has interesting acceptability and elicit some definition and deletion gestures. We finally describe a design space for MIS-based interaction, and show how such concept may be used for multi-screen control, as well as screen sharing in pervasive environments.