Fog screens are a projection medium for mid-air graphics, they are see-through and reach-through. Fog screens are traditionally generated by laminar flows coming out of linear outlets and thus are usually static. Here, we show that ultrasonic beams create a streaming field of constrained airflow that direct aerosols in a fast and controllable way. We characterize various aerosols under streaming and optical diffusion, and evaluate control methods for obtaining different aerosol shapes. Aerosol columns are raised, moved and pushed in a few hundred milliseconds to serve as a projection medium. Hands and other objects do not disturb the fog columns significantly, allowing for direct interaction. We showcase applications by projecting on multiple columns that can be moved continuously within the display volume, reach-through interactions and mixed-reality for tabletop games.

International audience

This paper proposes a method for generating non-contact sensations using low-frequency sound waves without requiring user instrumentation. This method leverages the fundamental acoustic response of a confined space to produce predictable pressure spatial distributions at low frequencies, called modes. These modes can be used to produce sensations either throughout the body, in localized areas of the body, or within the body. We first validate the location and strength of the modes simulated by acoustic modeling. Next, a perceptual study is conducted to show how different frequencies produce qualitatively different sensations across and within the participants’ bodies. The low-frequency sound offers a new way of delivering non-contact sensations throughout the body. The results indicate a high accuracy for predicting sensations at specific body locations.

A laser pointer is a commonly used device that does not require communication with the display system or modifications on the applications, the presenter can just take a pointer and start using it. When a laser pointer is used on a volumetric display, a line rather than a point appears, making it not suitable for pointing at 3D locations. PointerVol is a modified laser pointer that allows users to point to 3D positions inside a swept volumetric display. We propose two PointerVol implementations based on timing and distance measurements, we evaluate the pointing performance using them. Finally, we present other features such as multi-user pointing, line patterns and a multi-finger wearable. PointerVol is a simple device that can help to popularize volumetric displays, or at least to make them more usable for presentations with true-3D content.

Acoustic levitation has emerged as a promising approach for mid-air displays, by using multiple levitated particles as 3D voxels, cloth and thread props, or high-speed tracer particles, under the promise of creating 3D displays that users can see, hear and feel with their bare eyes, ears and hands. However, interaction with this mid-air content always occurred at a distance, since external objects in the display volume (e.g. user’s hands) can disturb the acoustic fields and make the particles fall. This paper proposes TipTrap, a co-located direct manipulation technique for acoustically levitated particles. TipTrap leverages the reflection of ultrasound on the users’ skin and employs a closed-loop system to create functional acoustic traps 2.1 mm below the fingertips, and addresses its 3 basic stages: selection, manipulation and deselection. We use Finite-Differences Time Domain (FDTD) simulations to explain the principles enabling TipTrap, and explore how finger reflections and user strategies influence the quality of the traps (e.g. approaching direction, orientation and tracking errors), and use these results to design our technique. We then implement the technique, characterizing its performance with a robotic hand setup and finish with an exploration of the ability of TipTrap to manipulate different types of levitated content.

Wireless power transfer creates new opportunities for interaction with tangible and wearable devices, by freeing designers from the constraints of an integrated power source. We explore the use of focused ultrasound as a means of transferring power to a distal device, transforming passive props into dynamic active objects. We analyse the ability to transfer power from an ultrasound array commonly used for mid-air haptic feedback and investigate the practical challenges of ultrasonic power transfer (e.g., receiving and rectifying energy from sound waves). We also explore the ability to power electronic components and multimodal actuators such as lights, speakers and motors. Finally, we describe exemplar wearable and tangible device prototypes that are activated by UltraPower, illustrating the potential applications of this novel technology.

Eating is one of the most multisensory experiences in everyday life. All of our five senses (i.e. taste, smell, vision, hearing and touch) are involved, even if we are not aware of it. However, while multisensory integration has been well studied in psychology, there is not a single platform for testing systematically the effects of different stimuli. This lack of platform results in unresolved design challenges for the design of taste-based immersive experiences. Here, we present LeviSense: the first system designed for multisensory integration in gustatory experiences based on levitated food. Our system enables the systematic exploration of different sensory effects on eating experiences. It also opens up new opportunities for other professionals (e.g., molecular gastronomy chefs) looking for innovative taste-delivery platforms. We describe the design process behind LeviSense and conduct two experiments to test a subset of the crossmodal combinations (i.e., taste and vision, taste and smell). Our results show how different lighting and smell conditions affect the perceived taste intensity, pleasantness, and satisfaction. We discuss how LeviSense creates a new technical, creative, and expressive possibilities in a series of emerging design spaces within Human-Food Interaction.

While pressing can enable a wide variety of interesting applications, most press sensing techniques operate only at close distances and rely on fragile electronics. We present EchoTube, a robust, modular, simple, and inexpensive system for sensing low-resolution press events at a distance. EchoTube works by emitting ultrasonic pulses inside a flexible tube which acts as a waveguide and detecting reflections caused by deformations in the tube. EchoTube is deployable in a wide variety of situations: the flexibility of the tubes allows them to be wrapped around and affixed to irregular objects. Because the electronic elements are located at one end of the tube, EchoTube is robust, able to withstand crushing, impacts, water, and other adverse conditions. In this paper, we detail the design, implementation, and theory behind EchoTube; characterize its performance under different configurations; and present a variety of exemplar applications that illustrate its potential.

LeviProps are tangible structures used to create interactive mid-air experiences. They are composed of an acoustically- transparent lightweight piece of fabric and attached beads that act as levitated anchors. This combination enables real- time 6 Degrees-of-Freedom control of levitated structures which are larger and more diverse than those possible with previous acoustic manipulation techniques. LeviProps can be used as free-form interactive elements and as projection surfaces. We developed an authoring tool to support the creation of LeviProps. Our tool considers the outline of the prop and the user constraints to compute the optimum locations for the anchors (i.e. maximizing trapping forces), increasing prop stability and maximum size. The tool produces a final LeviProp design which can be fabricated following a simple procedure. This paper explains and evaluates our approach and showcases example applications, such as interactive storytelling, games and mid-air displays.

Recent improvements in ultrasound imaging enable new opportunities for hand pose detection using wearable devices. Ultrasound imaging has remained under-explored in the HCI community despite being non-invasive, harmless and capable of imaging internal body parts, with applications including smart-watch interaction, prosthesis control and instrument tuition. In this paper, we compare the performance of different forearm mounting positions for a wearable ultrasonographic device. Location plays a fundamental role in ergonomics and performance since the anatomical features differ among positions. We also investigate the performance decrease due to cross-session position shifts and develop a technique to compensate for this misalignment. Our gesture recognition algorithm combines image processing and neural networks to classify the flexion and extension of 10 discrete hand gestures with an accuracy above 98%. Furthermore, this approach can continuously track individual digit flexion with less than 5% NRMSE, and also differentiate between digit flexion at different joints.

The screen in our laptops or the white surface of a projector screen are common examples of displays that we use in our daily life to visualize information. Here, we propose to use levitated particles above these displays to enrich the presented information. By means of acoustic or magnetic levitation we show that it is possible to move a particle in mid-air in the space above these displays. We present different configurations that are feasible with current technology, as well as use cases. In this demo, the visitors will be able to explore and interact with various levitators integrated with traditional displays.

We present two realizations of TastyFloats, a novel system that uses acoustic levitation to deliver food morsels to the users' tongue. To explore TastyFloats' associated design framework, we first address the technical challenges to successfully levitate and deliver different types of foods on the tongue. We then conduct a user study, assessing the effect of acoustic levitation on users' taste perception, comparing three basic taste stimuli (i.e., sweet, bitter and umami) and three volume sizes of droplets (5μL, 10μL and 20μL). Our results show that users perceive sweet and umami easily, even in minimal quantities, whereas bitter is the least detectable taste, despite its typical association with an unpleasant taste experience. Our results are a first step towards the creation of new culinary experiences and innovative gustatory interfaces.

Gestures have become an important tool for natural interaction with computers and thus several wearables have been developed to detect hand gestures. However, many existing solutions are unsuitable for practical use due to low accuracy, high cost or poor ergonomics. We present SensIR, a bracelet that uses near-infrared sensing to infer hand gestures. The bracelet is composed of pairs of infrared emitters and receivers that are used to measure both the transmission and reflection of light through/off the wrist. SensIR improves the accuracy of existing infrared gesture sensing systems through the key idea of taking measurements with all possible combinations of emitters and receivers. Our study shows that SensIR is capable of detecting 12 discrete gestures with 93.3% accuracy. SensIR has several advantages compared to other systems such as high accuracy, low cost, robustness against bad skin coupling and thin form-factor.

A tool able to generate remote forces would allow us to handle dangerous or fragile materials without contact or occlusions. Acoustic levitation is a suitable technology since it can trap particles in air or water. However, no approach has tried to endow humans with an intertwined way of controlling it. Previously, the acoustic elements were static, had to surround the particles and only translation was possible. Here, we present the basic manoeuvres that can be performed when levitators are attached to our moving hands. A Gauntlet of Levitation and a Sonic Screwdriver are presented with their manoeuvres for capturing, moving, transferring and combining particles. Manoeuvres can be performed manually or assisted by a computer for repeating patterns, stabilization and enhanced accuracy or speed. The presented prototypes still have limited forces but symbolize a milestone in our expectations of future technology.

We present JOLED, a mid-air display for interactive physical visualization using Janus objects as physical voxels. The Janus objects have special surfaces that have two or more asymmetric physical properties at different areas. In JOLED, they are levitated in mid-air and controllably rotated to reveal their different physical properties. We made voxels by coating the hemispheres of expanded polystyrene beads with different materials, and applied a thin patch of titanium dioxide to induce electrostatic charge on them. Transparent indium tin oxide electrodes are used around the levitation volume to create a tailored electric field to control the orientation of the voxels. We propose a novel method to control the angular position of individual voxels in a grid using electrostatic rotation and their 3D position using acoustic levitation. We present a display in which voxels can be flipped independently, and two mid-air physical games with a voxel as the playable character that moves in 3D across other physical structures and rotates to reflect its status in the games. We demonstrate a voxel update speed of 37.8 ms/flip, which is video-rate.

LeviPath is a modular system to levitate objects across 3D paths. It consists of two opposed arrays of transducers that create a standing wave capable of suspending objects in mid-air. To control the standing wave, the system employs a novel algorithm based on combining basic patterns of movement. Our approach allows the control of multiple beads simultaneously along different 3D paths. Due to the patterns and the use of only two opposed arrays, the system is modular and can scale its interaction space by joining several LeviPaths. In this paper, we describe the hardware architecture, the basic patterns of movement and how to combine them to produce 3D path visualisations.

Three input modalities for manipulation techniques in Mobile Augmented Reality have been compared. The first one employs only multi-touch input. The second modality uses the movements of the device. Finally, the third one is a hybrid approach based on a combination of the two previous modalities. A user evaluation (N=12) on a 6 DOF docking task suggests that combining multi-touch input and device movement offers the best results in terms of task completion time and efficiency. Nonetheless, using solely the device is more intuitive and performs worse only in large rotations. Given that mobile devices are increasingly supporting movement tracking, the presented results encourage the addition of device movement as an input modality.

A collage is an artistic composition made by assembling different parts to create a new whole. This procedure can be applied for assembling tridimensional objects. In this paper we present CollARt, a Mobile Augmented Reality application which permits to create 3D photo collages. Virtual pieces are textured with pictures taken with the camera and can be blended with real objects. A preliminary user study (N=12) revealed that participants were able to create interesting works of art. The evaluation also suggested that the possibility of itinerantly mixing virtual pieces with the real world increases creativity.

Social networks have become a massive repository of information used by both applications and games in order to improve their performance and effectiveness, nevertheless all of the possibilities have not yet been explored. In this article we propose a new mechanic, Game Entity Social Mapping (GESM), based on fetching data about the player or his contacts from social networking sites and mapping it over different game entities. The paper also presents a preliminary evaluation and exposes the most promising research lines. At this consortium I hope to receive comments on the most interesting ways of continuing the investigation.

Traditionally, e-Health solutions were located at the point of care (PoC), while the new ubiquitous user-centered paradigm draws on standard-based personal health devices (PHDs). Such devices place strict constraints on computation and battery efficiency that encouraged the International Organization for Standardization/IEEE11073 (X73) standard for medical devices to evolve from X73PoC to X73PHD. In this context, low-voltage low-power (LV-LP) technologies meet the restrictions of X73PHD-compliant devices. Since X73PHD does not approach the software architecture, the accomplishment of an efficient design falls directly on the software developer. Therefore, computational and battery performance of such LV-LP-constrained devices can even be outperformed through an efficient X73PHD implementation design. In this context, this paper proposes a new methodology to implement X73PHD into microcontroller-based platforms with LV-LP constraints. Such implementation methodology has been developed through a patterns-based approach and applied to a number of X73PHD-compliant agents (including weighing scale, blood pressure monitor, and thermometer specializations) and microprocessor architectures (8, 16, and 32 bits) as a proof of concept. As a reference, the results obtained in the weighing scale guarantee all features of X73PHD running over a microcontroller architecture based on ARM7TDMI requiring only 168 B of RAM and 2546 B of flash memory.