The sound produced when touching fabrics, like a blanket, often provides information regarding the fabric’s texture properties (e.g., its roughness). Fabric roughness is one of the most important aspects of assessing fabric tactile properties. Prior research has demonstrated that touch-related sounds can alter the perception of textures. However, understanding touch-related sound of digital fabric textures, and how they could convey affective responses remain a challenge. In this study, we mapped digital fabric textures using mid-air haptics stimuli and examined how auditory manipulation influences people’s roughness perception. Through qualitative interviews, participants detailed that while rubbing sounds smoothen fabric texture perception, pure tone sounds of 450Hz and 900Hz accent roughness perception. The rubbing sound of fabric evoked associations with soft-materials and led to more calming experiences. In addition, we discussed how haptic interaction can be extended to multisensory modes, revealing a new perspective of mapping multisensory experiences for digital fabrics.

The rapid development of touchless systems has introduced many innovations in social interaction scenarios in recent years. People now can interact with touchless systems in social applications that are aimed to be used in everyday situations in the future. This accelerated development makes us ask, what will the next generation of touchless systems be like? How can we responsibly develop new touchless technologies in the future? To answer the first question, we brought together 20 experts to ideate, speculate, and evaluate possible touchless applications for social interactions. A total of 48 ideas were generated from two consecutive workshops. Then, to answer the second question, we critically analyzed those ideas through a thematic analysis using a responsible innovation (RI) framework, and identified key ethical considerations to guide developers, practitioners when designing future touchless systems. We argue that the social scenarios described, and the RI framework proposed in this paper are a useful starting point for responsibly designing the next generation of touchless systems.

Ultrasound mid-air haptic technology provides a large space of design possibilities, as one can modulate the ultrasound intensity in a continuous 3D space at a high speed over time. Yet, the need for programming the patterns limits rapid ideation and testing of alternatives. We present Feellustrator, a graphical design tool for quickly creating and editing ultrasound mid-air haptics. With Feellustrator, one can create custom ultrasound patterns, layer or sequence them into complex effects, project them on the user’s hand, and export them for use in external programs (e.g., Unity). To create the tool, we interviewed 13 designers who had from a few months to several years of experience with ultrasound, then derived a set of requirements for supporting ultrasound design. We demonstrate the design power of Feellustrator through example applications and an evaluation with 15 participants. Then, we outline future directions for ultrasound haptic design.

Ultrasound mid-air haptic (UMH) devices are a novel tool for haptic feedback, capable of providing localized vibrotactile stimuli to users at a distance. UMH applications largely rely on generating tactile shape outlines on the users’ skin. Here we investigate how to achieve sensations of continuity or gaps within such two-dimensional curves by studying the perception of pairs of amplitude-modulated focused ultrasound stimuli. On the one hand, we aim to investigate perceptual effects that may arise from providing simultaneous UMH stimuli. On the other hand, we wish to provide perception-based rendering guidelines for generating continuous or discontinuous sensations of tactile shapes. Finally, we hope to contribute toward a measure of the perceptually achievable resolution of UMH interfaces. We performed a user study to identify how far apart two focal points need to be to elicit a perceptual experience of two distinct stimuli separated by a gap. Mean gap detection thresholds were found at 32.3-mm spacing between focal points, but a high within- and between-subject variability was observed. Pairs spaced below 15 mm were consistently (>95%) perceived as a single stimulus, while pairs spaced 45 mm apart were consistently (84%) perceived as two separate stimuli. To investigate the observed variability, we resort to acoustic simulations of the resulting pressure fields. These show a non-linear evolution of actual peak pressure spacing as a function of nominal focal point spacing. Beyond an initial threshold in spacing (between 15 and 18 mm), which we believe to be related to the perceived size of a focal point, the probability of detecting a gap between focal points appears to linearly increase with spacing. Our work highlights physical interactions and perceptual effects to consider when designing or investigating the perception of UMH shapes.

The widespread of audio-visual online communication highlights the importance of enhancing their ability to transmit emotional color and personal experience. Non-verbal biometric cues and signals were recently recognized to convey such missing emotional context when added to virtual interactions. Motivated by that, we present a haptic system allowing for a biometric signal transfer in a fully touchless and seamless way. We utilize camera-based heart-rate signal readings and ultrasonic mid-air haptic technology to affect the audience with a temporal tactile pattern representing the speaker’s heart rate. We assess the usability and engagement enhancement of such a system in two user-studies involving one-way communication, i.e., watching a short emotional video. Our analysis of biometric data and subjective responses hints toward changed values of arousal and valence as well as physiological responses when the haptic feedback was applied in a group of participants. Finally, we outline a further research agenda to confirm our observations with different emotions and communication scenarios.

We present a mid-air haptic rendering method to explore volumetric data in augmented reality (AR) environments. Users directly interact with a volume-rendered hologram using their bare hands. Since the volume rendering method accumulates color along with its transparency, the depth perception of the user’s hand within the hologram is vague when used with only visual feedback. Therefore, to enhance the localization of internal structures within the volume data, we propose an AR system that provides a tactile presence for boundaries and associated information (e.g., texture). Leveraging Low-High (LH) histograms, our system provides boundary information to the user both visually and haptically. Specifically, when the user interacts with the volume data using their hands, the system computes a set of focal points or a set of tactile patterns using a GPGPU-based estimation. Additionally, we developed mid-air haptic rendering methods using amplitude modulation (AM) and spatiotemporal modulation (STM) techniques. Both methods were implemented on HoloLens 2 and could run in real-time. The effectiveness of each proposed method was evaluated with respect to various volume data sets, including synthetic and computed tomography (CT) scan data. Our results show that while both haptic rendering methods produce tangible experiences from the hologram interaction, our spatiotemporal modulation method generates better shape discrimination performance with respect to the amplitude modulation method in the case of multiple region of interests.

Ultrasound mid-air haptic interfaces can display highly reconfigurable vibrotactile shapes in mid-air for human-computer interaction (HCI) applications. The choice of stimulus shape, spatial, temporal and modulation parameters yields a complex design space, yet relatively little is known about the impact of these design choices on perceived stimulus properties. We define the combination of a spatial discretization of an abstract shape and a set of rules for the temporal display order and intensity modulation of the resulting points as a sampling strategy. We developed DOLPHIN, an open-source framework to aid in designing mid-air stimuli. DOLPHIN allows the study of the impact of rendering parameters on perceived stimulus properties. This platform-agnostic framework standardizes stimulus descriptions as a step toward more replicability and easier communication in the field. It enables reproduction of stimuli between perceptual experiments and ensures stimuli used in applications correspond to those evaluated in prior perceptual studies. We validated DOLPHIN’s usability by conducting a user study assessing the impact of sampling strategy design on curvature discrimination for dynamic mid-air haptic stimuli. The Weber fractions for just-noticeable differences (JNDs) in curvature were found to range between 1 and 1.4, yet no significant effect of the number of spatial sampling points on curvature discrimination was found. This result shows a practical use-case for DOLPHIN and provides insight into rendering mid-air haptic curvature.

Wireless power transfer creates new opportunities for interaction with tangible and wearable devices, by freeing designers from the constraints of an integrated power source. We explore the use of focused ultrasound as a means of transferring power to a distal device, transforming passive props into dynamic active objects. We analyse the ability to transfer power from an ultrasound array commonly used for mid-air haptic feedback and investigate the practical challenges of ultrasonic power transfer (e.g., receiving and rectifying energy from sound waves). We also explore the ability to power electronic components and multimodal actuators such as lights, speakers and motors. Finally, we describe exemplar wearable and tangible device prototypes that are activated by UltraPower, illustrating the potential applications of this novel technology.

Ultrasonic mid-air haptic feedback enables the tactile exploration of virtual objects in digital environments. However, an object’s shape and texture is perceived multimodally, commencing before tactile contact is made. Visual cues, such as the spatial distribution of surface elements, play a critical first step in forming an expectation of how a texture should feel. When rendering surface texture virtually, its verisimilitude is dependent on whether these visually inferred prior expectations are experienced during tactile exploration. To that end, our work proposes a method where the visual perception of roughness is integrated into the rendering algorithm of mid-air haptic texture feedback. We develop a machine learning model trained on crowd-sourced visual roughness ratings of texture images from the Penn Haptic Texture Toolkit (HaTT). We establish tactile roughness ratings for different mid-air haptic stimuli and match these ratings to our model’s output, creating an end-to-end automated visuo-haptic rendering algorithm. We validate our approach by conducting a user study to examine the utility of the mid-air haptic feedback. This work can be used to automatically create tactile virtual surfaces where the visual perception of texture roughness guides the design of mid-air haptic feedback.

Mid-air tactile stimulation using ultrasonics has been used in a variety of human computer interfaces in the form of prototypes as well as products. When generating these tactile patterns with mid-air tactile ultrasonic displays, the common approach has been to sample the patterns using the hardware update rate capabilities to their full extent. In the current study we show that the hardware update rate can impact perception, but unexpectedly we find that higher update rates do not improve pattern perception. In a first user study, we highlight the effect of update rate on the perceived strength of a pattern, especially for patterns rendered at slow rate of less than 10 Hz. In a second user study, we identify the evolution of the optimal update rate according to variations in pattern size. Our main results show that update rate should be designated as additional parameter for tactile patterns. We also discuss how the relationships we defined in the current study can be implemented into designer tools so that designers remain oblivious to this additional complexity.