The ability to manipulate and physically feel virtual objects without any real object being present and without equipping the user has been a long-standing goal in virtual reality (VR). Emerging ultrasound mid-air haptics (UMH) technology could potentially address this challenge, as it enables remote tactile stimulation of unequipped users. However, to date, UMH has received limited attention in the field of haptic exploration and manipulation in virtual environments. Existing work has primarily focused on interactions requiring a single hand and thus the delivery of unimanual haptic feedback. Despite being fundamental to a large part of haptic interactions with our environments, bimanual tasks have rarely been studied in the field of UMH interaction in VR. In this paper, we propose the use of non-coplanar mid-air haptic devices for providing simultaneous tactile feedback to both hands during bimanual VR manipulation. We discuss coupling schemes and haptic rendering algorithms for providing bimanual haptic feedback in bimanual interactions with virtual environments. We then present two human participant studies, assessing the benefits of bimanual ultrasound haptic feedback in a two-handed grasping and holding task and in a shape exploration task. Results suggest that the use of multiple non-coplanar UMH devices could be an interesting approach for enriching unencumbered haptic manipulation in virtual environments.

Ultrasound mid-air haptic (UMH) interfaces focus acoustic energy to generate high-pressure focal points in mid-air, creating tactile sensations. When these focal points are modulated over time, they can render tactile patterns with various properties in a 3D workspace. While UMH has been used to render curves and 2D shapes, such as tactile icons, rendering 3D objects in full is nowadays an open problem, as it would require more powerful devices. To overcome this limitation, designers typically only render a representation of the intersection between the user’s hand and the virtual object. However, there is no consensus in the literature on how best to compute this intersection.

Haptic exoskeleton gloves are a widespread solution for providing force-feedback in Virtual Reality (VR), especially for 3D object manipulations. However, they are still lacking an important feature regarding in-hand haptic sensations: the palmar contact. In this paper, we present PalmEx, a novel approach which incorporates palmar force-feedback into exoskeleton gloves to improve the overall grasping sensations and manual haptic interactions in VR. PalmEx's concept is demonstrated through a self-contained hardware system augmenting a hand exoskeleton with an encountered palmar contact interface - physically encountering the users' palm. We build upon current taxonomies to elicit PalmEx's capabilities for both the exploration and manipulation of virtual objects. We first conduct a technical evaluation optimising the delay between the virtual interactions and their physical counterparts. We then empirically evaluate PalmEx's proposed design space in a user study (n=12) to assess the potential of a palmar contact for augmenting an exoskeleton. Results show that PalmEx offers the best rendering capabilities to perform believable grasps in VR. PalmEx highlights the importance of the palmar stimulation, and provides a low-cost solution to augment existing high-end consumer hand exoskeletons.

Rich, informative and realistic haptic feedback is key to enhancing Virtual Reality (VR) manipulation. Tangible objects provide convincing grasping and manipulation interactions with haptic feedback of e.g., shape, mass and texture properties. But these properties are static, and cannot respond to interactions in the virtual environment. On the other hand, vibrotactile feedback provides the opportunity for delivering dynamic cues rendering many different contact properties, such as impacts, object vibrations or textures. Handheld objects or controllers in VR are usually restricted to vibrating in a monolithic fashion. In this article, we investigate how spatialiazing vibrotactile cues within handheld tangibles could enable a wider range of sensations and interactions. We conduct a set of perception studies, investigating the extent to which spatialization of vibrotactile feedback within tangible objects is possible as well as the benefits of proposed rendering schemes leveraging multiple actuators in VR. Results show that vibrotactile cues from localized actuators can be discriminated and are beneficial for certain rendering schemes.

Ultrasound mid-air haptic (UMH) devices are a novel tool for haptic feedback, capable of providing localized vibrotactile stimuli to users at a distance. UMH applications largely rely on generating tactile shape outlines on the users’ skin. Here we investigate how to achieve sensations of continuity or gaps within such two-dimensional curves by studying the perception of pairs of amplitude-modulated focused ultrasound stimuli. On the one hand, we aim to investigate perceptual effects that may arise from providing simultaneous UMH stimuli. On the other hand, we wish to provide perception-based rendering guidelines for generating continuous or discontinuous sensations of tactile shapes. Finally, we hope to contribute toward a measure of the perceptually achievable resolution of UMH interfaces. We performed a user study to identify how far apart two focal points need to be to elicit a perceptual experience of two distinct stimuli separated by a gap. Mean gap detection thresholds were found at 32.3-mm spacing between focal points, but a high within- and between-subject variability was observed. Pairs spaced below 15 mm were consistently (>95%) perceived as a single stimulus, while pairs spaced 45 mm apart were consistently (84%) perceived as two separate stimuli. To investigate the observed variability, we resort to acoustic simulations of the resulting pressure fields. These show a non-linear evolution of actual peak pressure spacing as a function of nominal focal point spacing. Beyond an initial threshold in spacing (between 15 and 18 mm), which we believe to be related to the perceived size of a focal point, the probability of detecting a gap between focal points appears to linearly increase with spacing. Our work highlights physical interactions and perceptual effects to consider when designing or investigating the perception of UMH shapes.

Ultrasound mid-air haptic (UMH) devices can remotely render vibrotactile shapes on the skin of unequipped users, e.g., to draw haptic icons or render virtual object shapes. Spatio-temporal modulation (STM), the state-of-the-art UMH shape-rendering method, provides large freedom in shape design and produces the strongest possible stimuli for this technology. Yet, STM shapes are often reported to be blurry, complicating shape identification. Dynamic tactile pointers (DTP) were recently introduced as a technique to overcome this issue. By tracing a contour with an amplitude-modulated focal point, they significantly improve shape identification accuracy over STM, but at the cost of much lower stimulus intensity. Building upon this, we propose spatio-temporally-modulated Tactile Pointers (STP), a novel method for rendering clearer and sharper UMH shapes while at the same time producing strong vibrotactile sensations. We ran two human participant experiments, which show that STP shapes are perceived as significantly stronger than DTP shapes, while shape identification accuracy is significantly improved over STM and on par with that obtained with DTP. Our work has implications for effective shape rendering with UMH and provides insights that could inform future psychophysical investigation into vibrotactile shape perception in UMH.

With robotics rapidly advancing, more effective human–robot interaction is increasingly needed to realize the full potential of robots for society. While spoken language must be part of the solution, our ability to provide spoken language interaction capabilities is still very limited. In this article, based on the report of an interdisciplinary workshop convened by the National Science Foundation, we identify key scientific and engineering advances needed to enable effective spoken language interaction with robotics. We make 25 recommendations, involving eight general themes: putting human needs first, better modeling the social and interactive aspects of language, improving robustness, creating new methods for rapid adaptation, better integrating speech and language with other communication modalities, giving speech and language components access to rich representations of the robot’s current knowledge and state, making all components operate in real time, and improving research infrastructure and resources. Research and development that prioritizes these topics will, we believe, provide a solid foundation for the creation of speech-capable robots that are easy and effective for humans to work with.

Ultrasound mid-air haptic interfaces can display highly reconfigurable vibrotactile shapes in mid-air for human-computer interaction (HCI) applications. The choice of stimulus shape, spatial, temporal and modulation parameters yields a complex design space, yet relatively little is known about the impact of these design choices on perceived stimulus properties. We define the combination of a spatial discretization of an abstract shape and a set of rules for the temporal display order and intensity modulation of the resulting points as a sampling strategy. We developed DOLPHIN, an open-source framework to aid in designing mid-air stimuli. DOLPHIN allows the study of the impact of rendering parameters on perceived stimulus properties. This platform-agnostic framework standardizes stimulus descriptions as a step toward more replicability and easier communication in the field. It enables reproduction of stimuli between perceptual experiments and ensures stimuli used in applications correspond to those evaluated in prior perceptual studies. We validated DOLPHIN’s usability by conducting a user study assessing the impact of sampling strategy design on curvature discrimination for dynamic mid-air haptic stimuli. The Weber fractions for just-noticeable differences (JNDs) in curvature were found to range between 1 and 1.4, yet no significant effect of the number of spatial sampling points on curvature discrimination was found. This result shows a practical use-case for DOLPHIN and provides insight into rendering mid-air haptic curvature.

In the US, the abuse of individuals with intellectual and developmental disabilities (I/DD) is at epidemic proportions; however, the reporting of such abuse has been severely lacking. It has been found that individuals with I/DD are more aware of when and how to report abuse if they have received abuse prevention training. Consequently, in this paper we present the design of a mobile-computing app called Recognize to teach individuals with I/DD about abuse. Our research team is diverse, with both individuals with I/DD and neurotypical individuals. We leveraged this diversity by utilizing a co-design process with our team members who live with I/DD. Our team developed three initial prototypes of the app and performed a qualitative, within-group user study with six separate individuals with I/DD who are themselves experienced teachers to other individuals with I/DD. We found that, overall, the app would be viable for use by individuals with I/DD. We end the paper with a brief discussion of the implications of our findings toward building a full prototype of the app.

During the COVID-19 pandemic practical activities for undergraduate students have been severely disrupted. Activities in the field of computing, control, electrical and electronic engineering at the University of Sheffield have been taught during the autumn term of 2020/2021 academic year through socially distanced in-lab sessions, remote access to in-lab equipment, take-home kits, and other online methods. Students were asked to leave feedback for each activity using an anonymous online questionnaire, designed to capture their perceptions on their learning experience. Based on the responses received, a number of recommendations have been formulated to help practical educators make decisions on the modes of delivery of certain activities and for certain student cohorts, when pivoting to increased distance learning. Students indicated they would prefer to conduct some activities using take-home kits in the future, paving the way for beneficial long-term changes to the delivery of some practical activities beyond the times of the pandemic.

In this paper, we present the design of ten different 3D user interactions using passive haptics and embedded in an escape game scenario in which users have to escape from a pyramid in a limited time. Our solution is innovative by its modularity, allowing interactions with virtual objects using tangible props manipulated either directly using the hands and feet or indirectly through a single prop held in the hand, in order to perform several interactions with the virtual environment (VE). We also propose a navigation technique based on the “impossible spaces” design, allowing users to naturally walk through several overlapping rooms of the VE. All together, our different interaction techniques allow the users to solve several enigmas built into a challenging scenario inside a pyramid.

In areas where the habitats of elephants and humans are rapidly encroaching on each other, real-time monitoring of the elephants' locations has the potential to drastically improve the co-existence of elephants and humans, resulting in reduced deaths in both groups. However, as tagging (using GPS collars) elephants to obtain such location information is difficult and costly, it is important to ensure very long lifetimes of the tags, which can only be achieved using energy harvesting. In this paper, we present a kinetic energy harvester that uses magnetic levitation and ferro fluid bearings to generate energy from an elephant's movements. In order to determine the feasibility of using this kinetic energy harvester for powering the tags on elephants, we obtained real acceleration data collected from an Asian elephant over a 10 day period, and this data was then used to tune the system to maximize the harvested energy. Using experimentally validated analytical and simulation models, and the actual elephant acceleration data, we find that our prototype can generate 88.91J of energy per day. This energy is not only sufficient to power the tags to acquire and transmit locations 24 times a day to a distance of 114Km (line of sight), but provides a surplus of at least 35.40J, which can be used to increase the frequency of position updates or to support alternative communication options such as GPRS. Therefore, this shows the viability of long-term tracking of elephants.

This paper introduces a tactile or contact method whereby an autonomous robot equipped with suitable sensors can choose the next sensing action involving touch in order to accurately localize an object in its environment. The method uses an information gain metric based on the uncertainty of the object's pose to determine the next best touching action. Intuitively, the optimal action is the one that is the most informative. The action is then carried out and the state of the object's pose is updated using an estimator. The method is further extended to choose the most informative action to simultaneously localize and estimate the object's model parameter or model class. Results are presented both in simulation and in experiment on the DARPA Autonomous Robotic Manipulation Software (ARM-S) robot.

This paper develops an estimation framework for sensor-guided manipulation of a rigid object via a robot arm. Using an unscented Kalman Filter (UKF), the method combines dense range information (from stereo cameras and 3D ranging sensors) as well as visual appearance features and silhouettes of the object and manipulator to track both an object-fixed frame location as well as a manipulator tool or palm frame location. If available, tactile data is also incorporated. By using these different imaging sensors and different imaging properties, we can leverage the advantages of each sensor and each feature type to realize more accurate and robust object and reference frame tracking. The method is demonstrated using the DARPA ARM-S system, consisting of a Barrett™WAM manipulator.

This paper presents a model based approach to autonomous dexterous manipulation, developed as part of the DARPA Autonomous Robotic Manipulation (ARM) program. The developed autonomy system uses robot, object, and environment models to identify and localize objects, and well as plan and execute required manipulation tasks. Deliberate interaction with objects and the environment increases system knowledge about the combined robot and environmental state, enabling high precision tasks such as key insertion to be performed in a consistent framework. This approach has been demonstrated across a wide range of manipulation tasks, and in independent DARPA testing archived the most successfully completed tasks with the fastest average task execution of any evaluated team.