Ultrasound beam propulsion, a propulsion system that uses airborne ultrasound phased arrays (AUPAs) to propel a blimp in an indoor environment to propel a blimp, has advantages for operations near humans such as no audible noises and no risk of propeller strike. To achieve the high mobility with limited actuation force of AUPAs, the dynamics should be fully exploited. In this paper, we propose a two-degree-of-freedom controller specifically tailored for ultrasound beam propulsion. We investigate the trajectory of bang-bang control with constant thrust (BBCT control), where a blimp accelerates and then decelerates with constant thrust reaching the terminal point at rest, as one of the most basic trajectories. First, we analytically derive the trajectory of a blimp under aerodynamic drag. Then, we provide a trajectory generator that derives the maximum constant thrust for an arrangement of AUPAs and the constraints on the control input. Finally, we integrate the trajectory generator and a PID-based feedback controller in a physical setup. We evaluated the proposed controller in physical and numerical experiments. The results showed that the proposed method allows a blimp to reach the terminal point almost in expected time. We also showed that the flight time is shorter than a PID-based one-degree-of-freedom controller, which was typically used in previous studies, by 19.0 − 43.2 %.

Avoiding collisions with humans during daily life is essential in human–robot interaction (HRI) environments. To this end, we propose a neural network (NN) to predict walking motions—the most frequent human motion in HRI. In a previous study, an NN was set up for predicting the walking position 0.5 s ahead by using skeletal coordinates with a low root mean square error. However, the prediction accuracy was found to be lower when predicting the four limbs, possibly because their motions had strong position uncertainty during walking. In the present study, a particle filter (PF) was used to predict a possible range, rather than the specific coordinates, of foot positions. The PF was used in three steps: initialization, sampling importance resampling, and moving the particle to predict. Once the initial distribution was set up, only the last two steps needed to be repeated frame-by-frame. The probability state distribution results of the feet were verified using three measures, and the reliability of these results was verified.

This paper introduces a midair balloon interface, a fast and soft interactive object in mid-air. Our approach tackles the trade-off between safety and speed by controlling a soft helium-filled balloon with external actuators and sensors. We developed a prototype system that uses airborne ultrasound phased arrays to propel a balloon and high-speed stereo cameras to track its motion. This configuration realizes both a high thrust/weight ratio and such a soft body that is safe-to-collide. We describe a sight-based interaction and a touch-based interaction that leverage the safety and speed of a midair balloon interface. A sight-based interaction allows the user to keep the object inside her/his view within reach by controlling a balloon to follow the direction of the user’s face. A touch-based interaction allows the user to manipulate the object directly with his/her hand, issue a command by moving his/her finger on the surface, and receive vibrotactile feedback produced by vibrating the balloon with amplitude-modulated ultrasound. We describe the implementation and evaluation of the prototype and explore the application scenarios.

We propose LipNotif, a non-contact tactile notification system that uses airborne ultrasound tactile presentation to lips. Lips are suitable for non-contact tactile notifications because they have high tactile sensitivity comparable to the palms, are less occupied in daily life, and are constantly exposed outward. LipNotif uses tactile patterns to intuitively convey information to users, allowing them to receive notifications using only their lips, without sight, hearing, or hands. We developed a prototype system that automatically recognizes the position of the lips and presents non-contact tactile sensations. Two experiments were conducted to evaluate the feasibility of LipNotif. In the first experiment, we found that directional information can be notified to the lips with an average accuracy of ± 11.1° in the 120° horizontal range. In the second experiment, we could elicit significantly different affective responses by changing the stimulus intensity. The experimental results indicated that LipNotif is practical for conveying directions, emotions, and combinations of them. LipNotif can be applied for various purposes, such as notifications during work, calling in the waiting room, and tactile feedback in automotive user interfaces.

In this paper, we present BaLuna, a prototype of an externally actuated midair display for indoor use in a room-scale workspace. This system is the first battery-less midair display with a one-meter-cubic workspace. The system projects an image onto a balloon screen whose position is controlled by airborne ultrasound phased array (AUPA) devices. Users can naturally manipulate the screen position by dragging and dropping the screen directly with their hands. We adapted feedback-based acoustic manipulation technology that enables sparsely distributed AUPA devices to control the screen position. This is combined with a depth-image-based tracking and a three-dimensionally calibrated projector.

We present HaptoMime, a mid-air interaction system that allows users to touch a floating virtual screen with hands-free tactile feedback. Floating images formed by tailored light beams are inherently lacking in tactile feedback. Here we propose a method to superpose hands-free tactile feedback on such a floating image using ultrasound. By tracking a fingertip with an electronically steerable ultrasonic beam, the fingertip encounters a mechanical force consistent with the floating image. We demonstrate and characterize the proposed transmission scheme and discuss promising applications with an emphasis that it helps us 'pantomime' in mid-air.

In this paper, we propose a remote grip force sensing method based on skin surface hardness measurement. The measurement system is composed of an airborne ultrasound phased array and a laser displacement sensor. The surface hardness distribution of a human hand is estimated by optically measured surface vibration amplitude induced by the ultrasound radiation pressure. The realtime grip force is estimated using the relationship between the hand surface compliance and the grip force which is obtained experimentally in advance. The feasibility of grip force estimation is confirmed for a gripping hand keeping a constant posture.

In this paper, we propose a remote measurement system of surface compliance distributions for haptic broadcasting. Our system is composed of an ultrasound phased array generating acoustic radiation pressure on the remote object surface and a laser displacement sensor. The compliance is evaluated by the ratio of the surface displacement to the applied force. We set up a system to examine the feasibility of the method. In the experiments, the distribution of the surface compliance comparable to the human skin was successfully measured for a flat object surface.