This study aims to enhance the training efficiency of guide dogs by employing computer vision to collect training data and analyze the movements of both trainers and dogs. This task is challenging, owing to the constant movement of cameras and unstable reference points for camera calibration, which stem from the complexities of the guide dog training process and the surrounding environment. In addition, trainers and dogs walk side by side, making it difficult for 2D videos to capture complete interactions without obstacles. We present a comprehensive system that starts with 2D video footage from multicamera setups, proceeds to extrinsic camera calibration from the moving trainer's joints, and reconstructs the 3D poses of guide dogs and trainers. This process includes human 2D/3D pose estimation, camera calibration, and dog 2D/3D pose estimation. A novel aspect of the proposed approach involves modifying the existing calibration method for multiple cameras. This modification is designed to achieve extrinsic camera calibration and accommodating complex camera settings in real-world situations, including both fixed and moving multicamera setups without calibration objects. We can create a 3D representation of the training sessions by detecting the trainer's 2D and 3D skeletons and using calibrated cameras to triangulate the dog's 3D pose. This allows for a detailed analysis and adjustment of guide dog training methods based on the 3D pose data of trainers and guide dogs, thereby improving the overall training process.

Many studies have been conducted on companion robots that follow behind a human leader; however, this strategy puts the robot out of sight of the person it is accompanying. To stay within sight, the robot needs to follow the leader from a different position. This paper presents a front-following system for an autonomous mobile robot using a Kinect sensor. Research effort is concentrated on control of the robot, which walks in front of the human leader. For a general human-following system, especially for front-following, both localization of the robot and the prediction of the human's motion and state position are necessary. However, the framework proposed in this study uses a machine-learning-based prediction system to direct the robot ahead of the human without the need for robot localization. The proposed human motion prediction neural network predicts the 3D coordinates of a human walking behind the robot and, when combined with a proportional-integral-differential controller to control robot movement, enables accurate following for turning angles up to$100^{\circ}$. Since robot localization is not required, only one Kinect sensor is needed. The front-following system is validated via both simulation and real-time experiments, demonstrating overall success in front-following for wide and narrow spaces.

In recent years, the need for guide dogs has increased, and a more efficient methodology for training guide dog trainers is accordingly required. One challenge is that haptic information, which is a significant part of guide-dog training, is difficult to explain and visualize. Virtual reality (VR) systems have attracted attention owing to their ability to support high-level immersive interactions with the presence of multisensory experiences. A haptic-enabled VR system could address the limitations of the conventional method and support novice trainers in practicing in an immersive and remote transmission manner. This study presents a handle-based sensing setup for recording haptic experiences of trainers during guide dog training for use in a VR system. Considering trainers perceive and apply forces via handle to obtain and control dog motion status, the applied forces are decoded as haptic information and recorded using corresponding sensors. While accuracy of proposed setup for collecting required data is checked to be high(average errors of validating forces and yaw angles are 0.4N and 0.93° respectively), we believe that the proposed recording system is able to measure haptic experiences for further haptic experience re-establishments in proposed VR guide-dog training system and could finally facilitate the education of a large number of dog trainers.

Ultrasound beam propulsion, a propulsion system that uses airborne ultrasound phased arrays (AUPAs) to propel a blimp in an indoor environment to propel a blimp, has advantages for operations near humans such as no audible noises and no risk of propeller strike. To achieve the high mobility with limited actuation force of AUPAs, the dynamics should be fully exploited. In this paper, we propose a two-degree-of-freedom controller specifically tailored for ultrasound beam propulsion. We investigate the trajectory of bang-bang control with constant thrust (BBCT control), where a blimp accelerates and then decelerates with constant thrust reaching the terminal point at rest, as one of the most basic trajectories. First, we analytically derive the trajectory of a blimp under aerodynamic drag. Then, we provide a trajectory generator that derives the maximum constant thrust for an arrangement of AUPAs and the constraints on the control input. Finally, we integrate the trajectory generator and a PID-based feedback controller in a physical setup. We evaluated the proposed controller in physical and numerical experiments. The results showed that the proposed method allows a blimp to reach the terminal point almost in expected time. We also showed that the flight time is shorter than a PID-based one-degree-of-freedom controller, which was typically used in previous studies, by 19.0 − 43.2 %.

Human–robot interaction (HRI) has been widely researched in diverse applications. A robot following a person is one such scenario investigated in the HRI field. However, human movements and actions are complex and can change dramatically. We herein demonstrate a machine learning-based system that allows a person-following robot to track in real-time the predicted future motion of a walking human, from a first-person perspective. We assume that a depth sensor that can detect the human skeleton is loaded on a mobile robot to provide data on the user’s motion from a first-person perspective. The system calculates the coordinates of the center of gravity (COG) and 25 body joints of the user. These coordinates of COG and 25 body joints are relative to the robot based on the position of the person tracked, and these are used for the input dataset of a neural network (NN) that predicts human motion. A five-layered NN estimates the relative vectors in real-time between the current person’s COG and the future position of the 25 body joints. Using a proportional–integral–derivative (PID) controller, the person-following robot can track the predicted position of a walking human 0.5 s in advance to increase the robustness of following and to avoid delays.

This paper proposes a method of extracting features with little individual difference from motion time-series data using domain confusion. Several studies have extracted features for classification tasks using transfer learning. In addition, researchers have applied feature extraction to style conversion, such as generative adversarial networks to transform poses; however, only a few studies have implemented learning to reduce individual differences. If motion features with little individuality are obtained, reproducing motions with individual characteristics becomes possible. When applied to style conversion, it enables person-to-person mapping of movements and detecting anomalies, among others. This study first proposes a method for extracting features with little individual difference using graph structure layers in neural networks; subsequently, we applied it to predict the movement of a person in the near future and compared its accuracy with the conventional method that does not explicitly consider individual differences. The accuracy improvement was confirmed in many cases for prediction up to 0.4 s ahead. For motion prediction up to 0.32 s ahead, accuracy improved in 14 out of 15 motions, and for the average score, accuracy improved by approximately 38%.

Avoiding collisions with humans during daily life is essential in human–robot interaction (HRI) environments. To this end, we propose a neural network (NN) to predict walking motions—the most frequent human motion in HRI. In a previous study, an NN was set up for predicting the walking position 0.5 s ahead by using skeletal coordinates with a low root mean square error. However, the prediction accuracy was found to be lower when predicting the four limbs, possibly because their motions had strong position uncertainty during walking. In the present study, a particle filter (PF) was used to predict a possible range, rather than the specific coordinates, of foot positions. The PF was used in three steps: initialization, sampling importance resampling, and moving the particle to predict. Once the initial distribution was set up, only the last two steps needed to be repeated frame-by-frame. The probability state distribution results of the feet were verified using three measures, and the reliability of these results was verified.

This paper introduces a midair balloon interface, a fast and soft interactive object in mid-air. Our approach tackles the trade-off between safety and speed by controlling a soft helium-filled balloon with external actuators and sensors. We developed a prototype system that uses airborne ultrasound phased arrays to propel a balloon and high-speed stereo cameras to track its motion. This configuration realizes both a high thrust/weight ratio and such a soft body that is safe-to-collide. We describe a sight-based interaction and a touch-based interaction that leverage the safety and speed of a midair balloon interface. A sight-based interaction allows the user to keep the object inside her/his view within reach by controlling a balloon to follow the direction of the user’s face. A touch-based interaction allows the user to manipulate the object directly with his/her hand, issue a command by moving his/her finger on the surface, and receive vibrotactile feedback produced by vibrating the balloon with amplitude-modulated ultrasound. We describe the implementation and evaluation of the prototype and explore the application scenarios.

We propose LipNotif, a non-contact tactile notification system that uses airborne ultrasound tactile presentation to lips. Lips are suitable for non-contact tactile notifications because they have high tactile sensitivity comparable to the palms, are less occupied in daily life, and are constantly exposed outward. LipNotif uses tactile patterns to intuitively convey information to users, allowing them to receive notifications using only their lips, without sight, hearing, or hands. We developed a prototype system that automatically recognizes the position of the lips and presents non-contact tactile sensations. Two experiments were conducted to evaluate the feasibility of LipNotif. In the first experiment, we found that directional information can be notified to the lips with an average accuracy of ± 11.1° in the 120° horizontal range. In the second experiment, we could elicit significantly different affective responses by changing the stimulus intensity. The experimental results indicated that LipNotif is practical for conveying directions, emotions, and combinations of them. LipNotif can be applied for various purposes, such as notifications during work, calling in the waiting room, and tactile feedback in automotive user interfaces.

Augmented Human (AH) is a research field enhancing human physical abilities or supporting human activity using advanced technologies. As one of the AH approaches, previous studies have attached an actuator to a human body or tools used for an activity. The attached actuators are used to control their movements to support an activity. In this study, instead of attaching actuators, we propose to directly apply noncontact ultrasound force to a lightweight tool to manipulate it. The advantage of using noncontact force is that users do not need to wear a specific device and to process tools used for the activity. As a proof-of-concept system, we developed an ultrasound-based curveball system by which table tennis players can shoot a curveball regardless of their physical ability. In the system, a moving ping-pong ball (PPB) is a target tool for remote manipulation. The system curves the trajectory of a moving PPB by continuously focusing ultrasound on it. Users can control the curve timing and the curve direction (left or right) using a racket-shaped controller. In the user study, we conducted an actual table tennis match using the curveball system and qualitatively confirmed that the player using the system had the upper hand. Another user study using a ball dispenser quantitatively showed that the ultrasound-driven curveball increased the number of mistakes of the opponent player 2.95 times. These results indicate that the proposed concept is feasible.

In this study, we construct a neural network that learns a falling motion by measuring a behavior that simulates a fall forward due to a trip, in order to realize a system to predict a fall in advance. Recent advances in machine learning techniques have enabled the development of methods for predicting behavior in real time. This is expected to be used for walking to predict a fall and provide support in advance to reduce injuries. Although many systems have been proposed to measure and detect a fall, there are few studies on data measured when a fall is caused by an unexpected disturbance during normal walking. Therefore, we do not know how long it takes for a person to fall over after a disturbance occurs, and we do not have much understanding of how predictable the phenomenon is in principle. In this study, we constructed a system to simulate a fall with a disturbance and measured the 3D skeletal data. From these results, the average time between the disturbance and the start of the fall was calculated. By using a neural network to make predictions, we confirmed that falls can be predicted at the point of the disturbance.

In this study, we prototype and examine a system that allows a user to manipulate a 3D virtual object with multiple fingers without wearing any device. An autostereoscopic display produces a 3D image and a depth sensor measures the movement of the fingers. When a user touches a virtual object, haptic feedback is provided by ultrasound phased arrays. By estimating the cross section of the finger in contact with the virtual object and by creating a force pattern around it, it is possible for the user to recognize the position of the surface relative to the finger. To evaluate our system, we conducted two experiments to show that the proposed feedback method is effective in recognizing the object surface and thereby enables the user to grasp the object quickly without seeing it.

In this paper, we clarified the temporal condition in predicting human walking through a neural network. In order to examine the relationship between human walking and prediction, we analyzed the walking motion through two different types of prediction system. One analyzed the positional error of predicted body joints to examine which moment's motion while walking influences the prediction accuracy. As a result, the error peaked around the time when the system uses the information of Toe-Off (TO) for prediction. The other analysis uses the ternary classification which predicts the sudden change of walking behaviors: turn left, turn right or stop. By dividing the temporal data into a shorter period, we examine which phase contains the essence of the proceeding direction. As a result, the correct answer rate was high in the phase around TO. From these results, it is possible to improve the accuracy of gait prediction by using the skeleton data around TO moment since the TO is the time to determine the moving direction.

In this paper, we show the interaction with a future self-silhouette image using human body motion prediction. There has been a method of extracting skeleton information of a person and learning by using a neural network so as to predict the future body movement 0.5 seconds ahead and present it in real-time. By using this method, it has been shown that the position of the center of gravity can be estimated with an accuracy of about several centimeters for a limited motion such as jump and walking. In this paper, we focus on the fact that this previous method can estimate body movement and display it in real-time. The prediction and display enable to show the image 0.3 seconds before the actual motion, which is slightly longer than the time that a person can react after seeing the prediction result. Thus displaying user's own future action in 0.3 seconds in real-time may change their behavior by seeing it. In this paper, we focus on the jump action and project its predicted motion on the ground. By improving the previously reported method, it is possible to predict the motion with an error of about 2 cm on average on the movement of the center of gravity 0.5 seconds ahead. We projected the silhouette image under three different timing conditions: 1) delay, 2) real-time, and 3) prediction. The paper shows how three different conditions are felt and how they affect human actual movement.

Physically moving real objects via a computational force connects computers and the real world and has been applied to tangible interfaces and mid-air display. Many researchers have controlled only a stationary real object by computational force. On the other hand, controlling a moving object can expand the real space that is controllable by the computer. In this paper, we explore the potential of computational force from the viewpoint of changing the trajectory of a moving object. Changing the trajectory is the primitive model to control a moving object, and it is the technological challenge requiring high-speed measurement and non-contact force with high-spatial resolution. As a proof-of-concept, we introduce Hopping-Pong changing the trajectory of a flying Ping-Pong Ball (PPB) using ultrasound force. The result shows that Hopping-Pong changes the trajectory of a PPB 344 mm. We conclude that a computational force is capable of controlling a moving object in the real world. This research contributes to expanding the computationally controlled space with applications for augmented sports, HCI and factory automation.

In this paper, we present BaLuna, a prototype of an externally actuated midair display for indoor use in a room-scale workspace. This system is the first battery-less midair display with a one-meter-cubic workspace. The system projects an image onto a balloon screen whose position is controlled by airborne ultrasound phased array (AUPA) devices. Users can naturally manipulate the screen position by dragging and dropping the screen directly with their hands. We adapted feedback-based acoustic manipulation technology that enables sparsely distributed AUPA devices to control the screen position. This is combined with a depth-image-based tracking and a three-dimensionally calibrated projector.

In this paper, we show that an inverted pendulum can be stabilized manually even when a user does not know the physical characteristics and the current state of the pendulum. We display two markers: one indicates current position of the base of the pendulum and the other indicates the target position where the base should be located 0.3 seconds later. Subjects can stabilize the pendulum for a significantly longer time than seeing the real pendulum directly, just by chasing the target marker.

In this paper, we propose a machine learning-based system named "Computational Foresight" that can forecast human body motion 0.5 seconds before the actual motion in real-time. This forecasting system can be used to estimate human gestures in advance to the actual action for reducing delays in interactive system. In addition, the system can be applied to instruct sports actions properly, and prevent elderly from falling to the ground, and so on. Proposed system detects 25 human body joints to use those data for input dataset of machine learning. We created 5-layered neural network to estimate human body motion in real-time. In our experiment, we measured jump motions of subjects for learning. In our evaluation, the prototype system scored that the center of gravity of whole body can be forecasted 0.5 sec before with its accuracy of 7.9 cm.

In this paper, we propose a novel interactive system that mutually copies adjacent 3D environments optically and physically. The system realizes mutual user interactions through haptics without wearing any devices. A realistic volumetric image is displayed using a pair of micro-mirror array plates (MMAPs). The MMAP transmissively reflects the rays from an object, and a pair of them reconstructs the floating aerial image of the object. Our system can optically copy adjacent environments based on this technology. Haptic feedback is also given by using an airborne ultrasound tactile display (AUTD). Converged ultrasound can give force feedback in midair. Based on the optical characteristics of the MMAPs, the cloned image and the user share an identical coordinate system. When a user touches the transferred clone image, the system gives force feedback so that the user can feel the mechanical contact and reality of the floating image.

A method to present volumetric haptic objects in the air using spatial modulation of ultrasound is proposed. Previous methods of airborne ultrasonic tactile display were based on vibrotactile radiation pressure and sensor feedback systems, which result in low spatial receptive resolution. The proposed approach produces a spatially standing haptic image using stationary ultrasonic waves that enable users to touch 3D images without depending on vibrotactile stimulation and sensor feedback. The omnidirectional spatial modulated haptic images are generated by a phased array surrounding a workspace, which enables enough power to feel shapes without vibrotactile technique. Compared with previous methods, the proposed method can create a completely silent image without temporal ultrasonic modulation noise that is free of the problems caused by feedback delay and errors. To investigate the active touch profiles of an ultrasonic image, this paper discusses a method to synthesize a haptic holographic image, the evaluation of our algorithm, and the results of pressure measurement and subjective experiments.

We present a sensing technology and input method that uses skin deformation estimated through a thin band-type device attached to the human body, the appearance of which seems socially acceptable in daily life. An input interface usually requires feedback. SenSkin provides tactile feedback that enables users to know which part of the skin they are touching in order to issue commands. The user, having found an acceptable area before beginning the input operation, can continue to input commands without receiving explicit feedback. We developed an experimental device with two armbands to sense three-dimensional pressure applied to the skin. Sensing tangential force on uncovered skin without haptic obstacles has not previously been achieved. SenSkin is also novel in that quantitative tangential force applied to the skin, such as that of the forearm or fingers, is measured. An infrared (IR) reflective sensor is used since its durability and inexpensiveness make it suitable for everyday human sensing purposes. The multiple sensors located on the two armbands allow the tangential and normal force applied to the skin dimension to be sensed. The input command is learned and recognized using a Support Vector Machine (SVM). Finally, we show an application in which this input method is implemented.

PINOKY is a wireless ring-like device that can be externally attached to any plush toy as an accessory that animates the toy by moving its limbs. A user is thus able to instantly convert any plush toy into a soft robot. The user can control the toy remotely or input the movement desired by moving the plush toy and having the data recorded and played back. Unlike other methods for animating plush toys, PINOKY is non-intrusive, so alterations to the toy are not required. In a user study, 1) the roles of plush toys in the participants' daily lives were examined, 2) how participants played with plush toys without PINOKY was observed, 3) how they played with plush toys with PINOKY was observed, and their reactions to the device were surveyed. On the basis of the results, potential applications were conceptualized to illustrate the utility of PINOKY.

In this paper, we show experimental results indicating that tactile mechanoreceptors have non-linear sensitivity to an applied vibration. We focus on the perceptual characteristic of high frequency amplitude-modulated (AM) vibration. It is known that humans can feel AM vibration even when its carrier frequency is higher than the perceivable range. Non-linearity of perception should be taken into consideration for explaining this characteristic. We observed a deformation of the skin when the AM vibration is applied. We found that we can detect an envelope of the applied AM vibration, even when the skin surface deforms as a linear elastic body; whereas harmonic vibratory pairs cannot be perceived. This result indicates that only a squared value of a displacement, which is proportional to strain energy density, cannot explain the non-linearity of mechanoreceptors. These perceptual characteristics should be taken into consideration for designing a tactile display with multi-spectral vibrations. Otherwise, the simultaneous multi-spectral vibrations can affect each other and may change perceptions because of the non-linearity of the perceptual process.

In this paper, we propose a large-area tactile display by controlling suction pressure. This research is based on our discovery of tactile illusion that pulling a skin through a hole with air suction causes a sensation as if something like a stick is pushing the skin. This illusion implies that our mechanoreceptors are insensitive to the sign of stress (negative or positive), i.e. we detect not stress directly but strain energy. There are two key concepts to realize our tactile display. One is the tactile illusion mentioned above and the other is "multi primitive tactile stimulation." We explain our approach to produce various tactile sensations from a sharp edge to a plane surface with a simple structure of display device based on air pressure control, and report the experimental results.