This study aims to enhance the training efficiency of guide dogs by employing computer vision to collect training data and analyze the movements of both trainers and dogs. This task is challenging, owing to the constant movement of cameras and unstable reference points for camera calibration, which stem from the complexities of the guide dog training process and the surrounding environment. In addition, trainers and dogs walk side by side, making it difficult for 2D videos to capture complete interactions without obstacles. We present a comprehensive system that starts with 2D video footage from multicamera setups, proceeds to extrinsic camera calibration from the moving trainer's joints, and reconstructs the 3D poses of guide dogs and trainers. This process includes human 2D/3D pose estimation, camera calibration, and dog 2D/3D pose estimation. A novel aspect of the proposed approach involves modifying the existing calibration method for multiple cameras. This modification is designed to achieve extrinsic camera calibration and accommodating complex camera settings in real-world situations, including both fixed and moving multicamera setups without calibration objects. We can create a 3D representation of the training sessions by detecting the trainer's 2D and 3D skeletons and using calibrated cameras to triangulate the dog's 3D pose. This allows for a detailed analysis and adjustment of guide dog training methods based on the 3D pose data of trainers and guide dogs, thereby improving the overall training process.

Many studies have been conducted on companion robots that follow behind a human leader; however, this strategy puts the robot out of sight of the person it is accompanying. To stay within sight, the robot needs to follow the leader from a different position. This paper presents a front-following system for an autonomous mobile robot using a Kinect sensor. Research effort is concentrated on control of the robot, which walks in front of the human leader. For a general human-following system, especially for front-following, both localization of the robot and the prediction of the human's motion and state position are necessary. However, the framework proposed in this study uses a machine-learning-based prediction system to direct the robot ahead of the human without the need for robot localization. The proposed human motion prediction neural network predicts the 3D coordinates of a human walking behind the robot and, when combined with a proportional-integral-differential controller to control robot movement, enables accurate following for turning angles up to$100^{\circ}$. Since robot localization is not required, only one Kinect sensor is needed. The front-following system is validated via both simulation and real-time experiments, demonstrating overall success in front-following for wide and narrow spaces.

In recent years, the need for guide dogs has increased, and a more efficient methodology for training guide dog trainers is accordingly required. One challenge is that haptic information, which is a significant part of guide-dog training, is difficult to explain and visualize. Virtual reality (VR) systems have attracted attention owing to their ability to support high-level immersive interactions with the presence of multisensory experiences. A haptic-enabled VR system could address the limitations of the conventional method and support novice trainers in practicing in an immersive and remote transmission manner. This study presents a handle-based sensing setup for recording haptic experiences of trainers during guide dog training for use in a VR system. Considering trainers perceive and apply forces via handle to obtain and control dog motion status, the applied forces are decoded as haptic information and recorded using corresponding sensors. While accuracy of proposed setup for collecting required data is checked to be high(average errors of validating forces and yaw angles are 0.4N and 0.93° respectively), we believe that the proposed recording system is able to measure haptic experiences for further haptic experience re-establishments in proposed VR guide-dog training system and could finally facilitate the education of a large number of dog trainers.

Human–robot interaction (HRI) has been widely researched in diverse applications. A robot following a person is one such scenario investigated in the HRI field. However, human movements and actions are complex and can change dramatically. We herein demonstrate a machine learning-based system that allows a person-following robot to track in real-time the predicted future motion of a walking human, from a first-person perspective. We assume that a depth sensor that can detect the human skeleton is loaded on a mobile robot to provide data on the user’s motion from a first-person perspective. The system calculates the coordinates of the center of gravity (COG) and 25 body joints of the user. These coordinates of COG and 25 body joints are relative to the robot based on the position of the person tracked, and these are used for the input dataset of a neural network (NN) that predicts human motion. A five-layered NN estimates the relative vectors in real-time between the current person’s COG and the future position of the 25 body joints. Using a proportional–integral–derivative (PID) controller, the person-following robot can track the predicted position of a walking human 0.5 s in advance to increase the robustness of following and to avoid delays.

Avoiding collisions with humans during daily life is essential in human–robot interaction (HRI) environments. To this end, we propose a neural network (NN) to predict walking motions—the most frequent human motion in HRI. In a previous study, an NN was set up for predicting the walking position 0.5 s ahead by using skeletal coordinates with a low root mean square error. However, the prediction accuracy was found to be lower when predicting the four limbs, possibly because their motions had strong position uncertainty during walking. In the present study, a particle filter (PF) was used to predict a possible range, rather than the specific coordinates, of foot positions. The PF was used in three steps: initialization, sampling importance resampling, and moving the particle to predict. Once the initial distribution was set up, only the last two steps needed to be repeated frame-by-frame. The probability state distribution results of the feet were verified using three measures, and the reliability of these results was verified.