Machine learning techniques rely on large and diverse datasets for generalization. Computer vision, natural language processing, and other applications can often reuse public datasets to train many different models. However, due to differences in physical configurations, it is challenging to leverage public datasets for training robotic control policies on new robot platforms or for new tasks. In this work, we propose a novel framework, ExAug to augment the experiences of different robot platforms from multiple datasets in diverse environments. ExAug leverages a simple principle: by extracting 3D information in the form of a point cloud, we can create much more complex and structured augmentations, utilizing both generating synthetic images and geometric-aware penalization that would have been suitable in the same situation for a different robot, with different size, turning radius, and camera placement. The trained policy is evaluated on two new robot platforms with three different cameras in indoor and outdoor environments with obstacles.

Learning provides a powerful tool for vision-based navigation, but the capabilities of learning-based policies are constrained by limited training data. If we could combine data from all available sources, including multiple kinds of robots, we could train more powerful navigation models. In this paper, we study how a general goal-conditioned model for vision-based navigation can be trained on data obtained from many distinct but structurally similar robots, and enable broad generalization across environments and embodiments. We analyze the necessary design decisions for effective data sharing across robots, including the use of temporal context and standardized action spaces, and demonstrate that an omnipolicy trained from heterogeneous datasets outperforms policies trained on any single dataset. We curate 60 hours of navigation trajectories from 6 distinct robots, and deploy the trained GNM on a range of new robots, including an underactuated quadrotor. We find that training on diverse data leads to robustness against degradation in sensing and actuation. Using a pre-trained navigation model with broad generalization capabilities can bootstrap applications on novel robots going forward, and we hope that the GNM represents a step in that direction. For more information on the datasets, code, and videos, please check out our project page11sites.google.com/view/drive-any-robot.

Inter-robot transfer of training data is a little explored topic in learning- and vision-based robot control. Here we propose a transfer method from a robot with a lower Degree-of-Freedom (DoF) to one with a higher DoF utilizing the omnidirectional camera image. The virtual rotation of the robot camera enables data augmentation in this transfer learning process. As an experimental demonstration, a vision-based control policy for a 6- DoF robot is trained using a dataset collected by a wheeled ground robot with only three DoFs. Towards the application of robotic manipulations, we also demonstrate a control system of a 6- DoF arm robot using multiple policies with different fields of view to enable object reaching tasks.

Self-supervised monocular depth estimation has been widely investigated to estimate depth images and relative poses from RGB images. This framework is promising because the depth and pose networks can be trained from just time-sequence images without the need for the ground truth depth and poses. In this work, we estimate the depth around a robot (360° view) using time-sequence spherical camera images, from a camera whose parameters are unknown. We propose a learnable axisymmetric camera model which accepts distorted spherical camera images with two fisheye camera images as well as pinhole camera images. In addition, we trained our models with a photo-realistic simulator to generate ground truth depth images to provide supervision. Moreover, we introduced loss functions to provide floor constraints to reduce artifacts that can result from reflective floor surfaces. We demonstrate the efficacy of our method using the spherical camera images from the GO Stanford dataset and pinhole camera images from the KITTI dataset to compare our method's performance with that of baseline method in learning the camera parameters.

Localization in topological maps is essential for image-based navigation using an RGB camera. Localization using only one camera can be challenging in medium-to-large-sized environments because similar-looking images are often observed repeatedly, especially in indoor environments. To overcome this issue, we propose a learning-based localization method that simultaneously utilizes the spatial consistency from topological maps and the temporal consistency from time-series images captured by a robot. Our method combines a convolutional neural network (CNN) to embed image features and a recurrent-type graph neural network to perform accurate localization. When training our model, it is difficult to obtain the ground truth (GT) pose of the robot when capturing images in real-world environments. Hence, we propose a sim2real transfer approach with semi-supervised learning that leverages simulator images with the GT pose in addition to real images. We evaluated the proposed method quantitatively and qualitatively and compared it with several state-of-the-art baselines. The proposed method outperformed the baselines in environments where the map contained similar images. Moreover, we evaluated an image-based navigation system incorporating our localization method and confirmed that navigation accuracy significantly improved in the simulator and real environments compared to the other baseline methods.

We present an unsupervised simultaneous learning framework for the task of monocular camera re-localization and depth estimation from unlabeled video sequences. Monocular camera re-localization refers to the task of estimating the absolute camera pose from an instance image in a known environment, which has been intensively studied for alternative localization in GPS-denied environments. In recent works, cam-era re-localization methods are trained via supervised learning from pairs of camera images and camera poses. In contrast to previous works, we propose a completely unsupervised learning framework for camera re-localization and depth estimation, requiring only monocular video sequences for training. In our framework, we train two networks that estimate the scene coordinates using directions and the depth map from each image which are then combined to estimate the camera pose. The networks can be trained through the minimization of loss functions based on our loop closed view synthesis. In experiments with the 7-scenes dataset, the proposed method outperformed the re-localization of the state-of-the-art visual SLAM, ORB-SLAM3. Our method also outperforms state-of-the-art monocular depth estimation in a trained environment.

Self-supervised learning for monocular depth estimation has been widely investigated as an alternative to the supervised learning approach. Uncertainty estimation in depth estimation is a crucial problem for applications, such as autonomous driving, in detecting unreliable depth. In this study, we propose a variational model to estimate depth uncertainty in self-supervised learning. Our approach leverages time-series images to handle the depth distribution from appearance variations in training. We introduce the Mahalanobis-Wasserstein distance between two consecutive frames to learn the uncertainty. In inference, our method estimates the uncertainty map and the depth image at each pixel from a single image. In experiments on KITTI, Make3D, and DIODE datasets, we show that our model achieves better uncertainty estimation than previous approaches as well as high accuracy of depth estimation

We propose a novel objective for penalizing geometric inconsistencies and improving the depth and pose estimation performance of monocular camera images. Our objective is designed using the Wasserstein distance between two point clouds, estimated from images with different camera poses. The Wasserstein distance can impose a soft and symmetric coupling between two point clouds, which suitably maintains geometric constraints and results in a differentiable objective. By adding our objective to those of other state-of-the-art methods, we can effectively penalize geometric inconsistencies and obtain highly accurate depth and pose estimations. Our proposed method was evaluated using the KITTI dataset.

Humans can robustly follow a visual trajectory defined by a sequence of images (i.e. a video) regardless of substantial changes in the environment or the presence of obstacles. We aim at endowing similar visual navigation capabilities to mobile robots solely equipped with a RGB fisheye camera. We propose a novel probabilistic visual navigation system that learns to follow a sequence of images with bidirectional visual predictions conditioned on possible navigation velocities. By predicting bidirectionally (from start towards goal and vice versa) our method extends its predictive horizon enabling the robot to go around unseen large obstacles that are not visible in the video trajectory. Learning how to react to obstacles and potential risks in the visual field is achieved by imitating human teleoperators. Since the human teleoperation commands are diverse, we propose a probabilistic representation of trajectories that we can sample to find the safest path. We evaluate our navigation system quantitatively and qualitatively in multiple simulated and real environments and compare to state-of-the-art baselines. Our approach outperforms the most recent visual navigation methods with a large margin with regard to goal arrival rate, subgoal coverage rate, and success weighted by path length (SPL). Our method also generalizes to new robot embodiments never used during training.

This paper addresses the problem of path prediction for multiple interacting agents in a scene, which is a crucial step for many autonomous platforms such as self-driving cars and social robots. We present SoPhie; an interpretable framework based on Generative Adversarial Network (GAN), which leverages two sources of information, the path history of all the agents in a scene, and the scene context information, using images of the scene. To predict a future path for an agent, both physical and social information must be leveraged. Previous work has not been successful to jointly model physical and social interactions. Our approach blends a social attention mechanism with physical attention that helps the model to learn where to look in a large scene and extract the most salient parts of the image relevant to the path. Whereas, the social attention component aggregates information across the different agent interactions and extracts the most important trajectory information from the surrounding neighbors. SoPhie also takes advantage of GAN to generates more realistic samples and to capture the uncertain nature of the future paths by modeling its distribution. All these mechanisms enable our approach to predict socially and physically plausible paths for the agents and to achieve state-of-the-art performance on several different trajectory forecasting benchmarks.

We present semi-supervised deep learning approaches for traversability estimation from fisheye images. Our method, GONet, and the proposed extensions leverage Generative Adversarial Networks (GANs) to effectively predict whether the area seen in the input image(s) is safe for a robot to traverse. These methods are trained with many positive images of traversable places, but just a small set of negative images depicting blocked and unsafe areas. This makes the proposed methods practical. Positive examples can be collected easily by simply operating a robot through traversable spaces, while obtaining negative examples is time consuming, costly, and potentially dangerous. Through extensive experiments and several demonstrations, we show that the proposed traversability estimation approaches are robust and can generalize to unseen scenarios. Further, we demonstrate that our methods are memory efficient and fast, allowing for real-time operation on a mobile robot with single or stereo fisheye cameras. As part of our contributions, we open-source two new datasets for traversability estimation. These datasets are composed of approximately 24h of videos from more than 25 indoor environments. Our methods outperform baseline approaches for traversability estimation on these new datasets.

The modeling and identification of a mechanical system is the most important issue for many control systems in order to realize the desired control specifications. In particular, the friction characteristics often deteriorate the control performance, such as in the fast and precise positioning performance in industrial robots, the force estimation accuracy based on a disturbance observer, and the posture control performance of an inverted pendulum robot. Rolling friction tends to cause overshoot, undershoot, or limit cycles of the target value in positioning systems. In previous research, some model structures for rolling friction have been proposed to express the hysteresis characteristics in order to overcome these control issues. However, it is difficult to identify the correct parameters for precise modeling. In this paper, the modeling of rolling friction based on a Recurrent Neural Network (RNN) using Long Short-Term Memory (LSTM) is proposed to precisely express the rolling friction characteristics. The initial value design of the RNN during supervised learning is also presented to achieve a better model. The effectiveness of the proposed approach is verified by comparison with conventional friction models using an actual experimental setup.

Advanced countries are currently experiencing an aging society, and many people could benefit from a personal robot that can support a comfortable lifestyle. However, excessive and premature robot assistance may deteriorate the user's physical abilities and can accelerate their aging process. The authors have already proposed a personal robot that can follow the user, even on uneven road surfaces. Therefore, people utilizing this personal robot do not need to carry heavy baggage, even after shopping. It is reported that a few brisk walks per week can be advantageous to our health management and can improve our quality of life, so this means that the personal robot can encourage the user to maintain an active lifestyle. For this personal robot, close following is required to ensure the safety of the baggage and the personal robot. However, it is difficult for the conventional approach to avoid collisions when the user stops suddenly. In this paper, a control approach based on model predictive control is proposed to achieve two conflicting requirements: close following and no collision. The proposed approach predicts multiple future outcomes, in which the user either stops or accelerates in the next step, in order to realize an appropriate relative distance and to consider the upper and lower boundaries for posture stabilization. The effectiveness of the proposed approach can be verified in numerical simulations using a multibody dynamics model consisting of actual 3D representations.

Various robots are being developed to support active human lifestyles throughout the world. In particular, robots that can support a comfortable lifestyle for elderly people are needed for the coming super-aging society in many countries. The authors have already proposed the Personal Robot (PR) shown in Fig. 1, which can follow a human being and carry their baggage; the PR user does not need to carry heavy bags, even after shopping. The PR, therefore, can encourage not only elderly people but also young people to walk outside. This means that the PR can support a life of wellness in the true sense. In conventional research, a control approach for the roll angle is proposed to ensure the stability margin for steady turning because in order to follow a human being, the PR must realize high traveling performance. In this paper, a method of reference generation for the roll angle and turning angular velocity is proposed to take into account the stability margin during the transient state using model predictive control. According to the proposed approach, the quickest turning motion can be realized by keeping the upper and lower boundary constraints for the zero moment point (ZMP). The effectiveness of the proposed approach is verified by experiment using the prototype PR.

Recently, a super-aging society has developed in many countries around the world. The research and development of PRs (personal robots) that improve the quality of human life is needed in order to accommodate the aging society. Elderly people will be able to spend their lives happily and effortlessly with the aid of useful and convenient PRs. However, excessive or premature use of PRs may cause health deterioration or contribute to the quick aging phenomenon. In this paper, a new prototype PR is proposed that can follow human beings with their baggage. Elderly people, therefore, will be able to go outside empty handed to shop, enjoy the fresh air, and visit friends. This PR will encourage people to walk outside and can eventually support an active lifestyle in its true sense. For actual use, PRs should have both a small footprint for coexistence in human society and high traveling performance for following the human wherever they go. Active posture control for the roll and pitch angles is applied to the PR to realize these requirements. The proposed structure and control approach using lateral acceleration as a control variable is verified by experiment using the new prototype robot.