We present a novel acoustic camera simulator (ACSim) that generates realistic sonar images by incorporating recursive ray tracing and sonar artifact modeling and provides various ground truth labels, enabling benchmarking and learning purposes. The 2D forward-looking sonar (FLS), also known as the acoustic camera, produces high-quality 2D images. Conducting real-world underwater experiments is challenging, making realistic sonar image simulation a necessary alternative. However, existing simulators often lack sufficient realism or are limited to specific scenes and phenomena. As a result, training on simulations and testing on real sonar images (i.e., sim-to-real) remains an open problem for deep learning-based applications. Our work introduces a novel sonar simulator with a customized rendering engine. We use recursive ray tracing to model multipath reflections in arbitrary scenes and propose physics-based shading for intensity computation. We propose a resampling method for anti-aliasing and model significant artifacts such as rolling shutter distortions and cross-talk noise. The simulator provides various ground truths for benchmarking and deep learning applications. We tested several tasks by training on synthetic images and demonstrated that the models also work on real images. We developed a Blender add-on for an enhanced user interface and will make the simulator open-source to advance future research.

2D forward-looking sonar is a crucial sensor for underwater robotic perception. A well-known problem in this field is estimating missing information in the elevation direction during sonar imaging. There are demands to estimate 3D information per image for 3D mapping and robot navigation during fly-through missions. Recent learning-based methods have demonstrated their strengths, but there are still drawbacks. Supervised learning methods have achieved high-quality results but may require further efforts to acquire 3D ground-truth labels. The existing self-supervised method requires pretraining using synthetic images with 3D supervision. This study aims to realize stable self-supervised learning of elevation angle estimation without pretraining using synthetic images. Failures during self-supervised learning may be caused by motion degeneracy problems. We first analyze the motion field of 2D forward-looking sonar, which is related to the main supervision signal. We utilize a modern learning framework and prove that if the training dataset is built with effective motions, the network can be trained in a self-supervised manner without the knowledge of synthetic data. Both simulation and real experiments validate the proposed method.

Retrieving the missing dimension information in acoustic images from 2D forward-looking sonar is a well-known problem in the field of underwater robotics. There are works attempting to retrieve 3D information from a single image which allows the robot to generate 3D maps with fly-through motion. However, owing to the unique image formulation principle, estimating 3D information from a single image faces severe ambiguity problems. Classical methods of multi-view stereo can avoid the ambiguity problems, but may require a large number of viewpoints to generate an accurate model. In this work, we propose a novel learning-based multi-view stereo method to estimate 3D information. To better utilize the information from multiple frames, an elevation plane sweeping method is proposed to generate the depth-azimuth-elevation cost volume. The volume after regularization can be considered as a probabilistic volumetric representation of the target. Instead of performing regression on the elevation angles, we use pseudo front depth from the cost volume to represent the 3D information which can avoid the 2D-3D problem in acoustic imaging. High-accuracy results can be generated with only two or three images. Synthetic datasets were generated to simulate various underwater targets. We also built the first real dataset with accurate ground truth in a large scale water tank. Experimental results demonstrate the superiority of our method, compared to other state-of-the-art methods.

In this paper, we propose a novel motion planning method for off-road unmanned ground vehicles, based on three-dimensional (3-D) terrain map information. Previous studies on the motion planning of a vehicle traveling on rough terrain dealt only with a relatively small environment. Furthermore, unique vehicle characteristics were not considered, and it was also impossible to incorporate regulations, such as maintaining driving speed and suppressing posture change. The proposed method enables vehicles to adaptively generate a path by considering vehicle characteristics and the regulations, in a large-scale environment, with rough terrain. A random sampling based scheme was applied to carrying out global path planning, based on a 3-D environmental model. Experimental results showed that the proposed off-road motion planner could generate an appropriate path, which satisfies vehicle characteristics and predefined regulations.

Localization is an important task for mobile service robots in indoor spaces. In this research, we propose a novel technique for indoor localization using a spherical camera. Spherical cameras can obtain a complete view of the surroundings allowing the use of global environmental information. We take advantage of this in order to estimate camera position and the orientation with respect to a known 3D line map of an indoor environment, using a single image. We robustly extract 2D line information from the spherical image via spherical-gradient filtering and match it to 3D line information in the line map. Our method requires no information about the 3D-2D line correspondences. In order to avoid a complicated six degrees of freedom (6 DoF) search for position and orientation, we use a Manhattan world assumption to decompose the line information in the image. The 6 DoF localization process is divided into two phases. First, we estimate the orientation by extracting the three principle directions from the image. Then, the position is estimated by robustly matching the distribution of lines between the image and the 3D model via a spherical Hough representation. This decoupled search can robustly localize a spherical camera using a single image, as we demonstrate experimentally.

We propose a prediction method for lane changes in other vehicles. According to previous research, over 90 % of car crashes are caused by human mistakes, and lane changes are the main factor. Therefore, if an intelligent system can predict a lane change and alarm a driver before another vehicle crosses the center line, this can contribute to reducing the accident rate. The main contribution of this work is to propose a new feature describing the relationship of a vehicle to adjacent vehicles. We represent the new feature using a dynamic characteristic potential field that changes the distribution depending on the relative number of adjacent vehicles. The new feature addresses numerous situations in which lane changes are made. Adding the new feature can be expected to improve prediction performance. We trained the prediction model and evaluated the performance using a real traffic dataset with over 900 lane changes, and we confirmed that the proposed method outperforms previous methods in terms of both accuracy and prediction time.

We present a novel rough terrain traversability analysis method for mobile robot navigation. We focused on the scenario of mobile robot operation in a disaster environment with limited sensor data. The robot builds a map in real time and analyzes the terrain traversability of its surrounding environment. The proposed method is based on fuzzy inference so that it can handle uncertainties in the sensor data. Two values associated with the terrain traversability, roughness and slope, are calculated from an elevation map built by a laser range finder mounted on the mobile robot. These two values are inputted to the fuzzy inference, and the traversability is analyzed. Based on the traversability output from the fuzzy inference, a vector field histogram (VFH) is generated. The mobile robot course is determined according to the VFH. We demonstrated our algorithm on an artificial environment. The experimental results showed that the mobile robot was able to reach the target position safely while avoiding untraversable areas.