Visual navigation rely heavily on semantic segmentation outcomes, which is invaluable for practical applications. However, the efficacy of this navigation method is compromised when the accuracy of semantic segmentation falls short. Crucially, the availability of an appropriate dataset containing pixel-wise class labels is imperative for constructing a robust classifier. To alleviate the burden of manual annotation, the authors have endeavored attempt to implement a semi-automatic process for generating a training dataset from 3D scanned data. To enhance the versatility of the approach, the present study introduces augmentation techniques that consider the semantic attributes of images within the target scenario: DMIT and ToD are employed to address color variations caused by seasonal changes lawn growth and fluctuations on the sun’s height, respectively. Experimental results based on images captured during the Tsukuba Challenge, a competition featuring autonomous moving robots in Japan, showed that the proposed methodology substantially enhances classification accuracy, particularly for images taken under conditions different from those during the creation of the 3D model.

Visual place recognition (VPR) seems effective for the localization process in visual navigation of a robot because a robot can recognize locations and poses using only visual information if the accuracy of VPR is sufficient. However, to apply VPR for actual scenarios where multiple locations to be distinguished are in a captured image, the classification accuracy should be improved. To solve this problem, we propose a novel method for likelihood computation in location matching composed of two kinds of components corresponding to local and global features. Experimental results using a novel dataset composed of actual images taken around the course of the Tsukuba Challenge showed that the proposed method improved accuracy by 1–6 % compared to conventional reranking methods using only local features.

A visual navigation method based on results of semantic segmentation showed interesting results in previous researches. The most significant problem of the method is that the moving performance is affected by the segmentation accuracy, which strongly depends on the training data even though SOTA methods are adopted. To create high-quality dataset for this application without huge human efforts, the authors tries to generate datasets for semantic segmentation from a 3D scanned data composed of colored point clouds. In this study, we investigate what kinds of variations are effective to construct a classifier: variation of augmentation considering shadows, shooting angles, and shooting locations. Experimental results using actual images for evaluation and generated dataset for training captured at the course of Tsukuba Challenge, which is the famous competition for autonomous moving robots in Japan, showed that adding shadows in training datasets improved the mIoU but random changes to the shooting angle and location did not always work well. By the result, it is shown that augmentation considering the characteristic of the target environment becomes important for practical use.

Semantic segmentation can provide significant information for a robot to actualize autonomous moving. To increase the classification accuracy of segmentation, appropriate datasets should be prepared, which requires huge human labors. The authors attempt to realize a semi-automatic method that generates two dimensional images having pixel-wise class labels from 3D point clouds obtained by a 3D scanner. In this study, dense point clouds are generated to improve the quality of training samples. Experimental results using data obtained around the course of the Tsukuba Challenge, which is the famous competition for autonomous moving robots in Japan, showed that the dense point clouds effective but appropriate variety of textures should be included in the training samples.

Visual odometry is a key technology for an autonomous robot to accurately determine its locations on a map accurately if a camera is the main external sensor. If IMU is available, the scale information can also be estimated by combining visual and IMU information, which is called Visual Inertial Odometry (VIO). This research attempts to modify VINS-Mono, a widely used VIO method, to improve the estimation accuracy in symbiotic environments with people in outdoor scenes, where estimation accuracy becomes worsens according to the dynamic obstacles in images. The proposed method replaces a method for feature point extraction in VINS-Mono and adds a process to limit the area for the feature point extraction using semantic information obtained from segmentation results. Experimental results using datasets created from actual environments demonstrate that SuperPoint showed the best accuracy for most scenes and that area limitation improved estimation accuracy when many dynamic obstacles were included in the input images.

An autonomous moving scheme with semantic information extracted from images captured by the monocular camera was proposed, providing accurate results of semantic segmentation. A training dataset must accommodate the moving environment to train a classifier for autonomous moving. However, preparing a large-scale dataset composed of images having pixel-wise manually-annotated class labels is impractical. We generated datasets automatically from 3D point clouds, for reducing manpower. The dataset had significant problems: domain gap and shadows. Therefore, style transfer is incorporated in the proposed scheme to bridge the gap to resolve this problem. Moreover, pseudo shadows were added to improve the classification accuracy of testing images with shadows. Experimental results using 3D point clouds and testing images taken around Tsukuba City showed that classification accuracy was improved by the proposed scheme. The classification accuracy of the sidewalk, the most significant class for autonomous moving at the Tsukuba Challenge, was 95.7%.

Visual odometry is a key technology for the lo-calization of autonomous robots in real-world environments. In particular, when a robot adopts a camera as an external sensor, visual odometry can provide useful information for fine-grained localization. Many approaches have been developed for this task; however, this study proposes a novel scheme that uses traditional feature extraction and tracking to realize computationally efficient but sufficiently accurate for practical use when used with a visual navigation scheme based on semantic segmentation. The key features of the proposed scheme can be summarized as follows: the elimination of feature points on moving obstacles, feature point extraction on object boundaries, feature tracking, and outlier elimination, considering semantic information. The proposed scheme reduced the estimation error of visual odometry to 12% of the existing scheme in the best case, according to experimental results, using datasets created with the CARLA simulator.

Visual navigation based on the results of semantic segmentation requires high classification accuracy. Previous research has proven that a classifier of semantic segmentation trained upon a dataset generated from a 3D model performs well when the input images are also generated from a 3D model. However, when the input images are real 2D images captured at the same location by a camera mounted on a robot, the average classification accuracy deteriorates. To overcome this issue, a novel scheme is proposed to improve the classification accuracy of semantic segmentation when the training data is generated from a 3D point cloud. The key features of the proposed scheme are filling in the missing data by inpainting and domain adaptation by histogram matching. To evaluate the proposed scheme, datasets composed of real images captured during a variety of seasons, weathers, and times were created. Experimental results showed that ICNet trained upon our dataset could provide accurate results for visual navigation.

Several proposed schemes have shown remarkable results in autonomous navigation in actual scenes. However, most schemes are dependent on expensive three-dimensional sensing devices such as 3D LiDAR. To solve this problem, in this study, a novel scheme supporting autonomous movement is proposed, particularly for road direction estimation using Virtual LiDAR signals generated through results of semantic segmentation and geometrical information of the camera. Experimental results using the CARLA simulator and actual images taken at outdoor scenes showed that the proposed scheme achieves accurate results, even when moving obstacles are present on the road.

The use of detailed metric maps for autonomous movement of robots has been popularized in recent times. Three-dimensional sensing devices, such as 3D LiDAR and RADAR, which are expensive yet indispensable, are utilized to generate these metric maps, and ultimately, perform localization. To reduce the cost of sensing devices, we try to realize autonomous movement of a robot using only cheap image sensors, such as webcams. For robot navigation, image processing tends to be applied to collision avoidance by finding obstacles. In contrast, our approach does not use visual object detection but adopts path planning based on movable area extraction from input images using semantic segmentation. To obtain accurate results for visual navigation, this paper proposes the use of novel datasets for semantic segmentation. Experimental results showed that ICNet could extract the movable area with more than 99% accuracy if it was trained with appropriate datasets, and a robot can run automatically based on the extracted movable area.