The authors attempt to actualize 3D reconstruction from a single view for previewing a room in virtual space using results of semantic segmentation. The segmentation accuracy has been drastically improved by state-of-the-art method, which enables pixel-wise classification of walls, floors, ceilings, and objects in the target room with sufficient accuracy. Assuming a room can be represented as a cuboid, its parameters can be computed analytically when semantic edges of the room structure are accurately obtained. In the actual process of the estimation, lines constructing the cuboid are estimated from detected edges. These edges are derived by spatial filtering to a semantic map corresponding to an input image. To enhance the effectiveness of edge detection on a semantic map, we adopted a simple differential filter that incorporates only two active values as filter coefficients, utilizing the smallest filter size. Experimental results using synthetic datasets showed no significant difference in the accuracy of line parameter estimation when comparing our method with a typical filter, despite using half the samples during the optimization process though sample numbers for parameter estimation became smaller obviously.

Visual navigation rely heavily on semantic segmentation outcomes, which is invaluable for practical applications. However, the efficacy of this navigation method is compromised when the accuracy of semantic segmentation falls short. Crucially, the availability of an appropriate dataset containing pixel-wise class labels is imperative for constructing a robust classifier. To alleviate the burden of manual annotation, the authors have endeavored attempt to implement a semi-automatic process for generating a training dataset from 3D scanned data. To enhance the versatility of the approach, the present study introduces augmentation techniques that consider the semantic attributes of images within the target scenario: DMIT and ToD are employed to address color variations caused by seasonal changes lawn growth and fluctuations on the sun’s height, respectively. Experimental results based on images captured during the Tsukuba Challenge, a competition featuring autonomous moving robots in Japan, showed that the proposed methodology substantially enhances classification accuracy, particularly for images taken under conditions different from those during the creation of the 3D model.

A visual navigation method based on results of semantic segmentation showed interesting results in previous researches. The most significant problem of the method is that the moving performance is affected by the segmentation accuracy, which strongly depends on the training data even though SOTA methods are adopted. To create high-quality dataset for this application without huge human efforts, the authors tries to generate datasets for semantic segmentation from a 3D scanned data composed of colored point clouds. In this study, we investigate what kinds of variations are effective to construct a classifier: variation of augmentation considering shadows, shooting angles, and shooting locations. Experimental results using actual images for evaluation and generated dataset for training captured at the course of Tsukuba Challenge, which is the famous competition for autonomous moving robots in Japan, showed that adding shadows in training datasets improved the mIoU but random changes to the shooting angle and location did not always work well. By the result, it is shown that augmentation considering the characteristic of the target environment becomes important for practical use.

Semantic segmentation can provide significant information for a robot to actualize autonomous moving. To increase the classification accuracy of segmentation, appropriate datasets should be prepared, which requires huge human labors. The authors attempt to realize a semi-automatic method that generates two dimensional images having pixel-wise class labels from 3D point clouds obtained by a 3D scanner. In this study, dense point clouds are generated to improve the quality of training samples. Experimental results using data obtained around the course of the Tsukuba Challenge, which is the famous competition for autonomous moving robots in Japan, showed that the dense point clouds effective but appropriate variety of textures should be included in the training samples.

Visual odometry is a key technology for an autonomous robot to accurately determine its locations on a map accurately if a camera is the main external sensor. If IMU is available, the scale information can also be estimated by combining visual and IMU information, which is called Visual Inertial Odometry (VIO). This research attempts to modify VINS-Mono, a widely used VIO method, to improve the estimation accuracy in symbiotic environments with people in outdoor scenes, where estimation accuracy becomes worsens according to the dynamic obstacles in images. The proposed method replaces a method for feature point extraction in VINS-Mono and adds a process to limit the area for the feature point extraction using semantic information obtained from segmentation results. Experimental results using datasets created from actual environments demonstrate that SuperPoint showed the best accuracy for most scenes and that area limitation improved estimation accuracy when many dynamic obstacles were included in the input images.

An autonomous moving scheme with semantic information extracted from images captured by the monocular camera was proposed, providing accurate results of semantic segmentation. A training dataset must accommodate the moving environment to train a classifier for autonomous moving. However, preparing a large-scale dataset composed of images having pixel-wise manually-annotated class labels is impractical. We generated datasets automatically from 3D point clouds, for reducing manpower. The dataset had significant problems: domain gap and shadows. Therefore, style transfer is incorporated in the proposed scheme to bridge the gap to resolve this problem. Moreover, pseudo shadows were added to improve the classification accuracy of testing images with shadows. Experimental results using 3D point clouds and testing images taken around Tsukuba City showed that classification accuracy was improved by the proposed scheme. The classification accuracy of the sidewalk, the most significant class for autonomous moving at the Tsukuba Challenge, was 95.7%.

Visual navigation based on the results of semantic segmentation requires high classification accuracy. Previous research has proven that a classifier of semantic segmentation trained upon a dataset generated from a 3D model performs well when the input images are also generated from a 3D model. However, when the input images are real 2D images captured at the same location by a camera mounted on a robot, the average classification accuracy deteriorates. To overcome this issue, a novel scheme is proposed to improve the classification accuracy of semantic segmentation when the training data is generated from a 3D point cloud. The key features of the proposed scheme are filling in the missing data by inpainting and domain adaptation by histogram matching. To evaluate the proposed scheme, datasets composed of real images captured during a variety of seasons, weathers, and times were created. Experimental results showed that ICNet trained upon our dataset could provide accurate results for visual navigation.