Mixup provides interpolated training samples and allows the model to obtain smoother decision boundaries for better generalization. The idea can be naturally applied to the domain adaptation task, where we can mix the source and target samples to obtain domain-mixed samples for better adaptation. However, the extension of the idea from classification to segmentation (i.e., structured output) is nontrivial. This paper systematically studies the impact of mixup under the domain adaptive semantic segmentation task and presents a simple yet effective mixup strategy called Bidirectional Domain Mixup (BDM). In specific, we achieve domain mixup in two-step: cut and paste. Given the warm-up model trained from any adaptation techniques, we forward the source and target samples and perform a simple threshold-based cut out of the unconfident regions (cut). After then, we fill-in the dropped regions with the other domain region patches (paste). In doing so, we jointly consider class distribution, spatial structure, and pseudo label confidence. Based on our analysis, we found that BDM leaves domain transferable regions by cutting, balances the dataset-level class distribution while preserving natural scene context by pasting. We coupled our proposal with various state-of-the-art adaptation models and observe significant improvement consistently. We also provide extensive ablation experiments to empirically verify our main components of the framework. Visit our project page with the code at https://sites.google.com/view/bidirectional-domain-mixup

With the increasing social demands of disaster response, methods of visual observation for rescue and safety have become increasingly important. However, because of the shortage of datasets for disaster scenarios, there has been little progress in computer vision and robotics in this field. With this in mind, we present the first large-scale synthetic dataset of egocentric viewpoints for disaster scenarios. We simulate pre- and post-disaster cases with drastic changes in appearance, such as buildings on fire and earthquakes. The dataset consists of more than 300K high-resolution stereo image pairs, all annotated with ground-truth data for the semantic label, depth in metric scale, optical flow with sub-pixel precision, and surface normal as well as their corresponding camera poses. To create realistic disaster scenes, we manually augment the effects with 3D models using physically-based graphics tools. We train various state-of-the-art methods to perform computer vision tasks using our dataset, evaluate how well these methods recognize the disaster situations, and produce reliable results of virtual scenes as well as real-world images. We also present a convolutional neural network-based egocentric localization method that is robust to drastic appearance changes, such as the texture changes in a fire, and layout changes from a collapse. To address these key challenges, we propose a new model that learns a shape-based representation by training on stylized images, and incorporate the dominant planes of query images as approximate scene coordinates. We evaluate the proposed method using various scenes including a simulated disaster dataset to demonstrate the effectiveness of our method when confronted with significant changes in scene layout. Experimental results show that our method provides reliable camera pose predictions despite vastly changed conditions.

Accurate and fast inventory management algorithms are essential in the modern distribution industry. However, the configuration process of inventory management algorithms is very expensive, and the direct comprehensive management of inventory procedures is labor intensive and inaccurate. Therefore, in this paper, we propose an optical character recognition (OCR)-based inventory management algorithm to resolve these practical issues. The main purpose of our inventory management algorithm is to automatically inspect whether a list of items and the actual items match. To this end, our method consists of three steps, namely, text detection, text recognition, and text matching. In addition, to expand our algorithm to real-world applications, we propose adversarial training to ensure robustness against various damaged images, including corruption, blur, and inappropriate viewpoints. To train the network, we construct a new inventory management dataset (IMD) consisting of 10,000 sheets in real retail store environments. We verify our algorithm on the public dataset and our new IMD. As a result, we experimentally demonstrate that our method is not only robust against various damaged images but is also easily applicable in both large and small scale distributions stores at a low cost. Our code and dataset is available at https://blog.airlab.re.kr/Deform-and-Recover/.

In this paper, we present the first large-scale synthetic dataset for visual perception in disaster scenarios, and analyze state-of-the-art methods for multiple computer vision tasks with reference baselines. We simulated before and after disaster scenarios such as fire and building collapse for fifteen different locations in realistic virtual worlds. The dataset consists of more than 300K high-resolution stereo image pairs, all annotated with ground-truth data for semantic segmentation, depth, optical flow, surface normal estimation and camera pose estimation. To create realistic disaster scenes, we manually augmented the effects with 3D models using physical-based graphics tools. We use our dataset to train state-of-the-art methods and evaluate how well these methods can recognize the disaster situations and produce reliable results on virtual scenes as well as real-world images. The results obtained from each task are then used as inputs to the proposed visual odometry network for generating 3D maps of buildings on fire. Finally, we discuss challenges for future research.

In this paper, we present a multi-camera sensor system along with its control algorithm for automated visual inspection from a moving vehicle. To accomplish this task, we propose a unique hardware configuration consisting of a frontal stereo vision system, six lateral cameras motorized to tilt, and a GPS/IMU sensor mounted on the roof of a car. From the frontal stereo system, we detect electric poles and estimate their corresponding 3D positions. Based on this 3D estimation, the tilt angles of the motorized lateral cameras are controlled in real-time to capture high resolution images of the equipment - typically installed a few meters above the road surface. In addition, inertial odometry information from the GPS/IMU module is utilized for pose estimation, object localization, and re-identification among cameras. Experimental results demonstrate the efficiency and robustness of our system for automated electric equipment maintenance, which can reduce human effort significantly.

We introduce a novel method to automatically adjust camera exposure for image processing and computer vision applications on mobile robot platforms. Because most image processing algorithms rely heavily on low-level image features that are based mainly on local gradient information, we consider that gradient quantity can determine the proper exposure level, allowing a camera to capture the important image features in a manner robust to illumination conditions. We then extend this concept to a multi-camera system and present a new control algorithm to achieve both brightness consistency between adjacent cameras and a proper exposure level for each camera. We implement our prototype system with off-the-shelf machine-vision cameras and demonstrate the effectiveness of the proposed algorithms on practical applications, including pedestrian detection, visual odometry, surround-view imaging, panoramic imaging, and stereo matching.

This paper presents a vision system and a depth processing algorithm for DRC-HUBO+, the winner of the DRC finals 2015. Our system is designed to reliably capture 3D information of a scene and objects and to be robust to challenging environment conditions. We also propose a depth-map upsampling method that produces an outliers-free depth map by explicitly handling depth outliers. Our system is suitable for robotic applications in which a robot interacts with the real-world, requiring accurate object detection and pose estimation. We evaluate our depth processing algorithm in comparison with state-of-the-art algorithms on several synthetic and real-world datasets.

This paper describes a new methodology for estimating the relative pose between two 2-D lidars. Scanned points of 2-D lidars do not have enough feature information for correspondence matching. For this reason, additional image sensors or artificial landmarks at known locations have been used to find the relative pose. We propose a novel method of estimating the relative pose between 2-D lidars without any additional sensors or artificial landmarks. By scanning two orthogonal planes, we utilize the coplanarity of the scan points on each plane and the orthogonality of the plane normals. Even if we capture planes which are not exactly orthogonal, the method provides good results using nonlinear optimization. Experiments with both synthetic and real data show the validity of the proposed method. We also derive two degenerate cases: one related to plane poses, and the other caused by the relative pose. To the best of our knowledge, this study provides the first solution for the problem.

Recently, there have been significant advances in self-driving cars, which will play key roles in future intelligent transportation systems. In order for these cars to be successfully deployed on real roads, they must be able to autonomously drive along collision-free paths while obeying traffic laws. In contrast to many existing approaches that use prebuilt maps of roads and traffic signals, we propose algorithms and systems using Unified Map built with various onboard sensors to detect obstacles, other cars, traffic signs, and pedestrians. The proposed map contains not only the information on real obstacles nearby but also traffic signs and pedestrians as virtual obstacles. Using this map, the path planner can efficiently find paths free from collisions while obeying traffic laws. The proposed algorithms were implemented on a commercial vehicle and successfully validated in various environments, including the 2012 Hyundai Autonomous Ground Vehicle Competition.

This paper describes a new methodology for estimating a relative pose of two 2D laser sensors. Two dimensional laser scan points do not have enough feature information for motion tracking. For this reason, additional image sensors or artificial landmarks have been used to find a relative pose. We propose the method to estimate a relative pose of 2D laser sensors without any additional sensor or artificial landmark. By scanning two orthogonal planes, we utilize only the coplanarity of the scan points on each plane and the orthogonality of the plane normals. Experiments with both synthetic and real data show the validity of the proposed method. To the best of our knowledge this works provides the first solution for the problem.

In this paper are presented simple and practical solutions to extrinsic calibration between a camera and a 2D laser sensor, without overlap. Previous methods utilized a plane or an intersecting line of two planes as a geometric constraint with enough common field-of-view. These required additional sensors to calibrate non-overlapping systems. In this paper, we present two methods for solving the problem - one utilizes a plane; the other utilizes an intersecting line of two planes. For each method, an initial solution of the relative positions of a non-overlapping camera and a laser sensor, was computed by adopting a reasonable assumption about geometric structures. Then we refined it via non-linear optimization, even if the assumption was not perfectly satisfied. Both simulation results and experiments using real data showed that the proposed methods provided reliable results compared to ground-truth, and similar or better results than those provided by a conventional method.

This paper presents a new structure-from-motion (SFM) technique called `generalized laser three-point' algorithm. It is designed to estimate the motion of the camera-laser fusion system which consists of a 2D laser sensor and multiple cameras. The laser points are projected onto the images and tracked to other frames to be used as 3D-2D correspondences. However, the typical three-point algorithms cannot estimate the motion of the system if three points are collinear. Using the laser points as 3D points, this case happens frequently if the laser sensor scans a large plane (e.g. open ground). Even in that case, two frames of the laser data are not collinear if the system is moved while it captures the frames. Among three point correspondences required to estimate the motion, we select two points and the other point from different frames. We estimate the relative pose between the frames by solving an 8-degree polynomial equation. The experimental results show that the proposed algorithm is more appropriate for our fusion system than the previous algorithms.

Building maps of unknown environments is a critical factor for autonomous navigation and homing, and this problem is especially challenging in large-scale environments. Recently, sensor fusion systems such as combinations of cameras and laser sensors have become popular in the effort to ensure a general level of performance in this task. In this paper, we present a new homing method in a large-scale environment using a laser-camera fusion system. Instead of fusing data to form a single map builder, we adaptively select sensor data to handle environments which contain ambiguity. For autonomous homing, we propose a new mapping strategy for building a hybrid map and a return strategy for selecting the next target waypoints efficiently. The experimental results demonstrate that the proposed algorithm enables the autonomous homing of a robot in a large-scale indoor environments in real time.

In this paper, we present a sensor fusion system of cameras and 2D laser sensors for 3D reconstruction. The proposed system is designed to capture data on a fast-moving ground vehicle. The system consists of six cameras and one 2D laser sensor. In order to capture data at high speed, we synchronized all sensors by detecting the laser ray at a specific angle and generating a trigger signal for the cameras. Reconstruction of 3D structures is done by estimating frame-by-frame motion and accumulating vertical laser scans. The difference between the proposed system and the previous works using two 2D laser sensors is that we do not assume 2D motion. The motion of the system in 3D space (including absolute scale) is estimated accurately by data-level fusion of images and range data. The problem of error accumulation is solved by loop closing, not by GPS. The moving objects are detected by utilizing the depth information provided by the laser sensor. The experimental results show that the estimated path is successfully overlayed on the satellite images.

This work introduces a modified endoscope structure that employs a spring as its backbone. This design allows backdrivability of the mechanism due to the flexibility of the spring structure. Furthermore, the full 3 DOF motion including compression and two rotational motions can be realized. The geometric analysis, kinematic modeling, and actuator sizing for this device are conducted. Both simulation and experiment have been performed to show the effectiveness of this device.