We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods, ours is directly applicable to scenarios with real-time constraints, such as robotics or mixed reality. To overcome the inherent challenges of online methods, we make two main contributions. First, to effectively extract information from the input RGB-D video stream, we jointly estimate geometry and semantic labels per frame in 3D. A key focus of our approach is to reason about semantic entities both in the 2D input and the local 3D domain to leverage differences in spatial context and network architectures. Our method predicts 2D features using an off-the-shelf segmentation network. The extracted 2D features are refined by a lightweight 3D network to enable reasoning about the local 3D structure. Second, to efficiently deal with an infinite stream of input RGB-D frames, a subsequent network serves as a temporal expert predicting the incremental scene updates by leveraging 2D, 3D, and past information in a learned manner. These updates are then integrated into a global scene representation. Using these main contributions, our method can enable scenarios with real-time constraints and can scale to arbitrary scene sizes by processing and updating the scene only in a local region defined by the new measurement. Our experiments demonstrate improved results compared to existing online methods that purely operate in local regions and show that complementary sources of information can boost the performance. We provide a thorough ablation study on the benefits of different architectural as well as algorithmic design decisions. Our method yields competitive results on the popular ScanNet benchmark and SceneNN dataset.

Although context-based monocular depth estimation has shown remarkable improvement, the adaptation to unseen contexts is still a major challenge. On the other hand, the use of physical depth cues, such as defocus associated with lens aberration, allows context-independent depth estimation. However, explicitly supervising physical depth cues would have a significant impact on cost and versatility, because of the need to use expensive equipment to obtain the ground truth. Therefore, we propose a novel self-supervised learning for a single-shot neural depth from defocus (DfD) utilizing structure from motion (SfM) images taken by the target lens. Since the scale of SfM depth is ambiguous, we used rank loss to train the network. To demonstrate the versatility of our method, we conducted validation experiments using not only DSLR cameras but also smartphones with small image sensors. We confirmed that our method is highly accurate by a large margin over state-of-the-art methods including the physically-calibrated neural single-shot DfD and context-based methods.

Predicting the future trajectories of surrounding vehicles is a key competence for safe and efficient real-world autonomous driving systems. Previous works have presented deep neural network models for predictions using a detailed prior map which includes driving lanes and explicitly expresses the road rules like legal traffic directions and valid paths through intersections. Since it is unrealistic to assume the existence of the detailed prior maps for all areas, we use a map generated from only perceptual data (3D points measured by a LiDAR sensor). Such maps do not explicitly denote road rules, which makes prediction tasks more difficult. To overcome this problem, we propose a novel generative adversarial network (GAN) based framework. A discriminator in our framework can distinguish whether predicted trajectories follow road rules, and a generator can predict trajectories following it. Our framework implicitly extracts road rules by projecting trajectories onto the map via a differentiable function and training positional relations between trajectories and obstacles on the map. We also extend our framework to multimodal predictions so that various future trajectories are predicted. Experimental results show that our method outperforms other state-of-the-art methods in terms of trajectory errors and the ratio of trajectories that fall on drivable lanes.

Implementation of safe and efficient autonomous driving systems requires accurate prediction of the long-term trajectories of surrounding vehicles. High uncertainty in traffic behavior makes it difficult to predict trajectories in urban environments, which have various road geometries. To over-come this problem, we propose a method called lane-based multimodal prediction network (LAMP-Net), which can handle arbitrary shapes and numbers of traffic lanes and predict both the future trajectory along each lane and the probability of each lane being selected. A vector map is used to define the lane geometry and a novel lane feature is introduced to represent the generalized geometric relationships between the vehicle state and lanes. Our network takes this feature as the input and is trained to be versatile for arbitrarily shaped lanes. Moreover, we introduce a vehicle motion model constraint to our network. Our prediction method combined with the constraint significantly enhances prediction accuracy. We evaluate the prediction performance on two datasets which contain a wide variety of real-world traffic scenarios. Experimental results show that our proposed LAMP-Net outperforms state-of-the-art methods.

We propose an uncertainty-based sensor fusion framework for visual-inertial odometry, which is the task of estimating relative motion using images and measurements from inertial measurement units. Visual-inertial odometry enables robust and scale-aware estimation of motion by incorporating sensor states, such as metric scale, velocity, and the direction of gravity, into the estimation. However, the observability of the states depends on sensor motion. For example, if the sensor moves in a constant velocity, scale and velocity cannot be observed from inertial measurements. Under these degenerate motions, existing methods may produce inaccurate results because they incorporate erroneous states estimated from non-informative inertial measurements. Our proposed framework is able to avoid this situation by adaptively switching estimation modes, which represents the states that should be incorporated, based on their uncertainties. These uncertainties can be obtained at a small computational cost by reusing the Jacobian matrices computed in bundle adjustment. Our approach consistently outperformed conventional sensor fusion in datasets with different motion characteristics, namely, the KITTI odometry dataset recorded by a ground vehicle and the EuRoC MAV dataset captured from a micro aerial vehicle.

Visual relocalization addresses a problem of estimating the camera pose where a given query image is taken. In this paper, we focus on a scene-independent approach, which utilizes a database to maintain information of the scene. This approach thus enables adaptation to a new scene by switching a database. A standard procedure in this approach is to first retrieve images similar to the query from a database, then match 2D keypoints of the query image to 3D points visible in the retrieved images, and finally solve the perspective-n-point problem to estimate the pose. Recently, convolutional neural networks (CNNs) have been used for the retrieval and matching tasks in this process, and demonstrated promising accuracy and robustness. These CNNs are separately trained for retrieval and matching tasks, which could result in suboptimal relocalization accuracy. In this paper, we propose the first CNN-based relocalization framework, which is both scene-independent and end-to-end trainable. This framework jointly optimizes retrieval and matching tasks to maximize the relocalization accuracy by backpropagating the relocalization errors to both tasks. We demonstrate the effectiveness of the end-to-end training, robustness against new scenes, and the state-of-the-art accuracy on indoor and outdoor datasets, with computation done in real-time.

Several machine learning tasks require to represent the data using only a sparse set of interest points. An ideal detector is able to find the corresponding interest points even if the data undergo a transformation typical for a given domain. Since the task is of high practical interest in computer vision, many hand-crafted solutions were proposed. In this paper, we ask a fundamental question: can we learn such detectors from scratch? Since it is often unclear what points are interesting, human labelling cannot be used to find a truly unbiased solution. Therefore, the task requires an unsupervised formulation. We are the first to propose such a formulation: training a neural network to rank points in a transformation-invariant manner. Interest points are then extracted from the top/bottom quantiles of this ranking. We validate our approach on two tasks: standard RGB image interest point detection and challenging cross-modal interest point detection between RGB and depth images. We quantitatively show that our unsupervised method performs better or on-par with baselines.

This paper deals with deep neural networks for predicting accurate dense disparity map with Semi-global matching (SGM). SGM is a widely used regularization method for real scenes because of its high accuracy and fast computation speed. Even though SGM can obtain accurate results, tuning of SGMs penalty-parameters, which control a smoothness and discontinuity of a disparity map, is uneasy and empirical methods have been proposed. We propose a learning based penalties estimation method, which we call SGM-Nets that consist of Convolutional Neural Networks. A small image patch and its position are input into SGMNets to predict the penalties for the 3D object structures. In order to train the networks, we introduce a novel loss function which is able to use sparsely annotated disparity maps such as captured by a LiDAR sensor in real environments. Moreover, we propose a novel SGM parameterization, which deploys different penalties depending on either positive or negative disparity changes in order to represent the object structures more discriminatively. Our SGM-Nets outperformed state of the art accuracy on KITTI benchmark datasets.

We present the application of 3D reconstruction technology to the inspection and decommissioning work at the damaged Fukushima Daiichi nuclear power station in Japan. We discuss the challenges of this project, such as the difficult image capture conditions (including under water), required use of limited imaging hardware, and capture by personnel inexperienced in 3D reconstruction. We present an overview of the system developed for this project, a real-time reconstruction pipeline with robust camera pose estimation, low-latency probabilistic dense depth estimation and a novel descriptor for point cloud alignment - the Co-occurrence Histogram of Angle and Distance (CHAD). We discuss the modifications required to standard algorithms in order to perform reliably in such a scenario. As well as quantitative evaluations of these components on existing datasets, we show qualitative 3D reconstruction results of debris from the damaged plant and its spent fuel pool. Such results have enabled planning of the critical process of debris removal, without the harmful requirement of extensive human presence on site.

The last few years have seen considerable progress in pedestrian detection. Recent work has established a combination of oriented gradients and optic flow as effective features although the detection rates are still unsatisfactory for practical use. This paper introduces a new type of motion feature, the co-occurrence flow (CoF). The advance is to capture relative movements of different parts of the entire body, unlike existing motion features which extract internal motion in a local fashion. Through evaluations on the TUD-Brussels pedestrian dataset, we show that our motion feature based on co-occurrence flow contributes to boost the performance of existing methods.

Detecting pedestrians from a moving vehicle is a challenging problem since the essence of the task is to search non-rigid moving objects with various appearances in a dynamic and outdoor environment. In order to alleviate these difficulties, we propose a new human detection framework which makes the most use of stereo vision. While the conventional stereo-based detection methods initially generate regions of interest or ROIs on one of stereo images, the proposed one defines the ROIs on both left and right images. This paper presents two different ways for utilizing the stereo ROIs. Thefirst one is to classify the stereo ROIs individually and integrate the classification scores to obtain the final decision. The second one is to extend gradient-based local descriptors [1, 14] to multiple views and present new feature descriptors which we call Stereo HOG and Stereo CoHOG. Through experiments we show that both methods significantly reduce the false alarm rate while keeping the detection rate comparing with monocular-based methods.

Pedestrian detection is a difficult task for the following reasons. Firstly, pedestrians have many variations of size, pose, clothing, and motion. Secondly, the background tends to be crowded. Thirdly, many pedestrian-like patterns are observed in real environments. As a result, even if state-of-the-art pattern recognition method is used, a frame-by-frame detection has many false positives and negatives. This article presents a method of temporal integration for stereo-based pedestrian detection for improving the detection performance. In our method, a pedestrian is detected by evaluating consistency of the extracted pedestrian candidate for a short period of time. In order to get the consistency, the pedestrian candidate at each frame is combined with temporally corresponded ones through a hypothesis selection process. We demonstrate the effectiveness of our method by checking a recall and false positive number with 20,000 frames recorded in complex urban environments and public data sets. The proposed method reduces the number of false detection to one hundredth with holding recall.

In this paper, we propose a method for robust and real-time estimation of 3D motion. In our method, we use multiple sample points that are on an object before and after moving. Point correspondences between frames are not necessary. 3D motion is reliably estimated by using sample points with image positions, image features and 3D spatial positions, from both images. Our method achieves real-time performance due to usage of just sample points. We present the effectiveness of our method by means of experimental results and compare its performance with that of other methods.

In this paper, we propose a simultaneous estimation method of structure and motion in dynamic scenes. Usual methods for obtaining structure and motion using stereo cameras require two kinds of operations: stereo correspondence and tracking. Therefore, we must separately determine the correspondence between stereo images and sequential images. This necessity complicates the algorithm and increases the possibility of mismatches because of the object's motion and visibility change in the images. Our proposed method makes two contributions. The first contribution is the method of corresponding all stereo images and sequential images at once. Therefore, we can obtain the structure and motion simultaneously and more accurately. On the other hand, most stereo correspondence algorithms are limited to use under a synchronized status. In a stereo rig using unsynchronized cameras, as are most commercially available cameras, the structure cannot be obtained by stereo correspondence and triangulation because of the unknown time offset between cameras. Therefore, our second contribution is a method of estimating structure, motion, and time offset simultaneously using unsynchronized stereo cameras. This latter task is accomplished by taking advantage of the first contribution scheme. Additionally, our method requires no preprocessing such as motion segmentation for separating identical-motion objects and advance calibration of the time offset. Finally, we present the experimental results using both synthetic and real images.

This paper proposes a method for vehicle ego-motion estimation using vehicle-mounted stereo cameras. Estimating ego-motion using cameras requires extraction of static regions from the images. We first use stereo images to estimate regions which correspond to the road plane and can be considered as static areas. Subsequently, we propose a virtual projection plane (VPP) image that is equivalent to the top view of the road scene. A vehicle's ego-motion is obtained by matching the sequential VPP images of the road patterns in the extracted regions. We use a vehicle-motion model and consider a matching method to easily and accurately determine the ego-motion. Finally, we present experimental results obtained using our method