We address outdoor Neural Radiance Fields (NeRF) [23] with real-world camera views and LiDAR maps. Existing methods usually require densely-sampled source views and do not perform well with the open source camera-LiDAR datasets. In this paper, our design leverages 1) LiDAR sensors for strong 3D geometry priors that significantly improve the ray sampling locality, and 2) Conditional Adversarial Networks (cGANs) [15] to recover image details since aggregating embeddings from imperfect LiDAR maps causes artifacts. Our experiments show that while NeRF baselines produce either noisy or blurry results on Argoverse 2 [42], our system not only outperforms baselines in image quality metrics under both clean and noisy conditions, but also obtains closer Detectron2 [43] results to the ground truth images. Furthermore, this system can be used in data augmentation for training a pose regression network [3] and multi-season view synthesis. We hope this work to serve as a new LiDAR-based NeRF baseline that pushes this research direction forward (released here).

We address the problem of map sparsification for long-term visual localization. For map sparsification, a commonly employed assumption is that the pre-build map and the later captured localization query are consistent. However, this assumption can be easily violated in the dynamic world. Additionally, the map size grows as new data accumulate through time, causing large data overhead in the long term. In this paper, we aim to overcome the environmental changes and reduce the map size at the same time by selecting points that are valuable to future localization. Inspired by the recent progress in Graph Neural Network (GNN), we propose the first work that models SfM maps as heterogeneous graphs and predicts 3D point importance scores with a GNN, which enables us to directly exploit the rich information in the SfM map graph. Two novel supervisions are proposed: 1) a data-fitting term for selecting valuable points to future localization based on training queries; 2) a K-Cover term for selecting sparse points with full-map coverage. The experiments show that our method selected map points on stable and widely visible structures and out-performed baselines in localization performance

We address the problem of image registration to a compressed 3D map. While this is most often performed by comparing LiDAR scans to the point cloud based map, it depends on an expensive LiDAR sensor at run time and the large point cloud based map creates overhead in data storage and transmission. Recently, efforts have been underway to replace the expensive LiDAR sensor with cheaper cameras and perform 2D-3D localization. In contrast to the previous work that learns relative pose by comparing projected depth and camera images, we propose HyperMap, a paradigm shift from online depth map feature extraction to offline 3D map feature computation for the 2D-3D camera registration task through end-to-end training. In the proposed pipeline, we first perform offline 3D sparse convolution to extract and compress the voxelwise hypercolumn features for the whole map. Then at run-time, we project and decode the compressed map features to the rough initial camera pose to form a virtual feature image. A Convolutional Neural Network (CNN) is then used to predict the relative pose between the camera image and the virtual feature image. In addition, we propose an efficient occlusion handling layer, specifically designed for large point clouds, to remove occluded points in projection. Our experiments on synthetic and real datasets show that, by moving the feature computation load offline and compressing, we reduced map size by 87−94% while maintaining comparable or better accuracy.

We aim to assess the performance of LiDAR-to-map registration on compressive maps. Modern autonomous vehicles utilize pre-built HD (High-Definition) maps to perform sensor-to-map registration, which recovers pose estimation failures and reduces drift in a large-scale environment. However, sensor-to-map registration is usually realized by registering the sensor to a dense 3D model, which occupies massive storage space in the HD map and requires much data processing overhead. Although smaller 3D models are preferable, the optimal compressive map format for preservation of the best registration performance remains unclear.In this paper, we propose a novel and challenging benchmark to evaluate existing LiDAR-to-map registration methods from three perspectives: map compressibility, robustness, and precision. We compared various map formats, including raw points, hierarchical GMMs, and feature points, and show their performance trade-offs between compressibility and robustness on real-world LiDAR datasets: KITTI Odometry Dataset and Argoverse Tracking Dataset. Our benchmark reveals that state-of-the-art deep feature point based methods outperform traditional methods significantly when the map size budget is high. However, when map size budget is low, deep methods are outperformed by the methods using simpler models in Argoverse Tracking Dataset due to poor spatial coverage. In addition, we observe that the recently published TEASER++ significantly outperforms RANSAC for the feature point methods. Our analysis provides a valuable reference for the community to design budgeted real-world systems and find potential research opportunities. We will release the benchmark for public use.

We present Argoverse, a dataset designed to support autonomous vehicle perception tasks including 3D tracking and motion forecasting. Argoverse includes sensor data collected by a fleet of autonomous vehicles in Pittsburgh and Miami as well as 3D tracking annotations, 300k extracted interesting vehicle trajectories, and rich semantic maps. The sensor data consists of 360 degree images from 7 cameras with overlapping fields of view, forward-facing stereo imagery, 3D point clouds from long range LiDAR, and 6-DOF pose. Our 290km of mapped lanes contain rich geometric and semantic metadata which are not currently available in any public dataset. All data is released under a Creative Commons license at Argoverse.org. In baseline experiments, we use map information such as lane direction, driveable area, and ground height to improve the accuracy of 3D object tracking. We use 3D object tracking to mine for more than 300k interesting vehicle trajectories to create a trajectory forecasting benchmark. Motion forecasting experiments ranging in complexity from classical methods (k-NN) to LSTMs demonstrate that using detailed vector maps with lane-level information substantially reduces prediction error. Our tracking and forecasting experiments represent only a superficial exploration of the potential of rich maps in robotic perception. We hope that Argoverse will enable the research community to explore these problems in greater depth.

We propose a novel four-stage pipeline densely reconstructing depth from video sequences with small baselines with the goal of being fast. In our pipeline we make use of the sub-pixel precision of direct photometric bundle adjustment to reduce the number of tracked points required to estimate an accurate pose. Instead of using the exhaustive plane sweeping approach of existing small baseline methods, dense depth maps are calculated efficiently using an algorithm inspired by PatchMatch. Instead of a stereo matching error, our algorithm minimizes the variance over multiple frames with a robust mean estimation. The experiment results suggest that our method has the ability to cope with a wider range of baselines and sequence sizes. We also compare qualitative results on real small motion clips from Ha et al. in addition to our own, and show that our method outputs dense depth maps of similar or better quality and at least 10x faster.

This paper introduces a context-aware assisted active robotic walker for Parkinson's disease (PD) patients. Most of PD patients suffer from not only loss of balance but also abnormal gaits. These symptoms tend to make PD patients fall down more easily and result in low quality of life. We use Hidden Markov Model (HMM) to analyze the gait of PD patients, and then use our walker to help patients adjust their gait to become normal while applying auditory cues when abnormal gaits are recognized. To prevent user from leaning forward before falling down, the walker locks the motors when sudden forward pushing by the user is detected. Moreover, the walker can record the statics of gait from the user, making the therapists monitor the rehabilitation process relatively easier. Finally, the road conditions in front of the walker will be automatically analyzed, making user able to adjust his/her walking pace dynamically. To our best knowledge, the hereby proposed active robotic walker should be the first system which can provide walking aid to PD patients. In our experiments, the feasibility and performance of this system are evaluated by PD patients at two actual senior care units.

In this paper, we develop a multi-human detection system with a team of robots basically in an indoor environment. To start with, we propose a hybrid approach to resolve the problem of human leg detection using Laser Range Finder (LRF) for each robot, that returns not only “true” or “false” type of answer but also a probability. Specifically, the set of measurement data obtained from the laser range finder mounted on a robot is further decomposed into several sectors using an appropriate segmentation technique. Then, we apply a probabilistic model to compare these sectors with leg patterns to check if any of them belongs to the set of human leg patterns or not. Next, we examine the promising leg sectors with a modified Inscribe Angle Variance (IAV) method in order to confirm if these sectors are from human leg's arc feature or not. Moreover, we also use motion detector to check if these objects move or not as an enhancement of the detection. For the entire multi-human detection system, each robot of the team delivers the detected human information to our central control computer through the Inter-Process Communication (IPC). With prior map information of the residing environment and supposing each robot in the team has a localization module, we can then map these results of human detection from every robot into their global coordinates after process of data association. But in order to reduce the computational complexity while doing the data association among these robots in a team, we introduce a set of appropriate rules. Finally, we apply a particle filter based tracking algorithm to keep accurate track of people being detected and to improve the robustness of the detection outcome. This work has been evaluated through several experiments with a number of mobile robots and humans in an indoor environment, and promising performance has been observed.