3D occupancy prediction (3DOcc) is a rapidly rising and challenging perception task in the field of autonomous driving. Existing 3D occupancy networks (OccNets) are both computationally heavy and label-hungry. In terms of model complexity, OccNets are commonly composed of heavy Conv3D modules or transformers at the voxel level. Moreover, OccNets are supervised with expensive large-scale dense voxel labels. Model and label inefficiencies, caused by excessive network parameters and label annotation requirements, severely hinder the onboard deployment of OccNets. This paper proposes an EFFicient Occupancy learning framework, EFFOcc, that targets minimal network complexity and label requirements while achieving state-of-the-art accuracy. We first propose an efficient fusion-based OccNet that only uses simple 2D operators and improves accuracy to the state-of-the-art on three large-scale benchmarks: Occ3D-nuScenes, Occ3D-Waymo, and OpenOccupancy-nuScenes. On the Occ3D-nuScenes benchmark, the fusion-based model with ResNet-18 as the image backbone has 21.35M parameters and achieves 51.49 in terms of mean Intersection over Union (mIoU). Furthermore, we propose a multi-stage occupancy-oriented distillation to efficiently transfer knowledge to vision-only OccNet. Extensive experiments on occupancy benchmarks show state-of-the-art precision for both fusion-based and vision-based OccNets. For the demonstration of learning with limited labels, we achieve 94.38% of the performance (mIoU = 28.38) of a 100% labeled vision OccNet (mIoU = 30.07) using the same OccNet trained with only 40% labeled sequences and distillation from the fusion-based OccNet. Code is available at https://github.com/synsin0/EFFOcc.

Many researchers have used the map-matching algorithm to leverage inadequate traditional vehicle localization with HD maps for fast and efficient vehicle localization. The initialization process in the map-matching algorithm has always been problematic due to the nature of GNSS error, which might exceed the average lane width of approximately 3.5m. Thus, directly using the GNSS data to obtain the rough initial pose may cause the failure of the initialization process. The general solution is to randomly sample around the GNSS data and test the initialization with these hypotheses. However, this often leads to expensive computation and thus fails to run in realtime or online mode, as a dense sampling is required to achieve an acceptable level of initial estimation accuracy. As a viable alternative, we propose a multi-lane hypotheses approach to narrow down the search by limiting the sample pose to the number of lanes within the GNSS data's error radius. From these poses, we then perform an efficient map-matching to refine these pose candidates. Moreover, a novel belief function to evaluate the hypothesis is proposed to select the best hypothesis for system initialization robustly. Our evaluation result shows that we have outperformed the primary random sampling method in both accuracy and efficiency.

Autonomous vehicles often rely on high-definition (HD) maps to navigate around. However, lane markings (LMs) are not necessarily static objects due to wear & tear from usage and road reconstruction & maintenance. Therefore, the wrong matching between LMs in the HD map and sensor readings may lead to erroneous localization or even cause traffic accidents. It is imperative to keep LMs up-to-date. However, frequently recollecting data with dedicated hardware and specialists to update HD maps is not only cost-prohibitive but also unviable. Here we propose to utilize crowdsourced images from multiple vehicles at different times to help verify LMs for HD map maintenance. We obtain the LM distribution in the image space by considering the camera pose uncertainty in perspective projection. Both LMs in HD map and LMs in the image are treated as observations of LM distributions which allow us to construct posterior conditional distribution (a.k.a Bayesian belief functions) of LMs from either sources. An LM is consistent if belief functions from the map and the image satisfy statistical hypothesis testing. We further extend the Bayesian belief model into a sequential belief update using crowdsourced images. LMs with a higher probability of existence are kept in the HD map whereas those with a lower probability of existence are removed from the HD map. We verify our approach using real data. Experimental results show that our method is capable of verifying and updating LMs in the HD map.

Existing autonomous vehicle (AV) navigation algorithms treat lane recognition, obstacle avoidance, local path planning, and lane following as separate functional modules which result in driving behavior that is incompatible with human drivers. It is imperative to design human-compatible navigation algorithms to ensure transportation safety. We develop a new tightly-coupled perception-planning framework that combines all these functionalities to ensure human-compatibility. Using GPS-camera-lidar sensor fusion, we detect actual lane boundaries (ALBs) and propose availability-reasonability-feasibility (ARF) threefold tests to determine if we should generate virtual lane boundaries (VLBs) or follow ALBs. If needed, VLBs are generated using a dynamically adjustable multi-objective optimization framework that considers obstacle avoidance, trajectory smoothness (to satisfy vehicle kinodynamic constraints), trajectory continuity (to avoid sudden movements), GPS following quality (to execute global plan), and lane following or partial direction following (to meeting human expectation). Consequently, vehicle motion is more human compatible than existing approaches. We have implemented our algorithm and tested under open source data with satisfying results.

We propose a visual programming system that allows users to specify navigation tasks for mobile robots using high-level landmarks in a virtual reality (VR) environment constructed from the output of visual simultaneous localization and mapping (vSLAM). The VR environment provides a Google Street View-like interface for users to familiarize themselves with the robot's working environment, specify high-level landmarks, and determine task-level motion commands related to each landmark. Our system builds a roadmap by using the pose graph from the vSLAM outputs. Based on the roadmap, the high-level landmarks, and task-level motion commands, our system generates an output path for the robot to accomplish the navigation task. We present data structures, architecture, interface, and algorithms for our system and show that, given nssearch-type motion commands, our system generates a path in O(ns(nrlognr+mr)) time, where nrand mrare the number of roadmap nodes and edges, respectively. We have implemented our system and tested it on real world data.

We present a novel algorithm to automatically configure complex video processing systems when adapting them to new datasets or scenarios. This has the main benefit of significantly reducing the time spent by a system expert on parameter tuning. Our approach has two main components: (1) a configuration tree structure that organizes system parameters and procedures in a systematic manner and facilitates identifying viable system configurations; and (2) a probabilistic sampling approach on the configuration tree that efficiently searches for the optimal configuration. We have successfully applied the proposed approach to optimize the configurations for two different video processing modules: a motion detection & tracking pipeline and a streaming video segmentation algorithm. The automatically discovered configurations produced better performance than the best manual configurations, while requiring significantly less human effort and domain-specific expertise.

We present a real-time, full-frame, multi-target Wide Area Motion Imagery (WAMI) tracking system that utilizes distributed processing to handle high data rates while maintaining high track quality. The proposed architecture processes the WAMI data as a series of geospatial tiles and implements both process- and thread-level parallelism across multiple compute nodes. Each tile is processed independently, from decoding the image through generating tracks that are finally merged across all tiles by an inter-tile linker (ITL) module. A high performance PostgreSQL database with GIS extensions is used to control the flow of intermediate data between each tracking process. High quality tracks are produced efficiently due to robust, effective algorithmic modules including: multi-frame moving object detection and track initialization; tracking based on the fusion of motion and appearance with a goal of very pure tracks; and online track linking based on multiple features. In addition, we have configured a high-performance compute cluster using high density blade servers, Infiniband networking, and an HPC filesystem. The compute cluster enables full-frame, state-of-the-art tracking of vehicles or dismounts at the WAMI sensor's native 1.25Hz frame-rate, while only taking 7u of rack space and providing 210 megapixels/second throughput.

We develop a localization method to enable a team of mobile robots to search for multiple unknown transient radio sources. Because of signal source anonymity, short transmission durations, and dynamic transmission patterns, robots cannot treat the radio sources as continuous radio beacons. Moreover, robots do not know the source transmission power and have limited sensing ranges. To cope with these challenges, we pair up robots and develop a cooperative sensing model using signal strength ratios from the paired robots. We formally prove that the joint conditional posterior probability of source locations for the m-robot team can be obtained by combining the pairwise joint posterior probabilities, which can be derived from signal strength ratios. Moreover, we propose a pairwise ridge walking algorithm (PRWA) to coordinate the robot pairs based on the clustering of high-probability regions and the minimization of local Shannon entropy. We have implemented and validated the algorithm under both the hardware-driven simulation and physical experiments. Experimental results show that the PRWA-based localization scheme consistently outperforms the other four heuristics.

In this work, we present a framework to detect objects embedded in complex perspective geometry. Our goal is to accurately identify objects such as people standing in balconies or windows on building facades of surrounding buildings. Compared to traditional computer vision work focused on activity analysis from a horizontal view, our framework provides a solution for the application domain of mobile surveillance in urban areas. A novel solution for a monocular camera is formulated by tightly coupling various computational modules including geometric analysis, segmentation, scale estimation, and object detection. In particular, our proposed approach alleviates the effect of the perspective geometry and corresponding distortion in object appearance effectively, and provides accurate scale priors to eliminate unlikely object detection hypotheses. The experimental results on collected video dataset show that the proposed approach is more accurate than traditional detection approaches based on brute-force scanning windows.

We present a novel vanishing point detection algorithm for uncalibrated monocular images of man-made environments. We advance the state-of-the-art by a new model of measurement error in the line segment extraction and minimizing its impact on the vanishing point estimation. Our contribution is twofold: 1) Beyond existing hand-crafted models, we formally derive a novel consistency measure, which captures the stochastic nature of the correlation between line segments and vanishing points due to the measurement error, and use this new consistency measure to improve the line segment clustering. 2) We propose a novel minimum error vanishing point estimation approach by optimally weighing the contribution of each line segment pair in the cluster towards the vanishing point estimation. Unlike existing works, our algorithm provides an optimal solution that minimizes the uncertainty of the vanishing point in terms of the trace of its covariance, in a closed-form. We test our algorithm and compare it with the state-of-the-art on two public datasets: York Urban Dataset and Eurasian Cities Dataset. The experiments show that our approach outperforms the state-of-the-art.

We develop a localization method enabling a team of mobile robots to search for multiple unknown transient radio sources. Due to signal source anonymity, short transmission durations, and dynamic transmission patterns, robots cannot treat the radio sources as continuous radio beacons. Moreover, robots do not know the source transmission power and have limited sensing ranges. To cope with these challenges, we pair up robots and develop a sensing model using the signal strength ratio from the paired robots. We formally prove that the sensed conditional joint posterior probability of source locations for the m-robot team can be obtained by combining the pairwise joint posterior probabilities, which can be derived from signal strength ratios. Moreover, we propose a pairwise ridge walking algorithm (PRWA) to coordinate the robot pairs based on the clustering of high probability regions and the minimization of local Shannon entropy. We have implemented and validated the algorithm under hardware-driven simulation.

We report a new filter for assisting the search for rare bird species. Since a rare bird only appears in front of the camera with very low occurrence (e.g. less than ten times per year) for very short duration (e.g. less than a fraction of a second), our algorithm must have very low false negative rate. We verify the bird body axis information with the known bird flying dynamics from the short video segment. Since a regular extended Kalman filter (EKF) cannot converge due to high measurement error and limited data, we develop a novel Probable Observation Data Set (PODS)-based EKF method. The new PODS-EKF searches the measurement error range for all probable observation data that ensures the convergence of the corresponding EKF in short time frame. The algorithm has been extensively tested in experiments. The results show that the algorithm achieves 95.0% area under ROC curve in physical experiment with close to zero false negative rate.

We report a new filter to assist the search for rare bird species. Since a rare bird only appears in front of a camera with very low occurrence (e.g., less than ten times per year) for very short duration (e.g., less than a fraction of a second), our algorithm must have a very low false negative rate. We verify the bird body axis information with the known bird flying dynamics from the short video segment. Since a regular extended Kalman filter (EKF) cannot converge due to high measurement error and limited data, we develop a novel probable observation data set (PODS)-based EKF method. The new PODS-EKF searches the measurement error range for all probable observation data that ensures the convergence of the corresponding EKF in short time frame. The algorithm has been extensively tested using both simulated inputs and real video data of four representative bird species. In the physical experiments, our algorithm has been tested on rock pigeons and red-tailed hawks with 119 motion sequences. The area under the ROC curve is 95.0%. During the one-year search of ivory-billed woodpeckers, the system reduces the raw video data of 29.41 TB to only 146.7 MB (reduction rate 99.9995%).

We report an autonomous observation system with multiple pan-tilt-zoom (PTZ) cameras assisted by a fixed wide-angle camera. The wide-angle camera provides large but low resolution coverage and detects and tracks all moving objects in the scene. Based on the output of the wide-angle camera, the system generates spatiotemporal observation requests for each moving object, which are candidates for close-up views using PTZ cameras. Due to the fact that there are usually much more objects than the number of PTZ cameras, the system first assigns a subset of the requests/objects to each PTZ camera. The PTZ cameras then select the parameter settings that best satisfy the assigned competing requests to provide high resolution views of the moving objects. We solve the request assignment and the camera parameter selection problems in real time. The effectiveness of the proposed system is validated in comparison with an existing work using simulation. The simulation results show that in heavy traffic scenarios, our algorithm increases the number of observed objects by over 200%.

We report our work on the algorithmic development of an evolutionary methodology for automatic configuration of metaheuristic algorithms for solving complex combinatorial optimization problems.We term it Automatic Configuration Engine for Metaheuristics (ACEM). We first propose a novel Left Variation - Right Property (LVRP) tree structure to manage various metaheuristic procedures and properties. With LVRP tree, feasible configurations of metaheuristics can be easily specified. An evolutionary learning algorithm is then proposed to evolve the internal context of the trees based on pre-selected training set. Guided by a user-defined satisfaction function of the candidate algorithms, it converges to the optimal or a very good algorithm. The experimental comparison with two recent state-of-the-art algorithms for solving the quadratic assignment problem (QAP) shows that ACEM produces an hybrid-genetic algorithm with human-competitive or even better performance.

We report our algorithmic development of the pframe problem that addresses the need of coordinating a set of p networked robotic pan-tilt-zoom cameras for n, (n ≫ p), competing polygonal requests. We assume that the p frames have almost no overlap on the coverage between frames and a request is satisfied only if it is fully covered. We then propose a Resolution Ratio with Non-Partial Coverage (RRNPC) metric to quantify the satisfaction level for a given request with respect to a set of p candidate frames. We propose a latticebased approximation algorithm to search for the solution that maximizes the overall satisfaction. The algorithm builds on an induction-like approach that finds the relationship between the solution to the (p — 1)-frame problem and the solution to the p-frame problem. For a given approximation bound ε, the algorithm runs in O(n/ε3+p2/ε6) time. We have implemented the algorithm and experimental results are consistent with our complexity analysis.

In recent decades, many meta-heuristics, including genetic algorithm (GA), ant colony optimization (ACO) and various local search (LS) procedures have been developed for solving a variety of NP-hard combinatorial optimization problems. Depending on the complexity of the optimization problem, a meta-heuristic method that may have proven to be successful in the past might not work as well. Hence it is becoming a common practice to hybridize meta-heuristics and local heuristics with the aim of improving the overall performance. In this paper, we propose a novel adaptive GA-ACO-LS hybrid algorithm for solving quadratic assignment problem (QAP). Empirical study on a diverse set of QAP benchmark problems shows that the proposed adaptive GA-ACO-LS converges to good solutions efficiently. The results obtained were compared to the recent state-of-the-art algorithm for QAP, and our algorithm showed obvious improvement.