3D Semantic Scene Completion (3D SSC), aiming to infer complete 3D scene geometry and semantics from partial observations, is crucial for autonomous driving and robotics. Recent progress in SSC has leveraged the Transformer architecture, while UNet has shown strong multi-scale feature aggregation capabilities. Building on these, we propose MVFormer, integrating the strengths of both architectures to predict the dense geometry and semantics of a scene from 2D images. MVFormer utilizes depth information to separate 2D image features, improving object representation at varying distances, and providing rich geometric and semantic information for voxel initialization. We also introduce a UNet-like decoder with a novel Mix-Voxel attention mechanism, seamlessly integrating Transformer into the UNet structure. This decoder uses multi-scale information to guide voxel feature updates, enhancing multi-scale feature capture. Experiments on SemanticKITTI, SSCBench-KITTI-360 and NYUv2 show that MVFormer achieves state-of-the-art performance, with mIoU scores of 15.81%, 18.94% and 32.96% respectively, while significantly reducing the computational complexity of the model. The code will be available at https://github.com/Ricardovvvv/MVFormer.

To enhance 3D object detection in autonomous driving, recent work combines LiDAR and camera data. However, prior methods often suffer from inadequate image depth information and fixed-weight fusion strategies, limiting semantic extraction and adaptability. PVF-DectNet++ builds on our prior work by employing a perspective voxel projection technique to align both feature types. It introduces an adaptive image semantic feature extraction approach that interpolates image and point cloud intensity into a dense RGB-I multi-channel representation, facilitating the extraction of global, multi-level image features. Furthermore, during the fusion process, a learnable fusion module is designed to address the challenge of individual channels being unable to adapt to varying appearances, colors, and environmental conditions. Experiments on KITTI, nuScenes, and Waymo comprehensively validate PVF-DectNet++. On KITTI, it achieves detection accuracies of 66.3% for pedestrians, 78.8% for cyclists, and 86.8% for vehicles, yielding a 3.56% mAP improvement over PVF-DectNet. Additional tests show further gains, with mAP and NDS increases of 3.8% and 2.6% on nuScenes, and notable boosts in pedestrian and cyclist AP on Waymo. Compared with existing networks, PVF-DectNet++ consistently delivers superior performance, particularly for pedestrian and cyclist detection across diverse benchmarks. The code and model will be released at https://github.com/CQU-AVL/PVF-DectNet-.

Cross-domain detection frequently encounters a decline in detection accuracy, necessitating the application of domain adaptation techniques. One crucial approach to unsupervised domain adaptation is the pseudo label-based self-training method, which iteratively trains the model by treating the pseudo labels as ground truth. However, differences in distribution that can exist between the source and target domains can lead to potentially incorrect generated pseudo labels. This can result in the threshold-setting method failing to accurately select the pseudo labels. Therefore, to tackle the challenge of determining pseudo label thresholds in self-training, we propose an unsupervised 3D object detection domain adaptation method based on pseudo label regularization. Specifically, a self-training framework based on the fusion of two detection heads is used to obtain more accurate pseudo labels. The variance of the two detection heads is utilized as the noise information for the corresponding pseudo labels. Then, the noise information is incorporated as a regularization term to enhance the bounding box regression loss, thereby addressing the challenge of determining pseudo label thresholds in self-training. The experimental results demonstrate that the method proposed in this paper achieves higher cross-domain detection accuracy compared to existing domain adaptation methods for 3D object detection.

Precise environmental perception and reliable real-time localization are crucial for achieving advanced driver assistance functions. Against this backdrop, Simultaneous Localization and Mapping (SLAM), with its unique advantages, has become one of the indispensable key technologies in the field of autonomous driving (AD). However, as the traffic environment for autonomous vehicles (AVs) continues to become increasingly complex and diverse, such as urban streets, highways, and adverse weather conditions, these challenges pose higher demands on vehicles’ localization and mapping capabilities, also bring new opportunities and challenges for further optimization and application of SLAM. In this article, we conduct a comprehensive and in-depth analysis of the current research status and applications of SLAM in complex scenes of AD. First, we explore the challenges faced by AVs in various complex scenes, including the interference of dynamic objects, the precise localization and mapping requirements in large-scale scenes, as well as variable environmental and weather conditions. Subsequently, we delve into the coping strategies and methods of SLAM in these specific scenes. Finally, we compare and summarize the datasets related to SLAM in complex scenes of AD and point out some potential research directions for achieving high-level AD. We hope that this study will track and update the latest progress in SLAM for AVs. To promote the development of the open source community and future academic research, we created a repositoryhttps://github.com/herofly1/CQU-AVL-SLAMthat provides related review papers and methodological resources.

Deep learning is well used in the field of autonomous driving, but systems often encounter scenarios that are significantly different from their training data, known as out-of-distribution (OOD) scenarios. In the realm of autonomous driving safety, perception is the most crucial and complex component, with existing perception network frameworks still suffering from inadequate accuracy, compounded by the plethora of uncertain factors and OOD scenarios in real road environments. However, previous research mainly focused on uncertainty methods and OOD detection techniques, lacking comprehensive reviews that cover their origins in the context of perception safety, theoretical analyses, evaluation metrics, and their applicability in ensuring autonomous driving safety. This review analyzes the OOD and uncertainty issues faced in the autonomous driving perception domain, including some cause analyses and solution s to these issues. Additionally, the review discusses OOD detection evaluation metrics that meet the safety requirements of autonomous driving. Thirdly, it summarizes existing OOD detection methods and explores their strengths and weaknesses under autonomous driving safety with evaluation metrics. Lastly, the article analyzes the application of uncertainty techniques in OOD detection, including the shortcomings and categorization of uncertainty methods, spatial feature utilization, semantic information, and performance optimization techniques. The review also highlights the shortcomings and improvement directions of uncertainty quantification techniques in OOD detection, along with the potential for further integration with autonomous driving scenarios. This study aims to promote the research and application of uncertainty quantification and OOD detection technologies in the context of autonomous driving safety. To facilitate future research, we create a repository that includes links to relevant reviews and methodological code for learning at https://github.com/sotif-ma/OOD-Detection-Methods-and-Datasets

In the past decade, deep learning has been widely applied across various fields. However, its applicability in open-world scenarios is often limited due to the lack of quantifying uncertainty in both data and models. In recent years, a multitude of uncertainty quantification (UQ) approaches for neural networks have emerged and found applications in safety-critical domains such as autonomous vehicles and medical analysis. This paper aims to review the latest advancements in UQ methods and investigate their application specifically in the field of computer vision and autonomous vehicles. Initially, we identify several key qualifications, namely practicability, robustness, accuracy, scalability, and efficiency (referred to as PRASE), and employ them as evaluation criteria throughout this study. By considering these criteria as uniform measurements, we meticulously evaluate and compare the performance of different types of UQ methods, including Bayesian methods, ensemble methods, and single deterministic methods. Furthermore, we delve into the discussion of their application in diverse tasks within the autonomous vehicle domain, such as semantic segmentation, object detection, depth estimation, and end-to-end control. Through comprehensive analysis and comparison, we identify a range of challenges and propose future research directions in this field. Our findings shed light on the importance of addressing uncertainty quantification in deep learning models and provide insights into enhancing the reliability and performance of autonomous vehicles in real-world scenarios.

The traditional object detection algorithm in the intelligent vehicle perception system cannot maintain stable recognition performance in the unknown and changing road environment. We find that uncertainty quantification is of great significance in detecting unknown complex environments and helps to improve the robustness and safety of autonomous driving systems. Therefore, this paper proposes an Uncertaintybased Transformer (UBT) object detection algorithm. Firstly, the double Gaussian feature map network (DGF) is designed to quantify and utilize the uncertainty of the features derived from the backbone network. Secondly, we propose a RBFbased query filtering model(RBQF), which takes uncertainty sum as the index of query vector screening. At the same time, this paper proposes an uncertainty detection head (UDH); the final model output results are quantitative uncertainty, improved detection performance and enhanced algorithm reliability. To further prove the detection performance of the proposed method in real driving scenes, we use COCO, Cityscapes, FoggyCi-tyscapes, RainCityscapes and self-made traffic scene datasets for verification, which shows that our algorithm is well applicable to large datasets and complex road scenes. Our codebase and pre-trained models can be accessed at https://github.com/sotifma/UBT/tree/master

In the context of automated driving, navigating through challenging urban environments with dynamic objects, large-scale scenes, and varying lighting/weather conditions, achieving accurate localization is paramount for highly-automated (HAVs) or autonomous vehicles (AVs). An imprecise localization can greatly impact subsequent decision-making to manage an HAV or AV’s motion (planning and control tasks). In recent years, visual simultaneous localization and mapping (VSLAM) has shown substantial progress and equipping it can lead to handling non-standardized situations of real-world scenes and achieving higher localization and mapping accuracy. In this article, we present a comprehensive analysis of the current research status of VSLAM and its potential application to HAV or AV operating in complex urban environments. We first discuss the criteria to assess how well for the solutions that VSLAM methods offer to address the challenges, which include real-time performance, accuracy, robustness, and system operating cost. By employing these assessment criteria, we evaluate various VSLAM methods in four essential aspects including rejection and tracking of high dynamic objects, map construction in large-scale environments, loop detection and error correction, and sustainable operation and map updating. This evaluation provides valuable insights into the effectiveness of different VSLAM techniques. We then discuss potential research directions for leveraging VSLAM methods in achieving high-level automated driving in complex settings. We hope this article to serve as a timely update on recent progress and advances in VSLAM which are applicable to HAVs or AVs. To facilitate future research, we create a repository that includes links to relevant reviews and methodological papers for learning at https://github.com/bumblebee15138/VSLAM for HAVs and AVs.

In future mobility environment, virtual drivers of autonomous vehicles should be monitored for the sake of safety by evaluating their driving behaviors. Evaluating human drivers of non-autonomous vehicles also can be helpful to improve performance of safety control systems. This paper evaluates driving behaviors based on transition probabilities among multiple driving modes. We estimate transition probabilities with likelihoods of multiple modes from an interacting multiple model by proposing an online estimation approach. The proposed approach addresses the numerical issue found in our preliminary work, and it is verified with an extensive simulation. Furthermore, we evaluate driving behaviors by utilizing the estimated transition probabilities. The proposed method of driving behavior evaluation is demonstrated experimentally.

This paper addresses the trajectory planning problem for autonomous vehicles in traffic. We build a stochastic Markov decision process (MDP) model to represent the behaviors of the vehicles. This MDP model takes into account the road geometry and is able to reproduce more diverse driving styles. We introduce a new concept, namely, the “dynamic cell,” to dynamically modify the state of the traffic according to different vehicle velocities, driver intents (signals), and the sizes of the surrounding vehicles (i.e., truck, sedan, and so on). We then use Bézier curves to plan smooth paths for lane switching. The maximum curvature of the path is enforced via certain design parameters. By designing suitable reward functions, different desired driving styles of the intelligent vehicle can be achieved by solving a reinforcement learning problem. The desired driving behaviors (i.e., autonomous highway overtaking) are demonstrated with an in-house developed traffic simulator.

Performance of vehicle control systems, such as active safety systems and driver assistance systems, can be significantly improved by taking driver behavior information into consideration. This paper implements a handling limit-based algorithm for driver behavior characterization by introducing stochastic perspective with the interacting multiple model (IMM) estimation theory. The proposed algorithm constructs mathematical models for four vehicle dynamics categories. The IMM estimator is designed for each vehicle dynamics category to evaluate driver scores. The proposed algorithm is compared with an existing handling limit-based algorithm through experimental tests, and the results illustrate advantages of the proposed algorithm.

Given a route to a destination, a mobile robot still needs to locally plan a path to avoid collisions in continuously changing environment, e.g., a hall with pedestrians and moving obstacles. In this paper, diffusion maps are applied to find a local path for reaching a goal and avoiding collisions simultaneously. The proposed path planning algorithm plans a local path by utilizing a receding horizon approach, and therefore the algorithm repeats planning at every sample time. With this approach, mobile robots do not have to carry a prior map all the time because updated environment information is used for planning at every sample time. Extensive simulation is performed in different scenarios and demonstrates a good performance in collision avoidance.

This paper studies the decision making problem of autonomous vehicles in traffic. We model the interaction between an autonomous vehicle and the environment as a stochastic Markov decision process (MDP) and consider the driving style of an experienced driver as the target to be learned. The road geometry is taken into consideration in the MDP model in order to incorporate more diverse driving styles. By designing the reward function of the MDP, the desired, driving behavior of the autonomous vehicle is obtained using reinforcement learning. Simulated results demonstrate the desired driving behaviors of an autonomous vehicle.

This paper presents a robotic system implementation featuring an autonomy solution that incorporates human intents through effective human-robot interactions. More specifically, vision-based human pose estimation is used to identify gestures that correspond to human intents. These intents are mapped to predefined commands that schedule autonomous operations of the robotic system. The paper presents an overview of an autonomy solution for mobile robots, an approach to human-following operation, and an approach to automated decision-making for parking operations. Experimental results of the physical robot implementation are presented.

Evaluating driving performance of autonomous vehicles is as important as developing automated driving algorithms. In order to ensure passenger safety, evaluation of driving behavior is required before delivering autonomous vehicles to customers. An Interacting Multiple Model (IMM)-based driver evaluation algorithm was developed and it provides various information associated with multiple driving aggressiveness modes. This paper estimates transition probabilities by utilizing those information from the IMM-based evaluation algorithm, which are expected to unveil hidden driving performance of autonomous vehicles. Three estimation approaches are presented and they are tested with experimental drives.

This paper investigates a comfort-based route planner that considers both travel time and ride comfort. We first present a framework of simultaneous road profile estimation and anomaly detection with commonly available vehicle sensors. A jump-diffusion process-based state estimator is developed and used along with a multi-input observer for road profile estimation. The estimation framework is evaluated in an experimental test vehicle and promising performance is demonstrated. Second, three objective comfort metrics are developed based on factors such as travel time, road roughness, road anomaly, and intersection. A comfort-based route planning problem is then formulated with these metrics and an extended Dijkstra's algorithm is exploited to solve the problem. A cloud-based implementation of our comfort-based route planning approach is proposed to facilitate information access and fast computation. Finally, a real-world case study, comfort-based route planning from Ford Research and Innovation Center, Michigan to Ford Rouge Factory Tour, Michigan, is presented to illustrate the efficacy of the proposed route planning framework.

In the development of advanced driver-assist systems (ADAS) for lane-keeping or cornering, one important design objective is to appropriately share the steering control with the driver. The steering behavior of the driver must therefore be well characterized for the design of a high-performance ADAS controller. This paper adopts the well-known two-point visual driver model to characterize the steering behavior of the driver, and conducts a series of field tests to identify the model parameters and validate this model in real-world scenarios. An extended Kalman filter and an unscented Kalman filter are implemented for estimating the driver parameters using either a joint-state estimation algorithm or a dual estimation algorithm. The estimated parameters for different types of drivers are analyzed and compared. The results show that the two-point visual driver model captures realistic driving behavior with time-varying, but not necessarily constant, parameters. A wavelet analysis of the driver steering command shows that distinct driver classes can be identified by analyzing the smoothness of the driver command using the Lipschitz exponents of the recorded signals.

This paper considers a new clustering algorithm for processing time-evolving road anomaly reports. Two cluster categories, main and outlier, are defined to deal with outliers as well as to capture the evolving nature of road anomalies. The Mahalanobis distance is exploited to quantify the similarity between a new report and the existing clusters. The clusters are maintained online and the Woodbury matrix inverse lemma is used for their recursive updates. The proposed clustering algorithm can localize isolated anomalies and compress information for densely distributed anomalies. A simulation is presented to demonstrate the efficacy of the proposed algorithm.

Incorporating driver behavior information into vehicle control strategies can significantly improve performance of vehicle control systems, such as active safety systems and driver assistance systems. This paper proposes an algorithm for driver behavior characterization based on handling limits. In order to implement the handling limit-based algorithm, an interacting multiple model estimation technique is applied, which accounts for probabilistic correctness of the multiple models. The proposed algorithm is validated through experimental tests, and the results illustrate potential of the proposed algorithm as a stochastic approach for driver behavior characterization based on handling limits.

In the development of advanced driver-assist systems (ADAS) for lane-keeping, one important design objective is to appropriately share the steering control with the driver. Hence, the steering behavior of the driver must be well known beforehand. This paper adopts the well-known two-point visual driver model to characterize the steering behavior of the driver, and conducts a series of field tests to identify the model parameters to validate the two-point visual driver model in real scenarios. Both an extended Kalman filter and an unscented Kalman filter are implemented for estimating the unknown driver parameters, using a joint-state estimation algorithm and a dual estimation algorithm, and the results are compared.

This paper presents a safety-based route planner that exploits vehicle-to-cloud-to-vehicle (V2C2V) connectivity. Time and road risk index (RRI) are considered as metrics to be balanced based on user preference. To evaluate road segment risk, a road and accident database from the highway safety information system is mined with a hybrid neural network model to predict RRI. Real-time factors such as time of day, day of the week, and weather are included as correction factors to the static RRI prediction. With real-time RRI and expected travel time, route planning is formulated as a multiobjective network flow problem and further reduced to a mixed-integer programming problem. A V2C2V implementation of our safety-based route planning approach is proposed to facilitate access to real-time information and computing resources. A real-world case study, route planning through the city of Columbus, Ohio, is presented. Several scenarios illustrate how the "best" route can be adjusted to favor time versus safety metrics.

This paper concerns energy harvesting from vehicle suspension systems. Theoretical and experimental presentation of energy harvesting from suspension is explored. The generated power associated with random excitation inputs to vehicle is determined. The potential values of power generation from various road conditions are calculated. The effect of suspension energy regeneration on the ride comfort and road handling is presented. This includes study of tire deflection, damping, and spring stiffness for a regenerative suspension. The concept of self-powered actuation using the harvested energy from suspension is discussed with regards to applications of self-powered vibration control.

This paper proposes a novel multi-objective route planning approach within the framework of a Vehicle-to-Cloud-to-Vehicle (V2C2V) architecture. Time and road risk index (RRI) are both considered as metrics. To evaluate road segment risk, an accident database from the Highway Safety Information System (HSIS) is processed to build a comprehensive road risk assessment model. Route planning is formulated as a multi-objective network flow problem and further reduced to a Mixed Integer Programming (MIP) problem. A real-world case study, route planning through the city of Columbus, Ohio, is presented. The Vehicle-to-Cloud-to-Vehicle (V2C2V) based implementation of our safety-based route planning approach is proposed to facilitate access to real-time information and computing resources.

This paper considers an electromagnetic regenerative suspension system (ERSS) that recovers the kinetic energy originated from vehicle vibration, which is previously dissipated in traditional shock absorbers. It can also be used as a controllable damper that can improve the vehicle's ride and handling performance. The proposed electromagnetic regenerative shock absorbers (ERSAs) utilize a linear or a rotational electromagnetic generator to convert the kinetic energy from suspension vibration into electricity, which can be used to reduce the load on the alternator so as to improve fuel efficiency. A complete ERSS is discussed here that includes the regenerative shock absorber, the power electronics for power regulation and suspension control, and an electronic control unit (ECU). Different shock absorber designs are proposed and compared for simplicity, efficiency, energy density, and controlled suspension performances. Both simulation and experiment results are presented and discussed.

This paper studies characterizing the driving behavior during steady-state and transient car-following. An approach utilizing the online learning of an evolving Takagi-Sugeno fuzzy model that is combined with a probabilistic model is applied to capture the multi-model and evolving nature of the driving behavior. The approach is validated by testing on a vehicle during different driving conditions.

Advanced vehicle cabin technologies provide drivers infotainment, navigation, and enhanced convenient driving experiences. As interaction between the driver and in cabin technologies increases it is beneficial to provide tailored driver communication for an improved cabin experience. Assessment of the driving demand is of particular value to assist in modulating communication and vehicle system interactions with the driver. The complex vehicle, driver, and environment driving contexts require innovative prognostic approaches to driver workload inference. This paper presents a Hybrid Intelligent System for Driver Workload Estimation (HWLE). The real-time HWLE soft computing modules incorporate expert models, model-driven reasoning, and specialized computational intelligence techniques to compute an aggregated WLE-Index. The context depended WLE-Index facilitates tailoring vehicle-driver communication and interaction based on the driving demand. Results from application of the HWLE system under real-time conditions are presented.

This paper studies to characterize driver driving behavior or driver control structure in real time. The three proposed methods use some of the signals such as the driver actuation measurements, the relative ranges between a leading and a following vehicle during a car-following maneuver, and the vehicle dynamic responses such as the vehicle's longitudinal acceleration and deceleration. All the used signals exist in various electronic control systems. Vehicle tests were conducted on a test vehicle to illustrate the effectiveness of the proposed methods in identifying aggressive and cautious driving behaviors.

This paper analyzes a promising method of road grade estimation for its potential use in aiding ground vehicle electronic stability control systems. The method in question incorporates GPS and motion sensors into a basic Kalman Filter model to achieve clean, high update estimates of the vertical and longitudinal velocity states from which the grade can be easily calculated. The knowledge of road grade is incorporated into a sideslip estimation scheme, replacing the pitch rate sensor, and the improvement in the sideslip estimate is evaluated. Simulated results are shown, as are results from using experimental data comparing this sideslip estimate with one obtained which utilizes the pitch rate gyro.