The depth completion task is a critical problem in autonomous driving, involving the generation of dense depth maps from sparse depth maps and RGB images. Most existing methods employ a spatial propagation network to iteratively refine the depth map after obtaining an initial dense depth. In this paper, we propose DenseFormer, a novel method that integrates the diffusion model into the depth completion task. By incorporating the denoising mechanism of the diffusion model, DenseFormer generates the dense depth map by progressively refining an initial random depth distribution through multiple iterations. We propose a feature extraction module that leverages a feature pyramid structure, along with multi-layer deformable attention, to effectively extract and integrate features from sparse depth maps and RGB images, which serve as the guiding condition for the diffusion process. Additionally, this paper presents a depth refinement module that applies multi-step iterative refinement across various ranges to the dense depth results generated by the diffusion process. The module utilizes image features enriched with multi-scale information and sparse depth input to further enhance the accuracy of the predicted depth map. Extensive experiments on the KITTI outdoor scene dataset demonstrate that DenseFormer outperforms classical depth completion methods.

Controlling mixed platoons, which consist of both connected and automated vehicles (CAVs) and human-driven vehicles (HDVs), poses significant challenges due to the uncertain and unknown human driving behaviors. Data-driven control methods offer promising solutions by leveraging available trajectory data, but their performance can be compromised by noise and attacks. To address this issue, this paper proposes a Robust Data-EnablEd Predictive Leading Cruise Control (RDeeP-LCC) framework based on data-driven reachability analysis. The framework over-approximates system dynamics under noise and attack using a matrix zonotope set derived from data, and develops a stabilizing feedback control law. By decoupling the mixed platoon system into nominal and error components, we employ data-driven reachability sets to recursively compute error reachable sets that account for noise and attacks, and obtain tightened safety constraints of the nominal system. This leads to a robust data-driven predictive control framework, solved in a tube-based control manner. Human-in-the-loop experiments demonstrate that theRDeeP-LCCmethod significantly improves robustness against noise and attacks, while enhancing tracking accuracy, control efficiency, energy economy, driving comfort, and driving safety.

Safety is one of the most crucial challenges of autonomous driving vehicles, and one solution to guarantee safety is to employ an additional control revision module after the planning backbone. Control Barrier Function (CBF) has been widely used because of its strong mathematical foundation on safety. However, the incompatibility with heterogeneous perception data and incomplete consideration of traffic scene elements make existing systems hard to be applied in dynamic and complex real-world scenarios. In this study, we introduce a generalized control revision method for autonomous driving safety, which adopts both vectorized perception and occupancy grid map as inputs and comprehensively models multiple types of traffic scene constraints based on a new proposed barrier function. Traffic elements are integrated into one unified framework, decoupled from specific scenario settings or rules. Experiments on CARLA, SUMO, and OnSite simulator prove that the proposed algorithm could realize safe control revision under complicated scenes, adapting to various planning backbones, road topologies, and risk types. Physical platform validation also verifies the real-world application feasibility.

Visual bird’s eye view (BEV) perception, dute to its excellent perceptual capabilities, is progressively replacing costly LiDAR-based perception systems, especially in the realm of urban intelligent driving. However, this type of perception still relies on LiDAR data to construct ground truth databases, a process that is both cumbersome and time-consuming. Additionally, most mass-produced autonomous driving systems are equipped solely with surround camera sensors and lack the LiDAR data necessary for precise annotation. To tackle this challenge, we propose a fine-tuning method for BEV perception network based on visual 2D semantic perception, aimed at enhancing the model’s generalization capabilities in new scene data. Leveraging the maturity of 2D perception technologies, our method utilizes only 2D semantic segmentation labels and monocular depth estimations, thereby significantly reducing the dependence on expensive BEV ground truths and offering strong potential for industrial deployment. Extensive experiments and comparative analyses on the nuScenes and Waymo datasets demonstrate the effectiveness of our method. Specifically, it improves mAP and NDS by 2.51% and 1.93% on nuScenes, and by 1.21% and 0.78% on Waymo, respectively, validating its practical utility and robustness across diverse domains.

In V2X collaborative perception, the domain gaps between heterogeneous nodes pose a significant challenge for effective information fusion. Pose errors arising from latency and GPS localization noise further exacerbate the issue by leading to feature misalignment. To overcome these challenges, we propose V2X-DGPE, a high-accuracy and robust V2X feature-level collaborative perception framework. V2X-DGPE employs a Knowledge Distillation Framework and a Feature Compensation Module to learn domain-invariant representations, effectively reducing the feature distribution gap between vehicles and roadside infrastructure. Historical information is utilized to provide the model with a more comprehensive understanding of the current scene. Furthermore, a Collaborative Fusion Module leverages a heterogeneous self-attention mechanism to extract and integrate heterogeneous representations from vehicles and infrastructure. To address pose errors, V2X-DGPE introduces a deformable attention mechanism, enabling the model to capture feature misalignments by dynamically offsetting sampling points. Extensive experiments on the real-world DAIR-V2X dataset demonstrate that the proposed method outperforms existing approaches, achieving state-of-the-art detection performance. The code is available at https://github.com/wangsch10/V2X-DGPE.

Optimizing mixed vehicle platoons, which consist of connected and automated vehicles (CAVs) with human-driven vehicles (HDVs), is a critical challenge for intelligent transportation systems. While existing predictive control methods have improved modeling accuracy and control robustness, they are often constrained by their reliance on online optimization, limiting their applicability in real-time scenarios. To address this gap, this paper proposes a Robust Explicit Data-Driven Predictive Control (REDDPC) framework designed to provide robust and real-time control for mixed vehicle platoons. The framework begins by utilizing a deep Koopman operator network to learn the nonlinear dynamics of the system. Using this learned representation, the neural network-based control policy is then optimized through backpropagation, eliminating the need for online optimization. To enhance robustness, a reachability-based safety filter is integrated with the learned control policy to dynamically adjust control inputs, ensuring platoon safety under complex conditions. Simulation and experiment results demonstrate that the proposed method achieves superior tracking performance under noise, disturbance, and attack conditions, while significantly reducing online computational time, making it highly suitable for real-world deployment.

Coordinating connected and automated vehicles (CAVs) poses challenges in achieving safe and optimal pathfinding simultaneously. In this paper, we propose safer conflict-based search (Safer-CBS), an approach based on conflict-based search that generates risk-constrained optimal paths for multiple CAVs. Safer-CBS addresses these challenges by incorporating risk assessment and optimization techniques. By defining the conflict as the risk between two vehicles exceeding a predetermined threshold, the resultant conflict-free paths can provide safety guarantees. An optimization over risk constraints is devised to enhance the efficiency of the tree search. Additional improvements are also incorporated to expedite the search process. Theoretical analysis demonstrates the optimality of Safer-CBS in finding risk-constrained paths. Furthermore, Safer-CBS can seamlessly integrate trajectory planning to generate risk-constrained trajectories for CAVs. Numerical calculations on grid maps validate the effectiveness of the proposed optimization and additional improvements, resulting in increased success rates (by over 43% for 10 vehicles) and reduced runtime (by over 59%) when compared to the method without the optimization and improvements for Safer-CBS. Simulations in both unstructured and structured environments confirm that Safer-CBS, combined with trajectory planning, ensures that trajectory risks strictly remain below predetermined thresholds, while achieving a superior balance between safety and efficiency compared to the benchmarks.Note to Practitioners—Applying multi-CAV technologies in areas such as warehousing, transportation, and operations typically requires the planning of cooperative paths or trajectories for the CAVs. In this planning process, safety and efficiency issues are centrally considered and balanced. Cooperative performance can be ensured as long as each individual vehicle adheres to the central instructions. This study develops a method for risk-constrained cooperative planning, aiming to generate optimal paths and suboptimal trajectories for multiple CAVs. The output trajectories consist of path points with fixed time intervals, which the vehicle controllers can track accordingly. By maintaining the risk level among the vehicles below predetermined thresholds, overall safety can be achieved. Meanwhile, the planning process is optimized, thus improving overall efficiency.

The autonomous driving system faces challenges in selecting critical targets under dense environments with limited computation resources. Existing rule-based methods struggle with complex scenarios, while learning-based approaches lack interpretability and safety. This paper proposes a hybrid target selection model combining a lightweight long short-term memory (LSTM) based deep learning classifier and rule-based methods. Key input features are identified and processed to enhance training. The LSTM model is validated for accuracy and efficiency against bidirectional LSTM (Bi-LSTM) variations. Compared to the single approach, the hybrid model integrates the LSTM classifier and rule-based methods with a synthesizer, demonstrating improved accuracy, better interpretability, and a potentially higher functional safety level. Integrated into TDA4VM, the hybrid model shows timely and complementary target selection performance in actual urban and highway tests with low computation costs, proving its theoretical value and engineering prospects.

Bird’s-eye-view (BEV) semantic segmentation is becoming crucial in autonomous driving systems. It realizes ego-vehicle surrounding environment perception by projecting 2D multi-view images into 3D world space. Recently, BEV segmentation has made notable progress, attributed to better view transformation modules, larger image encoders, or more temporal information. However, there are still two issues: 1) a lack of effective understanding and enhancement of BEV space features, particularly in accurately capturing long-distance environmental features and 2) recognizing fine details of target objects. To address these issues, we propose OE-BevSeg, an end-to-end multimodal framework that enhances BEV segmentation performance through global environment-aware perception and local target object enhancement. OE-BevSeg employs an environment-aware BEV compressor. Based on prior knowledge about the main composition of the BEV surrounding environment varying with the increase of distance intervals, long-sequence global modeling is utilized to improve the model’s understanding and perception of the environment. From the perspective of enriching target object information in segmentation results, we introduce the center-informed object enhancement module, using centerness information to supervise and guide the segmentation head, thereby enhancing segmentation performance from a local enhancement perspective. Additionally, we designed a multimodal fusion branch that integrates multi-view RGB image features with radar/LiDAR features, achieving significant performance improvements. Extensive experiments show that, whether in camera-only or multimodal fusion BEV segmentation tasks, our approach achieves state-of-the-art results by a large margin on the nuScenes dataset for vehicle segmentation, demonstrating superior applicability in the field of autonomous driving. Our code will be released at https://github.com/SunJ1025/OE-BevSeghttps://github.com/SunJ1025/OE-BevSeg.

Training deep reinforcement learning (RL) agents necessitates overcoming the highly unstable nonconvex stochastic optimization inherent in the trial-and-error mechanism. To tackle this challenge, we propose a physics-inspired optimization algorithm called relativistic adaptive gradient descent (RAD), which enhances long-term training stability. By conceptualizing neural network (NN) training as the evolution of a conformal Hamiltonian system, we present a universal framework for transferring long-term stability from conformal symplectic integrators to iterative NN updating rules, where the choice of kinetic energy governs the dynamical properties of resulting optimization algorithms. By utilizing relativistic kinetic energy, RAD incorporates principles from special relativity and limits parameter updates below a finite speed, effectively mitigating abnormal gradient influences. In addition, RAD models NN optimization as the evolution of a multiparticle system where each trainable parameter acts as an independent particle with an individual adaptive learning rate. We prove RAD's sublinear convergence under general nonconvex settings, where smaller gradient variance and larger batch sizes contribute to tighter convergence. Notably, RAD degrades to the well-known adaptive moment estimation (ADAM) algorithm when its speed coefficient is chosen as one and symplectic factor as a small positive value. Experimental results show RAD outperforming nine baseline optimizers with five RL algorithms across twelve environments, including standard benchmarks and challenging scenarios. Notably, RAD achieves up to a 155.1% performance improvement over ADAM in Atari games, showcasing its efficacy in stabilizing and accelerating RL training.

Multi-lane formation control can significantly enhance the efficiency of traffic systems in multi-lane scenarios. However, existing multi-lane formation control methods are mostly developed for fully Connected and Automated Vehicle (CAV) environments and lack a mechanism for multi-lane formation control in mixed traffic. This paper proposes a CAV formation grouping method and an interaction mechanism of CAVs and Human-driven Vehicles (HDVs), which is suitable for different traffic volumes and penetration rates. Depending on the positional relationship between the CAV formation and HDVs, it flexibly chooses between conventional formation control methods or improved mixed traffic formation control methods. By conducting simulations at various input flow volumes and penetration rates, the applicability of this method to different conditions is verified. The paper also explores the extent to which multi-lane formation control methods can improve traffic efficiency and their relationship with traffic volume and penetration rate.

In this article, we present the first experimental results of data-driven predictive control for connected and autonomous vehicles (CAVs) in dissipating traffic waves. In particular, we consider a recent strategy of Data-EnablEd Predictive Leading Cruise Control (DeeP-LCC), which bypasses the need of identifying the driving behaviors of surrounding vehicles and directly relies on measurable traffic data to achieve safe and optimal CAV control in mixed traffic. We present the implementation details ofDeeP-LCC, including data collection, equilibrium estimation, and control execution. Based on a miniature experiment platform, we reproduce the phenomenon of stop-and-go waves in two typical traffic scenarios: 1) open straight-road scenario under external disturbances and 2) closed ring-road scenario with no bottlenecks. Our experiments clearly demonstrate thatDeeP-LCCenables one or a few CAVs to dissipate the traffic waves in both traffic scenarios. These experimental findings validate the great potential ofDeeP-LCCin smoothing practical traffic flow in the presence of noisy data, uncertain low-level vehicle dynamics, and communication and computation delays. The code and videos of our experimental results are available athttps://github.com/soc-ucsd/DeeP-LCC.

Various sensors are adopted by autonomous driving systems to perceive objects and surroundings. Thus, the multi-sensor data fusion techniques become essential to combine different sensors’ advantages for better perception performance. However, the current multi-sensor data fusion techniques suffer from the high cost of computation resources, low expansibility for more diverse sensors, and insufficient systematic consideration for modeling. This paper first constructs a detachable and expansible multi-sensor data fusion model based on three main modules: front fusion, global fusion, and synthesizer, where the methods for flexible association gating and virtual targets have been designed. The model can be disassembled and configured for different trim levels of vehicles and is easily expansible for adding more heterogeneous sensors. Next, the presented multi-sensor data fusion model is compared with the cheap Joint Probabilistic Data Association (C-JPDA) method. The comparison shows the superior accuracy of the designed model on false association and effective narrowing of the variance of object detection. Finally, the presented multi-sensor data fusion model is integrated into an embedded system and experimented on urban roads and highways with the engaged Level 3 autonomous driving function. The experiment results indicate that the proposed model has excellent sensor data fusion performance and provides accurate and timely object information in the Level 3 autonomous driving system.

Connected and automated vehicles have shown great potential in improving traffic mobility and reducing emissions, especially at unsignalized intersections. Previous research has shown that vehicle passing order is the key influencing factor in improving intersection traffic mobility. In this paper, we propose a graph-based cooperation method to formalize the conflict-free scheduling problem at an unsignalized intersection. Based on graphical analysis, a vehicle’s trajectory conflict relationship is modeled as a conflict directed graph and a coexisting undirected graph. Then, two graph-based methods are proposed to find the vehicle passing order. The first is an improved depth-first spanning tree algorithm, which aims to find the local optimal passing order vehicle by vehicle. The other novel method is a minimum clique cover algorithm, which identifies the global optimal solution. Finally, a distributed control framework and communication topology are presented to realize the conflict-free cooperation of vehicles. Extensive numerical simulations are conducted for various numbers of vehicles and traffic volumes, and the simulation results prove the effectiveness of the proposed algorithms.

Comprehensive and accurate perception of the real 3D world is the basis of autonomous driving. However, many perceptual methods focus on a single task or object type, and the accuracy of existing multi-task or multi-object methods is difficult to balance against their real-time performance. This paper presents a unified framework for concurrent dynamic multi-object joint perception, which introduces a real-time monocular joint perception network termed MJPNet. In MJPNet relative weightings are automatically learned by a series of developed network branches. By training an end-to-end deep convolutional neural network on a shared feature encoder and many proposed decoding sub-branches, the information of the 2D category and 3D position/pose/size of an object are reconstructed both simultaneously and accurately. Moreover, the effective information among subtasks is transferred by multi-stream learning, guaranteeing the accuracy of each task. Compared to various state-of-the-arts, comprehensive evaluations on the benchmark of challenging image sequences demonstrate the superior performance of our 2D detection and 3D reconstruction of depth, lateral distance, orientation, and heading angle. Moreover, on the KITTI test set, the real-time runtime (up to 15 fps) of MJPNet significantly outran the public state-of-the-art visual detection methods. Accompanying video:https://youtu.be/Z-goToOlI94.

Connected and autonomous vehicles (CAVs) have great potential to improve road transportation systems. Most existing strategies for CAVs’ longitudinal control focus on downstream traffic conditions, but neglect the impact of CAVs’ behaviors on upstream traffic flow. In this paper, we introduce a notion of Leading Cruise Control (LCC), in which the CAV maintains car-following operations adapting to the states of its preceding vehicles, and also aims to lead the motion of its following vehicles. Specifically, by controlling the CAV, LCC aims to attenuate downstream traffic perturbations and smooth upstream traffic flow actively. We first present the dynamical modeling of LCC, with a focus on three fundamental scenarios: car-following, free-driving, and Connected Cruise Control. Then, the analysis of controllability, observability, and head-to-tail string stability reveals the feasibility and potential of LCC in improving mixed traffic flow performance. Extensive numerical studies validate that the capability of CAVs in dissipating traffic perturbations is further strengthened when incorporating the information of the vehicles behind into the CAVs’ control.

Intelligent vehicles should be capable to understand the intention of other traffic participants when driving on urban roads. Yet, current approaches mostly emphasize the importance of the crossing/not-crossing (C/NC) problem and neglect the intention estimation task. To this end, we propose a pedestrian intention estimation method based on the extended theory of planned behavior (TPB). In contrast to previous qualitative modeling based on surveys and questionnaires, neural networks and hand-crafted rules are designed to quantitatively model the components of the extended TPB in the proposed architecture. Besides, the interaction between the components is simulated by a mixed classification strategy. Our pedestrian intention estimation model achieves 82% accuracy and outperforms the baseline method by 3% on the pedestrian intention estimation (PIE) dataset.

Connected and automated vehicles (CAVs) have a great potential to improve traffic efficiency in mixed traffic systems, which has been demonstrated by multiple numerical simulations and field experiments. However, some fundamental properties of mixed traffic flow, including controllability and stabilizability, have not been well understood. This paper analyzes the controllability of mixed traffic systems and designs a system-level optimal control strategy. Using the Popov-Belevitch-Hautus (PBH) criterion, we prove for the first time that a ring-road mixed traffic system with one CAV and multiple heterogeneous human-driven vehicles is not completely controllable, but is stabilizable under a very mild condition. Then, we formulate the design of a system-level control strategy for the CAV as a structured optimal control problem, where the CAV’s communication ability is explicitly considered. Finally, we derive an upper bound for reachable traffic velocity via controlling the CAV. Extensive numerical experiments verify the effectiveness of our analytical results and the proposed control strategy. Our results validate the possibility of utilizing CAVs as mobile actuators to smooth traffic flow actively.

Lane keeping is a key task in driving, and it plays an important role in staying safe while driving. However, conventional lane-keeping assistance systems (LKASs) have a limited level of automation, while fully autonomous lane-keeping systems have shortcomings regarding reliability. To address these issues, an advanced LKAS with two switchable assistance modes, namely, the lane departure prevention mode and the lane-keeping co-pilot mode, is proposed in this paper. First, the system structure is constructed. Then, the functions and control strategies for the two assistance modes are defined. Next, the controller algorithms are designed using the learning-based model predictive control (LBMPC) method to compensate for modeling errors. Within the framework of LBMPC, an oracle is built to learn the unmodeled dynamics using an extended Kalman filter. Moreover, optimization problems are formulated to achieve the control objectives regarding driving safety and driver acceptance. Finally, driver-in-the-loop experiments carried out on a driving simulator prove that both assistance modes are effective. Furthermore, the lane-keeping co-pilot mode can help reduce the driving burden in the sense of avoiding frequent steering correction.

This paper presents a general design of an autonomous driving platform. Firstly, we design a physic architecture and three-bus power system to guarantee safety and power redundancy. The logical architecture of the platform is designed in five sub-functions. In most of the cases, planning and control function is the critical and universal function, where we use curvature and environmental perception information to calculate vehicle speed. We also apply feedforward-feedback controller with preview point method to calculate steering angle. In the end, we conduct field test and road test to verify the functionality of the platform. Different navigation methods are also compared in the tests.

Pedestrian orientation is essential for trajectory prediction, intention and behavior analysis. Existing methods using low-level features result in imperfect performance. Aiming at continuous pedestrian orientation estimation from a single frame, we propose a novel approach based on high-level semantic features extracted from human keypoints locations. The proposed method first predicts human keypoints locations by pose estimation algorithm, based on human pose, body motion limitations and body parts occlusion. High-level semantic features are then extracted from keypoints locations. Finally, this method applies a soft classifier and interpolation to produce continuous results. Experiments conducted on public datasets show that the proposed method outperforms the state-of-the-art by a large margin, demonstrating the critical role of high-level semantic feature representation for the task of continuous pedestrian orientation estimation.

This paper is concerned with coordinated formation assignment and control of multiple connected vehicles in highway scenario. A dynamical interlaced layered formation generation method is introduced to provide safe distance among vehicles and efficiency for coordinated lane changing and formation switching simultaneously in real time. To assign vehicles to the generated formation, the optimal problem is modeled by introducing the function of numbers of changed lanes for all the vehicles with respect to the kinematic constraints, and on-board local controllers for the vehicles are developed. Simulation result indicates that, comparing to existing methods for lane assignment in multiple traffic scenarios, the method provided by this paper could increase the traffic efficiency by utilizing maximum road capacity while decreasing travel time for all the vehicles.

Connected and automated vehicles (CAVs) have a great potential to actively influence traffic systems. This has been demonstrated by large-scale numerical simulations and small-scale real experiments, whereas a comprehensive theoretical analysis is still lacking. In this paper, we focus on mixed traffic systems with one single CAV and heterogeneous human-driven vehicles, and present rigorous controllability analysis and optimal controller synthesis. Using the PBH controllability criterion, we reveal controllability properties of a linearized mixed traffic system in a ring road. It is proved that the mixed traffic flow can be stabilized by one single CAV under a very mild condition. We formulate the problem of designing CAV control strategies under a pre-specified communication topology as structured optimal controller synthesis. This formulation considers a system-level performance index that allows the CAV to actively dampen undesired perturbations in traffic flow. Numerical experiments verify the effectiveness of our results.

Only a few existing works exploit multiple modalities of data for road detection task in the context of autonomous driving. In this work, a deep learning based approach is developed to fuse LIDAR point cloud and camera image features over a bird's eye view representation. A two-stream fully-convolutional network is designed as encoder to extract general features of two types of data. Instead of limiting the fusion processing at a single stage or to a predefined extent, we propose a multi-stage residual fusion strategy to merge the feature maps in a residual learning fashion, and integrate the information at different network depth. Experiments conduct on KITTI road benchmark show that our proposed method has a significant improvement over single modality methods and other fusion approaches. And it is also among the top-performing algorithms.