Vehicle-to-everything (V2X) technologies offer a promising paradigm to mitigate the limitations of constrained observability in single-vehicle systems. Prior work primarily focuses on single-frame cooperative perception, which fuses agents' information across different spatial locations but ignores temporal cues and temporal tasks (e.g., temporal perception and prediction). In this paper, we focus on the spatio-temporal fusion in V2X scenarios and design one-step and multi-step communication strategies (when to transmit) as well as examine their integration with three fusion strategies - early, late, and intermediate (what to transmit), providing comprehensive benchmarks with 11 fusion models (how to fuse). Furthermore, we propose V2XPnP, a novel intermediate fusion framework within one-step communication for end-to-end perception and prediction. Our framework employs a unified Transformer-based architecture to effectively model complex spatio-temporal relationships across multiple agents, frames, and high-definition maps. Moreover, we introduce the V2XPnP Sequential Dataset that supports all V2X collaboration modes and addresses the limitations of existing real-world datasets, which are restricted to single-frame or single-mode cooperation. Extensive experiments demonstrate that our framework outperforms state-of-the-art methods in both perception and prediction tasks.

This paper introduces and tests a framework integrating traffic regulation compliance into automated driving systems (ADS). The framework enables ADS to follow traffic laws and make informed decisions based on the driving environment. Using RGB camera inputs and a vision-language model (VLM), the system generates descriptive text to support a regulation-aware decision-making process, ensuring legal and safe driving practices. This information is combined with a machine-readable ADS regulation database to guide future driving plans within legal constraints. Key features include: 1) a regulation database supporting ADS decision-making, 2) an automated process using sensor input for regulation-aware path planning, and 3) validation in both simulated and real-world environments. Particularly, the real-world vehicle tests not only assess the framework's performance but also evaluate the potential and challenges of VLMs to solve complex driving problems by integrating detection, reasoning, and planning. This work enhances the legality, safety, and public trust in ADS, representing a significant step forward in the field.

Existing Vehicle-to-Everything (V2X) cooperative perception methods rely on accurate multi-agent 3D annotations. Nevertheless, it is time-consuming and expensive to collect and annotate real-world data, especially for V2X systems. In this paper, we present a self-supervised learning framwork for V2X cooperative perception, which utilizes the vast amount of unlabeled 3D V2X data to enhance the perception performance. Specifically, multi-agent sensing information is aggregated to form a holistic view and a novel proxy task is formulated to reconstruct the LiDAR point clouds across multiple connected agents to better reason multi-agent spatial correlations. Besides, we develop a V2X bird-eye-view (BEV) guided masking strategy which effectively allows the model to pay attention to 3D features across heterogeneous V2X agents (i.e., vehicles and infrastructure) in the BEV space. Noticeably, such a masking strategy effectively pretrains the 3D encoder with a multi-agent LiDAR point cloud reconstruction objective and is compatible with mainstream cooperative perception backbones. Our approach, validated through extensive experiments on representative datasets (i.e., V2X-Real, V2V4Real, and OPV2V) and multiple state-of-the-art cooperative perception methods (i.e., AttFuse, F-Cooper, and V2X-ViT), leads to a performance boost across all V2X settings. Notably, CooPre achieves a 4% mAP improvement on V2X-Real dataset and surpasses baseline performance using only 50% of the training data, highlighting its data efficiency. Additionally, we demonstrate the framework’s powerful performance in cross-domain transferability and robustness under challenging scenarios. The code will be made publicly available at https://github.com/ucla-mobility/CooPre.

Accurate positioning and navigation capabilities are essential for Internet of Things (IoT) devices. Event cameras, inspired by biological vision sensors, exhibit robust performance in high-dynamic and low-texture environments and are particularly suitable for IoT applications. However, it faces challenges with accuracy and scale in conventional slow-motion scenarios. Conversely, light detection and ranging (LiDAR) offers high precision in normal motion conditions but degrades significantly under high-dynamic motion. To integrate the advantages of both sensors, this article introduces the EVLINS algorithm, a multisource elastic fusion method based on an extended Kalman filter (EKF). This algorithm combines event-visual-inertial odometry (EVIO), LiDAR-inertial odometry (LIO), and an inertial measurement unit (IMU), utilizing a loosely coupled trajectory layer post-processing technique. This algorithm leverages the robustness of event cameras in highly dynamic environments and the precision of LiDAR in conventional settings, utilizing normalized uncertainty and nonholonomic constraint (NHC) strategies to address LIO’s degradation and EVIO’s accuracy issues. Thorough testing in various indoor and outdoor scenarios with real-world data demonstrates that EVLINS exhibits significantly improved accuracy and robustness compared to both LIO and EVIO algorithms. In large-scale, high-dynamic outdoor environments, EVLINS achieves a 3-D position accuracy of 0.68% over 1333.58 m, improving by 33.21% over LIO and 96.10% over EVIO, which diverged mid-way. In extreme indoor dynamic scenarios, EVLINS reduces maximum position error by 41.55% compared to LIO and improves overall position accuracy by 43.48%, and 22.96% compared to EVIO.

Multisensor integrated navigation, combining the global navigation satellite system (GNSS), inertial navigation system (INS), and light detection and ranging (LiDAR), is at the forefront of high-precision positioning technology. However, existing integration methods rely on raw point cloud data, which cannot be obtained in some projects due to geospatial data privacy concerns. For this problem, this work introduces the trajectory-layer-enhanced GNSS/INS/LiDAR integrated navigation system (TL-GILNS), which operates without raw point cloud data. This novel approach faces two primary challenges: 1) the difficulty in resolving the divergence of LiDAR positioning results due to cumulative errors and 2) the challenge of accurately assigning weights to LiDAR data in the integrated system. To overcome these obstacles, we propose a trajectory-layer LiDAR positioning enhancement method to reduce cumulative errors and a trajectory-layer LiDAR positioning accuracy quantification method to determine the weight of LiDAR. Finally, the performance of TL-GILNS is verified through experiments conducted in semi-open and large-scale complex scenes. In these scenes, TL-GILNS achieves horizontal accuracy better than 0.3 m, vertical accuracy better than 0.7 m, and yaw angle accuracy better than 1°. These results demonstrate the potential of TL-GILNS as a leading approach for high-precision navigation and positioning in complex environments.

The road adhesion coefficient is crucial information for connected autonomous control. However, it is challenging to obtain based solely on current vehicle state sensors. This paper proposed a novel road adhesion coefficient estimation method based on the digital twin framework and combing of vehicle state sensor and vision sensor. Utilizing the vehicle state sensor, a nonlinear observer dynamics model of tire force is established. Then, an improved Innovation Adaptive Estimation Unscented Kalman Filter (IAE-UKF) algorithm is designed for road adhesion coefficient estimation. Simultaneously a deep convolutional neural network is adopted to classify the road surfaces type based on the images from visual sensor. Multi-sensor data fusion is performed by mapping visual identification labels to reference values via a lookup table, followed by spatiotemporal synchronization with the dynamics based approach. A distributed cooperative estimation mechanism is developed to address potential failures in either estimator. Simulation and experimental results show that the proposed strategy effectively integrates a variety of sensor information based on the digital twin framework. Compared to traditional dynamics based methods, the introduction of tire dynamic characteristics discrimination significantly reduces its reliance on high tire excitation. Furthermore, the proposed estimation method can maintain the accuraccy and reliability in the situation of vision sensor misidentification.

Lane-change is a common maneuver. However, completely egocentric lane-changing behavior may increase potential driving risks and cause oscillations in the movement of surrounding vehicles, referred to as disharmony. To improve the overall safety and efficiency of both ego vehicle and their surroundings, this paper proposes a convergent harmonious reinforcement learning (CHRL) approach to generate harmonious lane-changing strategies. It introduces a game-based model to measure the overall harmony cost. On this basis, a prosocial critic network is established to guide the policy toward harmony by decreasing the harmony cost. Meanwhile, CHRL identifies and penalizes discordant behaviors that may lead to high risk, accelerating the RL agent’s learning of harmonious driving strategies through expert demonstrations of the game model. Simulation and real-data tests validate that CHRL, compared to other lane-change methods and human drivers, improves the overall harmony of lane changes for autonomous vehicles.

LiDAR-Inertial Odometry (LIO) demonstrates outstanding accuracy and stability in general low-speed and smooth motion scenarios. However, in high-speed and intense motion scenarios, such as sharp turns, two primary challenges arise: firstly, due to the limitations of IMU frequency, the error in estimating significantly non-linear motion states escalates; secondly, drastic changes in the Field of View (FOV) may diminish the spatial overlap between LiDAR frame and pointcloud map (or between frames), leading to insufficient data association and constraint degradation.To address these issues, we propose a novel Adaptive Sliding window LIO framework (AS-LIO) guided by the Spatial Overlap Degree (SOD). Initially, we assess the SOD between the LiDAR frames and the registered map, directly evaluating the adverse impact of current FOV variation on pointcloud alignment. Subsequently, we design an adaptive sliding window to manage the continuous LiDAR stream and control state updates, dynamically adjusting the update step according to the SOD. This strategy enables our odometry to adaptively adopt higher update frequency to precisely characterize trajectory during aggressive FOV variation, thus effectively reducing the non-linear error in positioning. Meanwhile, the historical constraints within the sliding window reinforce the frame-to-map data association, ensuring the robustness of state estimation. Experiments show that our AS-LIO framework can quickly perceive and respond to challenging FOV change, outperforming other state-of-the-art LIO frameworks in terms of accuracy and robustness.

Cooperative driving automation attracts great attention for its potential to improve traffic safety. Knowing each vehicle’s accurate position serves as the cornerstone for the information fusion necessary in cooperative driving tasks. However, inherent errors within a vehicle’s self-localization system often necessitate correction to facilitate cooperative perception and downstream tasks. Leveraging intermediate features shared among other Connected Automated Vehicles (CAVs), we propose an end-to-end learning localization framework aimed at estimating the relative pose error between the ego vehicle and the CAV. We investigate factors that may influence learning performance and validate the algorithm’s performance using a simulation dataset. The proposed method is compared with the traditional point cloud matching-based relative localization method. Remarkably, our framework effectively corrects relative pose errors even when the vehicle exhibits significant initial localization inaccuracies, and it can be integrated into the cooperative perception system.

Cooperative perception algorithms based on fusing sensing data across multiple connected automated vehicles (CAVs) have shown promising performance to enhance the existing individual perception algorithm in terms of object detection tracking. However, existing cooperative perception algorithms are only developed offline given the constraints from data sharing and computational resources and none of them have been verified in real-time conditions. In this work, we propose a real-time cooperative perception framework called CooperFuse, which achieves cooperative perception in a late fusion scheme. Based on object detection and tracking results from individual vehicle, the late fusion cooperative perception algorithm considers object detection confidence score, kinematics, and dynamics consistency as well as scale consistency of detected objects. The algorithm computes the kinematic and dynamic consistency of the objects by solving for the energy consumption of inter-frame trajectories, and determines scale consistency by calculating inter-frame scale changes, enabling feature-based bounding box fusion. The experimental results demonstrate the real-time performance of the proposed algorithm and reveal its effective improvements in feature fusion and object detection accuracy when dealing with heterogeneous detection models across different cooperative intelligent agents.

The safety of different categories of road users comprising motorized vehicles and vulnerable road users (VRUs) such as pedestrians and cyclists is one of the priorities of automated driving and smart infrastructure services. Three-dimensional (3-D) LiDAR-based object detection has been a promising approach to perceiving road users. Despite accurate 3-D geometry information, the point cloud from LiDAR is usually nonuniform, and learning the effective point cloud abstract representations for diverse road users remains challenging for 3-D object detection, particularly for small objects such as VRUs. For inclusive object detection (IDetect), we propose a general foundation convolution component, called geometry-aware convolution (GA Conv) toward a foundation feature extraction model, to serve as basic convolution operations of the neutral network for inclusive 3-D object detection. Further, the GA Conv operations are then utilized as the elementary feature extraction layers to build a novel elegant and pyramid network for IDetect. It learns the effective geometric-related features from the unstructured point cloud data by implicitly learning the distribution property and geometry-related features from different categories of road users in particular for VRUs. The proposed IDetect is comprehensively evaluated on the large-scale benchmark Waymo open datasets with all categories of road users. The qualitative and quantitative experiment results demonstrate that IDetect can effectively consider the nonuniform distributed point clouds and learn the geometric features to assist the different categories of road user detection. In addition, the GA Conv has been integrated with other state-of-the-art neural networks and a performance boost for VRU detection has been demonstrated, showing the foundation functionality of the GA Conv and making it a general component in the future inclusive 3-D object detection foundation model.

Modern perception systems of autonomous vehicles are known to be sensitive to occlusions and lack the capability of long perceiving range. It has been one of the key bottlenecks that prevents Level 5 autonomy. Recent research has demonstrated that the Vehicle-to-Vehicle (V2V) cooperative perception system has great potential to revolutionize the autonomous driving industry. However, the lack of a real-world dataset hinders the progress of this field. To facilitate the development of cooperative perception, we present V2V4Real, the first large-scale real-world multi-modal dataset for V2V perception. The data is collected by two vehicles equipped with multi-modal sensors driving together through diverse scenarios. Our V2V4Real dataset covers a driving area of 410 km, comprising 20K LiDAR frames, 40K RGB frames, 240K annotated 3D bounding boxes for 5 classes, and HDMaps that cover all the driving routes. V2V4Real introduces three perception tasks, including cooperative 3D object detection, cooperative 3D object tracking, and Sim2Real domain adaptation for cooperative perception. We provide comprehensive benchmarks of recent cooperative perception algorithms on three tasks. The V2V4Real dataset can be found at research.seas.ucla.edu/mobility-lab/v2v4real/.

Recent advancements in Vehicle-to-Everything communication technology have enabled autonomous vehicles to share sensory information to obtain better perception performance. With the rapid growth of autonomous vehicles and intelligent infrastructure, the V2X perception systems will soon be deployed at scale, which raises a safety-critical question: how can we evaluate and improve its performance under challenging traffic scenarios before the real-world deployment? Collecting diverse large-scale real-world test scenes seems to be the most straightforward solution, but it is expensive and time-consuming, and the collections can only cover limited scenarios. To this end, we propose the first open adversarial scene generator V2XP-ASG that can produce realistic, challenging scenes for modern LiDAR-based multi-agent perception systems. V2XP-ASG learns to construct an adversarial collaboration graph and simultaneously perturb multiple agents' poses in an adversarial and plausible manner. The experiments demonstrate that V2XP-ASG can effectively identify challenging scenes for a large range of V2X perception systems. Meanwhile, by training on the limited number of generated challenging scenes, the accuracy of V2X perception systems can be further improved by 12.3% on challenging and 4% on normal scenes. Our code will be released at https://github.com/XHwind/V2XP-ASG.

Existing multi-agent perception systems assume that every agent utilizes the same model with identical parameters and architecture. The performance can be degraded with different perception models due to the mismatch in their confidence scores. In this work, we propose a model-agnostic multi-agent perception framework to reduce the negative effect caused by the model discrepancies without sharing the model information. Specifically, we propose a confidence calibrator that can eliminate the prediction confidence score bias. Each agent performs such calibration independently on a standard public database to protect intellectual property. We also propose a corresponding bounding box aggregation algorithm that considers the confidence scores and the spatial agreement of neighboring boxes. Our experiments shed light on the necessity of model calibration across different agents, and the results show that the proposed framework improves the baseline 3D object detection performance of heterogeneous agents. The code can be found at this url.

Vehicle control is one of the most critical challenges in autonomous vehicles (AVs) and connected and automated vehicles (CAVs), and it is paramount in vehicle safety, passenger comfort, transportation efficiency, and energy saving. This survey attempts to provide a comprehensive and thorough overview of the current state of vehicle control technology, focusing on the evolution from vehicle state estimation and trajectory tracking control in AVs at the microscopic level to collaborative control in CAVs at the macroscopic level. First, this review starts with vehicle key state estimation, specifically vehicle sideslip angle, which is the most pivotal state for vehicle trajectory control, to discuss representative approaches. Then, we present symbolic vehicle trajectory tracking control approaches for AVs. On top of that, we further review the collaborative control frameworks for CAVs and corresponding applications. Finally, this survey concludes with a discussion of future research directions and the challenges. This survey aims to provide a contextualized and in-depth look at the state of the art in vehicle control for AVs and CAVs, identifying critical areas of focus and pointing out the potential areas for further exploration.

Employing Vehicle-to-Vehicle communication to enhance perception performance in self-driving technology has attracted considerable attention recently; however, the absence of a suitable open dataset for benchmarking algorithms has made it difficult to develop and assess cooperative perception technologies. To this end, we present the first large-scale open simulated dataset for Vehicle-to-Vehicle perception. It contains over 70 interesting scenes, 11,464 frames, and 232,913 annotated 3D vehicle bounding boxes, collected from 8 towns in CARLA and a digital town of Culver City, Los Angeles. We then construct a comprehensive benchmark with a total of 16 implemented models to evaluate several information fusion strategies (i.e. early, late, and intermediate fusion) with state-of-the-art LiDAR detection algorithms. Moreover, we propose a new Attentive Intermediate Fusion pipeline to aggregate information from multiple connected vehicles. Our experiments show that the proposed pipeline can be easily integrated with existing 3D LiDAR detectors and achieve outstanding performance even with large compression rates. To encourage more researchers to investigate Vehicle-to-Vehicle perception, we will release the dataset, benchmark methods, and all related codes in https://mobility-lab.seas.ucla.edu/opv2v/.

There is an increasing popularity in exploiting modern vehicles as mobile sensors to obtain important road information such as potholes, black ice and road profile. Availability of such information has been identified as a key enabler for next-generation vehicles with enhanced safety, efficiency, and comfort. However, existing road information discovery approaches have been predominately performed in a single-vehicle setting, which is inevitably susceptible to vehicle model uncertainty and measurement errors. To overcome these limitations, this paper presents a novel cloud-assisted collaborative estimation framework that can utilize multiple heterogeneous vehicles to iteratively enhance estimation performance. Specifically, each vehicle combines its onboard measurements with a cloud-based Gaussian process (GP), crowdsourced from prior participating vehicles as “pseudo-measurements”, into a local estimator to refine the estimation. The resultant local onboard estimation is then sent back to the cloud to update the GP, where we utilize a noisy input GP (NIGP) method to explicitly handle uncertain GPS measurements. We employ the proposed framework to the application of collaborative road profile estimation. Promising results on extensive simulations and hardware-in-the-loop experiments show that the proposed collaborative estimation can significantly enhance estimation and iteratively improve the performance from vehicle to vehicle, despite vehicle heterogeneity, model uncertainty, and measurement noises.

Taking advantage of available measurement in Internet of Things (IoT) for intelligent transportation systems, a sideslip angle estimation method for autonomous vehicles is presented and experimentally verified by fusing global navigation satellite system (GNSS) and inertial measurement unit (IMU), and by constructing an observability index (OI). The correlation between the vehicle sideslip error and the inertial navigation system (INS) heading error is presented first. Then, the observability for the heading error in a velocity-based Kalman filter is discussed and a novel index is defined to check the observability of the heading error. The course from a single antenna GNSS in an autonomous vehicle is augmented to estimate the heading error when the observability of the heading error is low. To reject the course measurement for scenarios that include sideslip movement, a binary hypothesis test approach is applied to indicate whether the vehicle is sidesliping. In addition, based on the OI and the sideslip indicator, a hybrid feedback strategy is designed for the heading error correction. To improve the convergence rate of the heading error in the velocity-based Kalman filter, a tuning strategy is presented. The stochastical observability of the designed Kalman observer is investigated for known and stochastic initial conditions. Finally, the proposed sideslip angle estimator is experimentally validated through a vehicle test platform in critical driving scenarios. The results confirm that the proposed OI can effectively identify when the heading error is observable, and also corroborate the effectiveness of the hybrid feedback strategy and adaptation method in the Kalman observer.

Considering vehicle sideslip angle estimation difficulty under severe driving conditions with dynamic model based method due to vehicle nonlinear characteristic and parameter uncertainty, a novel kinematic model based method is proposed with the fusion of intelligent vehicle sensors. The state space models of the vehicle yaw angle and the roll angle are constructed based on the IMU and the lateral arrangement of the dual-GPS. In order to reduce the weight of the previously estimated value, an adaptive fading Kalman filtering algorithm is adopted to improve the filtering effect on the yaw and roll angle. A nonlinear adaptive observer is constructed to estimate the vehicle sideslip angle with the integration of the road line information from the camera, velocity from the GPS and acceleration/ angular velocity from the IMU. Furthermore, compared with the IMU, the information obtained from the GPS and the camera can't be utilized directly as large measurement delay. Thus, an observer-predictor is developed with multi-sensor fusion to handle the measurement delay problem. Finally, the proposed algorithm is validated through co-simulation under different maneuvers.

In this paper, an estimation method for attitude and lateral velocity for automated vehicle has been proposed using a six degree of freedom inertial measurement unit(IMU) aided by vehicle dynamics. This estimation method makes full use of the advantage of the IMU and vehicle dynamics and could run autonomously without aid from other outside information such as GNSS or camera. Based on Kalman filter theory, three observers have been developed: a kinematic model based attitude observer for pitch angle and roll angle using IMU, a kinematic model based lateral velocity observer for lateral velocity using IMU and a dynamic model based observer using vehicle dynamics. In small excitation condition, the estimated lateral velocity from dynamic model based observer is more reliable and it is forwarded to the two kinematic model based observers to prevent the accumulated estimation error. In larger excitation condition, the two kinematic model based observers run in open-loop mode. Slalom maneuver and double lane change(DLC) maneuvers have been conducted to validate the estimation method. The experiment results have proved the effectiveness of this estimation method.