Motion prediction is essential for autonomous vehicles to navigate complex environments and anticipate the behavior of other traffic participants. As new driving scenarios emerge, models must be continually updated without retraining from scratch. We propose DECODE, a continual learning framework that starts from a pre-trained generalized model and incrementally expands specialized models for distinct domains. Unlike existing approaches that pursue a single unified model, DECODE explicitly balances specialization and generalization through dynamic model selection. It employs a hypernetwork for parameter generation, which reduces storage costs, and utilizes a normalizing flow for real-time domain inference via likelihood estimation. Outputs from specialized and generalized models are fused using Bayesian uncertainty estimation. This integration ensures optimal performance in familiar conditions while maintaining robustness in novel scenarios. Extensive experiments show DECODE achieves a low forgetting rate of 0.044 and an average minADE of 0.584 m, outperforming prior methods and generalizing well across diverse driving domains. Furthermore, we demonstrate that DECODE can be extended beyond motion prediction to general continual learning tasks such as image classification, showcasing its broad applicability.

Trajectory prediction is crucial for autonomous vehicles to make safe and informed decisions. However, the lack of transparency in current trajectory prediction models introduces significant security risks, because their output contains almost no explanatory details. To address these challenges and bridge this research gap, we propose a novel approach IGTPT that not only predicts vehicle trajectories but also generates textual descriptions of the vehicle’s intent. The vehicle’s intent can be broadcast to nearby vehicles to enhance decision-making transparency and increase user trust. Our approach directly confronts the opaque nature of existing systems by providing clear, understandable explanations for autonomous decisions, thereby enhancing both the security and reliability of these systems. We enhanced the BDD-X dataset to create the BDD-XE, a specialized dataset for trajectory prediction and behavior description, which our framework uses to achieve superior results in both prediction accuracy and behavioral interpretation compared to established baseline methods. We demonstrate the practical applicability of our framework through a complete system that processes past raw driving videos and trajectory observations to deliver real-time predictions along with insightful behavioral narrations and reasoning.

Infrastructure-based cooperative perception has been researched for several years, but few automotive warning or control applications using this information have been published. Infrastructure sensing, such as with cameras or lidars, and a communication system, allows connected vehicles to receive information about all observed objects. An SAE standard, “V2X Sensor-Sharing for Cooperative and Automated Driving” (J3224), released in 2022, introduces the Sensor Data Sharing Message (SDSM) as the standard communication message for cooperative perception. This paper investigates the use of the SDSM for a vehicle application to provide warnings of potential collisions with vulnerable road users who will cross the street at the intersection. The application was tested in CARLA simulation under various roadside detection errors and communication conditions to assess the impact on the on-board application and estimate the minimum detection and communication requirements for effective use. In addition, the system was implemented and evaluated at the Mcity test facility. The results demonstrate that the proposed warning system can accurately and promptly warn the driver, given specific communication conditions, and show that the SDSM is viable for real-time on-board usage.

Roadside perception has become a critical component for connected and automated vehicles (CAVs), enhancing safety and offering a comprehensive view of the traffic environment that onboard detection systems alone cannot provide. By supplementing the limitations of onboard sensors, roadside perception systems improve the accuracy and reliability of detecting and localizing vehicles and pedestrians in challenging locations. Currently, a variety of cameras, including fisheye and regular cameras, are deployed along roadsides for surveillance purposes. These sensors have significant potential to improve vehicle and pedestrian detection. This paper extends our previous work on single image sensor vehicle detection by developing a comprehensive multiple sensor fusion framework. We take advantage of the complementary strengths of multiple fisheye and regular cameras to enhance the accuracy and robustness of the perception system. The proposed system has been extensively tested in Mcity, a controlled urban testing environment, through numerous field tests. The results demonstrate the effectiveness of our approach, showcasing promising improvements in vehicle and pedestrian detection and tracking accuracy.

Accurate trajectory prediction is vital for various applications, including autonomous vehicles. However, the complexity and limited transparency of many prediction algorithms often result in black-box models, making it challenging to understand their limitations and anticipate potential failures. This further raises potential risks for systems based on these prediction models. This study introduces the performance monitor for black-box trajectory prediction model (PMBP) to address this challenge. The PMBP estimates the performance of black-box trajectory prediction models online, enabling informed decision-making. The study explores various methods’ applicability to the PMBP, including anomaly detection, machine learning, deep learning, and ensemble, with specific monitors designed for each method to provide online output representing prediction performance. Comprehensive experiments validate the PMBP’s effectiveness, comparing different monitoring methods. Results show that the PMBP effectively achieves promising monitoring performance, particularly excelling in deep learning-based monitoring. It achieves improvement scores of 0.81 and 0.79 for average prediction error and final prediction error monitoring, respectively, outperforming previous white-box and gray-box methods. Furthermore, the PMBP’s applicability is validated on different datasets and prediction models, while ablation studies confirm the effectiveness of the proposed mechanism. Hybrid prediction and autonomous driving planning experiments further show the PMBP’s value from an application perspective. Project page: https://swb19.github.io/PMBP/.Note to Practitioners—This research presents PMBP, a valuable tool for practitioners in the automation industry. The PMBP enables online monitoring of black-box trajectory prediction models, enhancing system reliability and facilitating informed decision-making. The practical application of PMBP lies in improving safety and reliability in critical domains, especially in the context of autonomous vehicles. Black-box trajectory prediction models commonly used in these domains may exhibit unexpected deficiencies, potentially leading to risks. By monitoring the prediction performance online, systems can proactively identify potential insufficiencies and make informed decisions to ensure safer and more reliable operations. The PMBP offers practitioners different monitoring solutions based on various approaches, addressing their specific needs effectively. While the PMBP has shown promising outcomes, further exploration and testing are necessary to fully harness and apply its monitoring results in automated systems. Practitioners are encouraged to adopt the PMBP as an essential monitoring mechanism to enhance the reliability of their trajectory prediction models and achieve safer and more efficient automation in their domains.

This paper proposes a Learning-Enhanced Adaptive Decision-Making (LEAD) framework for autonomous vehicles (AVs) focusing on dynamic merging scenarios. To capture the competitive and strategic nature of vehicle interactions, we develop an interaction behavior model based on non-cooperative game theory. The behavior is modeled as a dynamic game, where each vehicle optimizes its actions using a multifactorial reward function. To optimize the behavior model parameters, maximum entropy inverse reinforcement learning (IRL) is employed to acquire optimal matching parameters. Additionally, a behavioral decision-making framework LEAD adapted to dynamic environments is proposed. By establishing a mapping between environmental variables and behavior model parameters, it enables parameters online learning and recognition, and achieves interactive behavior probabilities of AVs. Quantitative analysis employing naturalistic driving datasets (highD and exiD) and real-vehicle test data validates LEAD’s high consistency with human decision-making. In 188 tested interaction scenarios, the average human-like similarity rate is 81.73%, with a notable 83.12% in the highD dataset. Furthermore, in 145 dynamic interactions, LEAD matches human decisions at 77.12%, with 6913 consistence instances. Moreover, in real-vehicle tests, a 72.73% similarity with 0% safety violations is obtained. Results demonstrate the effectiveness of our LEAD framework in enabling AVs to make informed, adaptive behavior decisions in interactive environments.

Autonomous driving confronts great challenges in complex traffic scenarios, where the SOTIF risk can be triggered by the dynamic operational environment and system insufficiencies. The SOTIF risk is reflected not only intuitively in the collision risk with objects outside the autonomous vehicles, but also inherently in the performance limitation risk of the implemented algorithms. How to minimize the SOTIF risk for autonomous driving is currently a critical, difficult, and unresolved issue. Therefore, this paper proposes the “Self-Surveillance and Self-Adaption System” as a systematic approach to online minimize the SOTIF risk, which aims to provide a systematic solution for monitoring, quantification, and mitigation of inherent and external risks. As a demonstration of the system, the risk monitoring of the perception algorithm is highlighted. Moreover, the inherent perception algorithm risk and external collision risk are jointly quantified via SOTIF entropy, which is then propagated downstream to the decision-making module and mitigated. Finally, Hardware-in-the-Loop experiments are conducted to verify the efficiency and effectiveness of the system. The results demonstrate that the system enables dependable online monitoring, quantification, and mitigation of SOTIF risk in real-time critical traffic environments.

Motion prediction modules are crucial for autonomous vehicles to forecast the future behavior of surrounding road users. Failures in prediction modules can mislead a downstream planner to make unsafe decisions. Currently, deep learning technology has been widely used to design prediction models due to its impressive performance. However, such models may fail in long-tail driving scenarios where the training data are insufficient or unavailable, which represents the so-called epistemic uncertainty of prediction models. This paper proposes a risk-aware decision-making (RADM) framework to handle the epistemic uncertainty arising from training the prediction model on insufficient data. First, a multi-agent prediction network with epistemic uncertainty quantification is proposed. This network uses the historical states of nearby road users, map information, and traffic lights as inputs. Then, the RADM utilizes model predictive control technique to not only process the multi-agent prediction results but also to consider the epistemic uncertainty of the prediction model. In addition, the accuracy of the established prediction model is verified on real-world driving datasets. Furthermore, the proposed RADM is evaluated on the log-replay data obtained from real-world driving logs and using the SUMO simulator, considering multiple challenging cases where pedestrians and non-motorized vehicles cross the intersection illegally. The experimental results demonstrate that RADM can reduce the driving risk and improve driving safety and supplementary videos are provided athttps://github.com/SOTIF-AVLab/RADM.

Ride-sharing has important implications for improving the efficiency of mobility-on-demand systems. However, it remains a challenge due to the complex dynamics between vehicles and requests. This paper presents a decentralized ride-sharing algorithm suitable for shared autonomous vehicles (SAVs) deployment. The ride-sharing problem is formulated as a multi-agent reinforcement learning problem. We explore state representation with the request-vehicle graph to encode shareability and potential coordination information. We use a graph attention network to build a hierarchical structure that unifies ride-sharing assignments with rebalancing and handles real-world scenarios where hundreds of user requests can be associated with vehicles. We show results in both generic grid-world and SUMO simulation with real-world data from the Manhattan area. We empirically demonstrate that our proposed approach can achieve similar performance compared with a state-of-the-art centralized optimization method and higher computation efficiency.

Experienced human drivers always make safe driving decisions by selectively observing the front, rear and side- view mirrors. Several end - to-end methods have been pro-posed to learn driving models with multi-view visual infor-mation. However, these benchmark methods lack semantic understanding of multi-view image contents, where human drivers usually reason these information for decision making with different visual region of interests. In this paper, we propose an attention-based deep learning method to learn a driving model with input of surround-view visual information and the route planner, in which a multi-view attention module is designed for obtaining region of interests from human drivers. We evaluate our model on the Drive360 dataset with comparison of benchmarking deep driving models. Results demonstrate that our model achieves a competitive accuracy in both steering angle and speed prediction than benchmarking methods. Code is available at https://githuh.com/jet-uestc/MVA-Net.

We study the problem of eco-routing for Plug-In Hybrid Electric Vehicles (PHEVs) to minimize the overall energy consumption cost. We propose an algorithm which can simultaneously calculate an energy-optimal route (eco-route) for a PHEV and an optimal power-train control strategy over this route. In order to show the effectiveness of our method in practice, we use a HERE Maps API to apply our algorithms based on traffic data in the city of Boston with more than 110,000 links. Moreover, we validate the performance of our eco-routing algorithm using speed profiles collected from a traffic simulator (SUMO) as input to a high-fidelity energy model to calculate energy consumption costs. Our results show significant energy savings (around 12%) for PHEVs with a near real-time execution time for the algorithm.

Shared Mobility-on-Demand using automated vehicles can reduce energy consumption and cost for future mobility. However, its full potential in energy saving has not been fully explored. An algorithm to minimize fleet fuel consumption while satisfying customers’ travel time constraints is developed in this article. Numerical simulations with realistic travel demand and route choice are performed, showing that if fuel consumption is not considered, the Mobility-on-demand (MOD) service can increase fleet fuel consumption due to increased empty vehicle mileage. With fuel consumption as part of the cost function, we can reduce total fuel consumption by 7% while maintaining a high level of mobility service.

This article presents a model-based motion planning and control system for autonomous vehicles and its experimental validation. The system consists of four modules: 1) global routing; 2) behavior planner; 3) local trajectory generation; and 4) trajectory tracking. The algorithm and software of each module are detailed, including a behavior planner with unified models to handle typical scenarios in both highway and urban driving, a deterministic sampling algorithm for robust responsive trajectory generation, and a dynamics-and-delay-aware preview algorithm to achieve accurate trajectory tracking. The developed system is implemented and tested at the Mcity test facility with a full-size automated car and a dozen of challenging traffic scenarios.

Exiting from highways in crowded dynamic traffic is an important path planning task for autonomous vehicles (AVs). This task can be challenging because of the uncertain motion of surrounding vehicles and limited sensing/observing window. Conventional path planning methods usually compute a mandatory lane change (MLC) command, but the lane change behavior (e.g., vehicle speed and gap acceptance) should also adapt to traffic conditions and the urgency for exiting. In this paper, we propose a reinforcement learning-enhanced highway-exit planner. The learning-based strategy learns from past failures and adjusts the vehicle motion when the AV fails to exit. The reinforcement learning is based on the Monte Carlo tree search (MCTS) approach. The proposed learning-enhanced highway-exit planner is tested 6000 times in stochastic simulations. The results indicate that the proposed planner achieves a higher probability of successful highway exiting than a benchmark MLC planner.