Understanding traffic accident scenes is a long-standing research for vision-based safe driving. It seeks to answer why accidents occur, how near-crash scenes develop, and what the key elements of an accident are. This research is challenging due to the scarcity and fragmentation of accident data, as well as the complex accident environments. To study this, we present a framework of Abductive Driving accident Video understanding (ADVersa), which infers a plausible visual and textual explanation for the absent near-crash scenes. ADVersa underscores three groups of tasks: 1) visual past recovery of near-crash scenes, 2) visual prediction of near-crash scenes, and 3) accident cause involved video synthesis. To support the study, we first contribute MM-AU, a novel dataset for Multi-Modal Accident video Understanding. MM-AU contains 11,727 in-the-wild driving accident videos with temporally aligned text descriptions, 2.23 million well-annotated object boxes, and 58,650 pairs of video-based accident cause texts. We then propose an Abductive CLIP model and a Contrastive Graph Video Pre-training (CGVP) model, which exploit relation-aware cross-modal semantic learning to drive spatially abductive and temporally abductive accident video diffusion. Extensive experiments verify the superiority of ADVersa to the state-of-the-art approaches on different tasks, i.e., historical near-crash video frame recovering, crashing video frame prediction, textual accident cause and category reasoning, normal-to-accident video synthesis, and accident video editing. With these efforts, we hope this research can advance the progress on multimodal accident video understanding.

End-to-end autonomous driving has emerged as a promising paradigm for directly mapping sensor inputs to planning maneuvers using learning-based modular integrations. However, existing imitation learning (IL)-based models suffer from generalization to hard cases, and a lack of corrective feedback loop under post-deployment. While reinforcement learning (RL) offers a potential solution to tackle hard cases with optimality, it is often hindered by overfitting to specific driving cases, resulting in catastrophic forgetting of generalizable knowledge and sample inefficiency. To overcome these challenges, we propose Reinforced Refinement with Self-aware Expansion (R2SE), a novel learning pipeline that constantly refines hard domain while keeping generalizable driving policy for model-agnostic end-to-end driving systems. Through reinforcement fine-tuning and policy expansion that facilitates continuous improvement, R2SE features three key components: 1) Generalist Pretraining with hard-case allocation trains a generalist imitation learning (IL) driving system while dynamically identifying failure-prone cases for targeted refinement; 2) Residual Reinforced Specialist Fine-tuning optimizes residual corrections using reinforcement learning (RL) to improve performance in hard case domain while preserving global driving knowledge; 3) Self-aware Adapter Expansion dynamically integrates specialist policies back into the generalist model, enhancing continuous performance improvement. Experimental results in closed-loop simulation and real-world datasets demonstrate improvements in generalization, safety, and long-horizon policy robustness over state-of-the-art E2E systems, highlighting the effectiveness of reinforce refinement for scalable autonomous driving.

Autonomous Vehicles (AVs) have entered the stage of commercialization, yet their performance in interactive scenarios remains unsatisfactory due to challenges such as decision interpretability, human driver (HV) heterogeneity, and scenario diversity. Recent advances in Large Language Models (LLMs) provide a promising avenue to enhance AV interaction capabilities, but their high computational demand hinders practical deployment. To address these challenges, this paper introduces a parallel Actor–Reasoner framework designed to enable explicit and real-time bidirectional AV-HV interactions. First, the Reasoner employs a localized LLM with CoT reasoning and human instructions to progressively infer HV intent, style, AV action, and eHMI displays during the training stage. During testing, it continues to infer he above information except AV action. The Actor, in turn, is constructed as an interaction memory through the Reasoner’s interactions with heterogeneous simulated HVs across diverse scenarios, where the memory partition and two-layer retrieval modules are employed in the construction process. During testing, the Actor is used to retrieve feasible actions for the AV. Ablation studies across multiple scenarios demonstrate that the proposed modules improve interaction success rates by an average of 15% and 12%, respectively. Moreover, comparison studies in multi-vehicle scenarios further show that the proposed Actor–Reasoner framework achieves superior safety while simultaneously improving efficiency. Finally, with the integration of external Human–Machine Interface (eHMI) information derived from the Reasoner’s reasoning and feasible actions retrieved from the Actor, the framework is validated in real-world field interactions. Our code is available athttps://github.com/FanGShiYuu/Actor-Reasoner

Existing traffic simulation models often fall short in capturing the intricacies of real-world scenarios, particularly the interactive behaviors among multiple traffic participants, thereby limiting their utility in the evaluation and validation of autonomous driving systems. We introduce Versatile Behavior Diffusion (VBD), a novel traffic scenario generation framework based on diffusion generative models that synthesizes scene-consistent, realistic, and controllable multi-agent interactions. VBD achieves strong performance in closed-loop traffic simulation, generating scene-consistent agent behaviors that reflect complex agent interactions. A key capability of VBD is inference-time scenario editing through multi-step refinement, guided by behavior priors and model-based optimization objectives, enabling flexible and controllable behavior generation. Despite being trained on real-world traffic datasets with only normal conditions, we introduce conflict-prior and game-theoretic guidance approaches. These approaches enable the generation of interactive, customizable, or long-tail safety-critical scenarios, which are essential for comprehensive testing and validation of autonomous driving systems. Extensive experiments validate the effectiveness and versatility of VBD and highlight its promise as a foundational tool for advancing traffic simulation and autonomous vehicle development. Project website:https://sites.google.com/view/versatile-behavior-diffusion

Motion control is a fundamental task in autonomous vehicle systems, and data-driven approaches offer the advantage of independence from explicit vehicle dynamics modeling. This paper presents a novel data-driven, model-free predictive control method that operates within a receding horizon framework, eliminating the need for pre-identified system models. First, we establish a signal-based system representation and provide a theoretical validation demonstrating its capability to encapsulate vehicle dynamics without requiring explicit model approximation. This representation is then seamlessly integrated into the predictive control framework, ensuring real-time adaptability to dynamic uncertainties. Unlike conventional methods, the proposed approach continuously updates vehicle dynamics using historical driving data, thereby enhancing robustness against time-varying parameters. Moreover, the method is designed to function effectively even when only partial state measurements are available, addressing a key challenge in real-world applications where full state observability is often impractical. To validate its performance, Rapid Control Prototype (RCP) experiments are conducted under both fully and partially measurable state conditions. Results confirm that the proposed approach achieves superior tracking performance compared to traditional model predictive control (MPC) methods, particularly in scenarios where vehicle state estimation is incomplete.

In highly interactive driving environments, the collision accidents and severe congestion often occur due to the multi-modal driving behaviors and dynamic interactions of surrounding vehicles. Moreover, the balance between individual and overall traffic is hard to maintain as well. Such challenging scenarios have become a major obstacle for its real-world application. Owing to the significant advancements of the IoT and AI, multi agent method and the awareness of driving risk have become promising way to enhance the safety problems. To improve the safety and efficiency in highly dynamic and interactive environment, a predictive risk-aware cooperative driving policy is proposed based on multi-agent reinforcement learning (MARL) framework. Firstly, considering multiple future behaviors and dynamic evolution of surrounding vehicles, a predictive risk field is established based on elliptical collision boundary. And then, the predictive risk is introduced in our cooperative driving policy to enhance the safety in highly interactive environment. Furthermore, efficiency of individual vehicle and overall traffic flow are balanced when we design the reward function. Subsequently, to learn the optimal cooperative driving policy, an advantage actor-critic algorithm is also designed, in which the predictive risk is integrated into the centralized critic, facilitating comprehension of global traffic situation and coordinate all vehicles to ensure safety. Moreover, the risk-aware critic guides the policy updating of decentralized actor for each controlled vehicle. In this way, our approach could improve the comprehension of interactive and dynamic traffic environment. The results demonstrate that the introduction of predictive risk and multi-agent systems can make a trade off between individual and overall traffic efficiency within safety constraints, even in challenging traffic situations with high-density flows.

Human-vehicle cooperative driving serves as a vital bridge to fully autonomous driving by improving driving flexibility and gradually building driver trust and acceptance of autonomous technology. To establish more natural and effective human-vehicle interaction, we propose a Human-Oriented Cooperative Driving (HOCD) approach that primarily minimizes human-machine conflict by prioritizing driver intention and state. In implementation, we take both tactical and operational levels into account to ensure seamless human-vehicle cooperation. At the tactical level, we design an intention-aware trajectory planning method, using intention consistency cost as the core metric to evaluate the trajectory and align it with driver intention. At the operational level, we develop a control authority allocation strategy based on reinforcement learning, optimizing the policy through a designed reward function to achieve consistency between driver state and authority allocation. The results of simulation and human-in-the-loop experiments demonstrate that our proposed approach not only aligns with driver intention in trajectory planning but also ensures a reasonable authority allocation. Compared to other cooperative driving approaches, the proposed HOCD approach significantly enhances driving performance and mitigates human-machine conflict.

A hybrid Koopman deep learning algorithm is developed to predict the final synchronization of networked nonlinear dynamics with different topologies merely using neighboring state information. This algorithm introduces a nonlinear encoder as an observable function that maps the nonlinear state into a high-dimensional Hilbert space. By this means, a networked linear model is established to predict the future state of multiple transformed linear systems in the lifted space. Meanwhile, a nonlinear decoder is constructed, as the inverse of the lifting function, to retrieve the original nonlinear states. The virtue of the present algorithm lies in distilling and merging the linear features of multiple different topologies solely from the individual and/or neighboring state series. Therefore, the final synchronization states are calculated within the encoded linear space and subsequently decoded to recover the synchronization of the original nonlinear systems. Compared to most existing relevant algorithms that could only predict consensus values for linear networks, the present method could predict the final synchronization state of networked nonlinear dynamics with varying backbones. Sufficient conditions are derived to guarantee the prediction capability of the distributed final synchronization prediction (DFSP). Extensive numerical simulations verify its effectiveness.

Reinforcement learning (RL) with delays is challenging as sensory perceptions lag behind the actual events: the RL agent needs to estimate the real state of its environment based on past observations. State-of-the-art (SOTA) methods typically employ recursive, step-by-step forecasting of states. This can cause the accumulation of compounding errors. To tackle this problem, our novel belief estimation method, named Directly Forecasting Belief Transformer (DFBT), directly forecasts states from observations without incrementally estimating intermediate states step-by-step. We theoretically demonstrate that DFBT greatly reduces compounding errors of existing recursively forecasting methods, yielding stronger performance guarantees. In experiments with D4RL offline datasets, DFBT reduces compounding errors with remarkable prediction accuracy. DFBT’s capability to forecast state sequences also facilitates multi-step bootstrapping, thus greatly improving learning efficiency. On the MuJoCo benchmark, our DFBT-based method substantially outperforms SOTA baselines. Code is available at https://github.com/QingyuanWuNothing/DFBT.

Autonomous driving necessitates the ability to reason about future interactions between traffic agents and to make informed evaluations for planning. This paper introduces the Gen-Drive framework, which shifts from the traditional prediction and deterministic planning framework to a generation-then-evaluation planning paradigm. The framework employs a behavior diffusion model as a scene generator to produce diverse possible future scenarios, thereby enhancing the capability for joint interaction reasoning. To facilitate decision-making, we propose a scene evaluator (reward) model, trained with pairwise preference data collected through VLM assistance, thereby reducing human workload and enhancing scalability. Furthermore, we utilize an RL fine-tuning framework to improve the generation quality of the diffusion model, rendering it more effective for planning tasks. We conduct training and closed-loop planning tests on the nuPlan dataset, and the results demonstrate that employing such a generation-then-evaluation strategy outperforms other learning-based approaches. Additionally, the fine-tuned generative driving policy shows significant enhancements in planning performance. We further demonstrate that utilizing our learned reward model for evaluation or RL fine-tuning leads to better planning performance compared to relying on human-designed rewards. Project website: https://mczhi.github.io/GenDrive.

While modeling multi-contact manipulation as a quasi-static mechanical process transitioning between different contact equilibria, we propose formulating it as a planning and optimization problem, explicitly evaluating (i) contact stability and (ii) robustness to sensor noise. Specifically, we conduct a comprehensive study on multi-manipulator control strategies, focusing on dual-arm execution in a planar peg-in-hole task and extending it to the Multi-Manipulator Multiple Peg-in-Hole (MMPiH) problem to explore increased task complexity. Our framework employs Dynamic Movement Primitives (DMPs) to parameterize desired trajectories and Black-Box Optimization (BBO) with a comprehensive cost function incorporating friction cone constraints, squeeze forces, and stability considerations. By integrating parallel scenario training, we enhance the robustness of the learned policies. To evaluate the friction cone cost in experiments, we test the optimal trajectories computed for various contact surfaces, i.e., with different coefficients of friction. The stability cost is analytical explained and tested its necessity in simulation. The robustness performance is quantified through variations of hole pose and chamfer size in simulation and experiment. Results demonstrate that our approach achieves consistently high success rates in both the single peg-in-hole and multiple peg-in-hole tasks, confirming its effectiveness and generalizability. The video can be found at https://youtu.be/IU0pdnSd4tE.

Autonomous vehicle trajectory planning faces significant challenges in dynamic traffic environments due to the complex and mixed causal relationships between critical scene elements (e.g., pedestrians, vehicles, road markings) and safe decision-making. To identify the causal factors influencing planning outcomes, we propose Causal-Planner, which disentangles the scene interaction graph into causal and confounding components via attention-based adversarial graph learning. Additionally, we introduce a long-short-term episodic memory gating (LSTEM) module that enhances causal interaction disentangling by adaptively capturing evolving causal relationships in dynamic scenarios through bidirectional gated memory fusion. Extensive experiments on the nuPlan dataset suggest that Causal-Planner achieves competitive performance, performing well in both Test-random and Test-hard scenarios under open-loop and closed-loop evaluations. The code will be publicly available at https://github.com/Yyb-XJTU/Causal-Planner.

Driver monitoring systems face critical challenges in modern transportation, including limited multitasking capabilities and a lack of interpretability. These limitations hinder the accurate and comprehensive assessment of driver states such as distraction, drowsiness, and emotions, which are essential to ensure road safety. This paper introduces visual language models for multitask domain adaptation in driver monitoring (VLM-DM), a novel framework that addresses these challenges by leveraging advanced visual language models for the simultaneous execution of multiple driver monitoring tasks. By employing parameter-efficient training methods such as Low-Rank Adaptation (LoRA) and integrating dynamic prompt tuning, VLM-DM achieves superior performance compared to state-of-the-art methods. Our experiments on three benchmark datasets across different driver states, demonstrating significant improvements in multitask accuracy and interpretability. This work highlights the potential of advanced multitask and multimodal architectures in developing robust, scalable, and interpretable driver monitoring systems for real-world applications.

Modern autonomous vehicle perception systems often struggle with occlusions and limited perception range. Previous studies have demonstrated the effectiveness of cooperative perception in extending the perception range and overcoming occlusions, thereby enhancing the safety of autonomous driving. In recent years, a series of cooperative perception datasets have emerged; however, these datasets primarily focus on cameras and LiDAR, neglecting 4D Radar—a sensor used in single-vehicle autonomous driving to provide robust perception in adverse weather conditions. In this paper, to bridge the gap created by the absence of 4D Radar datasets in cooperative perception, we present V2X-Radar, the first large-scale, real-world multi-modal dataset featuring 4D Radar. V2X-Radar dataset is collected using a connected vehicle platform and an intelligent roadside unit equipped with 4D Radar, LiDAR, and multi-view cameras. The collected data encompasses sunny and rainy weather conditions, spanning daytime, dusk, and nighttime, as well as various typical challenging scenarios. The dataset consists of 20K LiDAR frames, 40K camera images, and 20K 4D Radar data, including 350K annotated boxes across five categories. To support various research domains, we have established V2X-Radar-C for cooperative perception, V2X-Radar-I for roadside perception, and V2X-Radar-V for single-vehicle perception. Furthermore, we provide comprehensive benchmarks across these three sub-datasets.

Evidences have been provided that the effect of poorly designed vehicle automation systems may propagate from the single car up to the traffic dynamics. A reported example consists of car-following driver assistance systems triggering destabilizing group phenomena like phantom traffic jams and stop-and-go waves. It then becomes fundamental to study ‘group-oriented’ vehicle shared control algorithms that are able to assist the driver while at the same time prevent the propagation of destabilizing effects in the traffic when these systems are widely deployed. A key challenge in vehicle shared control is that the heterogeneity and uncertainty of human driving characteristics require personalized adaptation: it is an open problem to realize stabilizing traffic properties from personalized adaptive vehicle shared control. The distinguishing contribution of this work is a car-following shared control method that, while adapting to the personal characteristics of each driver, contains a ‘group’ model with desirable traffic properties defined in terms of string stability and collision avoidance. It is proven analytically that the proposed shared control is able to assist each driver in approaching the group model adaptively (i.e., handling heterogeneity and uncertainties) and optimally (i.e., with minimum control authority over the driver). Numerical experiments performed in SUMO with Highway Fuel Economy Test Cycle (HWFET) data and stop-and-go wave data validate that the proposed assistance improves traffic smoothing while handling heterogeneity and uncertainties in the driver parameters.

Autonomous driving systems require a comprehensive understanding and accurate prediction of the surrounding environment to facilitate informed decision-making in complex scenarios. Recent advances in learning-based systems have highlighted the importance of integrating prediction and planning. However, this integration poses significant alignment challenges through consistency between prediction patterns, to interaction between future prediction and planning. To address these challenges, we introduce a Hybrid-Prediction integrated Planning (HPP) framework, which operates through three novel modules collaboratively. First, we introduce marginal-conditioned occupancy prediction to align joint occupancy with agent-specific motion forecasting. Our proposed MS-OccFormer module achieves spatial-temporal alignment with motion predictions across multiple granularities. Second, we propose a game-theoretic motion predictor, GTFormer, to model the interactive dynamics among agents based on their joint predictive awareness. Third, hybrid prediction patterns are concurrently integrated into the Ego Planner and optimized by prediction guidance. The HPP framework establishes state-of-the-art performance on the nuScenes dataset, demonstrating superior accuracy and safety in end-to-end configurations. Moreover, HPP's interactive open-loop and closed-loop planning performance are demonstrated on the Waymo Open Motion Dataset (WOMD) and CARLA benchmark, outperforming existing integrated pipelines by achieving enhanced consistency between prediction and planning.

Industry 5.0 heralds a new wave of the industrial revolution, placing a spotlight on human-centric intelligent manufacturing. At the core of Industry 5.0 lies the human-cyber-physical system (HCPS), a composite intelligent system where interactions among humans, cyberspace, and physical assets are orchestrated across diverse manufacturing levels and phases. Understanding the pivotal roles played by humans in these advanced systems is of paramount importance. Nonetheless, the exploration of HCPS within the context of Industry 5.0 remains in its infancy. This paper presents a holistic literature review of industrial HCPS from a human-centric perspective. A united architecture is employed to encompass the aspects of cognitive-to-technology integration and human-to-human interaction in HCPS, highlighting human-in-the-loop, human-on-the-loop, and human-in-the-society paradigms. The mechanisms of these paradigms and their effects on design, production, and service are investigated to expand the research landscape of intelligent manufacturing in Industry 5.0. Key enabling technologies that facilitate harmonious tri-space integration are introduced, and the future challenges of industrial HCPS are discussed. This work is expected to attract more open discussions and in-depth research on HCPS in the new industrial revolution era.Note to Practitioners—This paper is motivated by the emergence of Industry 5.0 that integrates humans into cyber-physical systems to offset drawbacks on both sides. It presents an overview of HCPS-related works to identify the state-of-the-art and open problems in the Industry 5.0 era. The review of HCPS applications in the design, production, and service phases can benefit engineers in the intelligent manufacturing area. Key enabling technologies on human ability augmentation, human-robot interaction, digital twin, human-cyber-physical data fusion, crowdsourcing, and system modeling, are analyzed to attract researchers in broader research fields to join in the development of industrial HCPS.

Homomorphic encryption enables secure control of networked systems with untrusted computing entities but greatly increases communication overhead compared with plaintext-based control. To address this, we propose a dynamic mode-switching secure control framework that alternates between plaintext and encrypted operations. In plaintext mode, sensor measurements are obfuscated using random dithered quantization, while in encrypted mode, measurements are fully encrypted to provide enhanced system security. To evaluate the framework, a worst-case eavesdropping scenario is introduced, where the adversary has complete knowledge of the system model and access to all plant-controller communications. Within this setting, we develop three switching strategies—periodic, random, and error-based—to govern the operational mode. Rigorous theoretical analysis establishes formal guarantees for both control performance and security, alongside deriving a critical parameter condition for decryption correctness. Ensuring the signal-to-noise ratio of the eavesdropper’s estimate remains below 10 dB, simulations show that periodic and random switching reduce communication by at least 30%. Error-based switching with an appropriate threshold (β = 1 × 10−4) achieves more than 70% reduction. These results confirm that the proposed framework effectively balances control performance, system security, and communication overhead, rendering it well-suited for resource-constrained networked systems.

Pedestrian crossing prediction (PCP) in driving scenes plays a critical role in ensuring the safe decision of intelligent vehicles. Due to the limited observations and annotations of pedestrian crossing behaviors in real situations, recent studies have begun to leverage synthetic data with flexible variation to boost prediction performance, employing domain adaptation frameworks. However, different domain knowledge has distinct cross-domain distribution gaps, which necessitates suitable domain knowledge adaption ways for PCP tasks. In this work, we propose a gated syn-to-real knowledge transfer approach for PCP (Gated-S2R-PCP), which has two aims: 1) designing the suitable domain adaptation ways for different kinds of crossing-domain knowledge, and 2) transferring suitable knowledge for specific situations with gated knowledge fusion. Specifically, we design a framework that contains three domain adaption methods including style transfer, distribution approximation, and knowledge distillation for various information, such as visual, semantic, depth, bounding boxes, etc. A learnable gated unit (LGU) is employed to fuse suitable cross-domain knowledge to boost pedestrian crossing prediction. We construct a new synthetic benchmark S2R-PCP-3181 with 3181 sequences (489,740 frames) which contains the pedestrian bounding boxes, RGB frames, semantic segmentation maps, and depth maps. With the synthetic S2R-PCP-3181, we transfer the knowledge to two real challenging datasets of PIE and JAAD, and superior PCP performance is obtained to the state-of-the-art methods.

The length of time for drivers to take over conditionally automated driving vehicles (CADV) is often influenced by numerous factors, especially the mixed traffic flow environment. To analyze the influencing factors of driver’s takeover time in the mixed traffic flow environment, we study a real-vehicle takeover experiment with the CADV and radar video integrated machine provided by Jiangsu University. DeepGBM algorithm and Shapley additive explanation (SHAP) are utilized to predict and analyze the takeover time based on the data obtained from the real-vehicle experiment. The results show that the DeepGBM algorithm performs a better accuracy in predicting the takeover time compared with the other algorithms. Moreover, the predicting effect of DeepGBM in the scene of average takeover time is better than the shorter and longer ones. On the other hand, when taking over CADV in the intersection area, the driver pays more attention to the vehicle’s speed. While in the non-intersection area, the driver is more attentive to the longitudinal distance difference with the vehicle ahead. This study further found that CADVs are subject to significant lateral interference for the takeover process in intersection areas, significantly impacting the driver’s takeover time and vehicle safety. This study can provide a theoretical reference for automated driving companies to design takeover times.

Simulation environments featuring data-driven traffic agents have become essential tools for training and assessing autonomous driving (AD) systems, offering a safer and more cost-effective alternative to real-world testing. However, prevalent data-driven simulations often focus on modeling individual traffic participants or indirectly representing their joint future motion distribution in a latent space. These approaches may encounter challenges due to the combinatorial explosion in the number of agents or may lack explainability. To tackle these issues, we introduce a novel learning-based traffic simulation framework aimed at directly modeling joint behaviors in the 2D plane. Our framework utilizes a graph-based scene representation to accommodate an arbitrary number of agents and road elements. It directly generates multiple joint goal sets for all agents in a given scenario and subsequently produces reference lines for each goal set. To enhance interaction with the ego, we design a predictive step simulation module to forecast the ego’s short-term motion and prevent collisions between the step rollout and the prediction during reference line tracking. Experimental results on the large-scale Waymo Open Motion Dataset validate that the proposed model can directly generate multiple socially consistent goal sets, showcasing enhancements in collision avoidance, road compliance, and overall realism. This approach shows promise in advancing the development and deployment of AD technology in a safer and more efficient manner.

Effective scene representation is critical for trajectory generation tasks in autonomous driving. Existing attention-based methods often rely on fixed-length input elements, limiting their ability to adapt to dynamic traffic environments, and frequently overlook lane connectivity. Methods that consider lane connectivity often segment lanes into smaller pieces, which consequently creates a more complex graph structure with an increased number of nodes and edges. Additionally, many current approaches select agent interactions based on fixed distance thresholds, which may miss important long-range or indirect interactions and ignore the fact that proximity does not indicate interaction. In this paper, we propose SceneGNN, a novel framework for learning traffic scenario representation through interaction graph learning and lane graph exploration. SceneGNN constructs a single heterogeneous graph that integrates both agents and lanes, leveraging high-definition maps without over-segmentation. To model lane connectivity more effectively, we introduce LaneGNN, which combines an Omnidirectional Lane Aggregator (OLA) and Directional Lane Explorer (DLE) to explore lane-to-lane interdependencies. Additionally, instead of relying on proximity-based heuristics, our interaction graph learner dynamically constructs inter-agent edges based on learned features, allowing the model to capture meaningful interactions beyond mere distance. We evaluate SceneGNN on the Waymo Open Motion Dataset, where it achieves competitive performance. Our results demonstrate that by efficiently capturing both agent-lane relationships and inter-agent interactions, SceneGNN improves the accuracy and scalability of multi-agent trajectory prediction for real-world autonomous driving applications.

Owing to the shortage of computing resources for autonomous vehicles and redundant modeling among similar tasks, multi-task models have become a feasible solution. The multi-task prediction model of autonomous vehicles refers to the realization of trajectory, behavior, and risk predictions through a multi-task deep neural network. However, whether the multi-task prediction networks can effectively share information between multiple inputs and whether the shared representations are interpretable remains a concern. To address the aforementioned concerns, this study proposes a multi-source multi-dimensional model interpretation (M3-interpretation) method for multi-task prediction neural network (MPNN). The MPNN proposed in this paper is designed with a structure that emphasizes a “task-specific pipeline as the main, high-level semantic information sharing as the supplement”. Then, based on the information entropy theory, this study creatively extends the information bottleneck attribution method to M3 and uses feature masks to display fine-grained interpretation results. Comparison and ablation experiments using naturalistic trajectory datasets indicated that the proposed model has better prediction performance than single-task models. In addition, fine-grained attribution analysis was conducted on specific behaviors in temporal, spatial, and feature dimensions to explore the laws that affect behavioral inference in MPNN.

Trajectory prediction is significant for intelligent vehicles to achieve high-level autonomous driving, and a lot of relevant research achievements have been made recently. Despite the rapid development, most existing studies solely focus on normal and safe scenarios while largely neglecting safety-critical scenarios, particularly those involving imminent collisions. This oversight may result in autonomous vehicles lacking the essential predictive ability in such situations, posing a significant threat to safety. To tackle these, this paper proposes a risk-aware trajectory prediction framework tailored to safety-critical scenarios. Leveraging distinctive hazardous features, we develop three core risk-aware components. First, we introduce a risk-incorporated scene encoder, which augments conventional encoders with quantitative risk information to achieve risk-aware encoding of hazardous scene contexts. Next, we incorporate endpoint-risk-combined intention queries as prediction priors in the decoder to ensure that the predicted multimodal trajectories cover both various spatial intentions and risk levels. Lastly, an auxiliary risk prediction task is implemented for the ultimate risk-aware prediction. Furthermore, to support model training and performance evaluation, we introduce a safety-critical trajectory prediction dataset and tailored evaluation metrics. We conduct comprehensive evaluations and compare our model with several SOTA models. Results demonstrate the superior performance of our model, with a significant improvement in most metrics. This prediction advancement enables autonomous vehicles to execute correct collision avoidance maneuvers under safety-critical scenarios, eventually enhancing road traffic safety.

Simultaneous localization and mapping (SLAM) is a critical component of autonomous vehicles, which can estimate their current pose and construct a precision map of the environment. However, its performance is often limited by insufficient perception ability and non-robust odometry. In this paper, we introduce a Progressive Multi-Modal Semantic Segmentation guided SLAM (PM2S2-SLAM), which utilizes tightly-coupled LiDAR-Visual-Inertial odometry with multi-modal semantic information to enhance the robustness and accuracy of SLAM. To address the limitations of a single sensor based perception method and the inefficiency of multi-modal semantic networks, a progressive multi-modal network is designed to efficiently extract multi-sensor semantic information in a segmentation network. This approach progressively enhances the subsequent point cloud segmentation network with calibration and image semantics prior, thereby improving the accuracy and efficiency of perception. Additionally, we propose semantic information enhanced tightly-coupled LiDAR-visual-inertial odometry, which employs the semantic trimmed iterative closest point method to enhance the robustness and accuracy of multi-modal odometry. Finally, the effectiveness of the PM2S2-SALM is verified by real-world experiments through the public datasets, which reduces the Absolute Trajectory Error by 25.1% compared with the state-of-the-art performance method.

Roadside perception can significantly enhance the safety of autonomous vehicles by extending their perceptual capabilities beyond the visual range and addressing occluded regions. However, current state-of-the-art vision-based roadside detection methods exhibit high accuracy on labeled scenes but perform poorly on new scenes. This limitation arises because roadside cameras remain stationary after installation and can only gather data from a single scene, leading the algorithm to overfit these roadside backgrounds and camera positions. To tackle this issue, we propose an innovativeScenarioGeneralization Framework forVision-based Roadside3DObject Detection, calledSGV3D. Specifically, we utilize a Background-suppressed Module (BSM) to reduce background overfitting in vision-centric pipelines by diminishing background features during the 2D to bird’s-eye-view projection. Furthermore, by introducing the Semi-supervised Data Generation Pipeline (SSDG) that employs unlabeled images from new scenes, we generate diverse foreground instances with varying camera poses, mitigating the risk of overfitting to specific camera positions. Experiments conducted on two large-scale roadside benchmarks demonstrate that SGV3D, with only a minimal increase in latency, effectively improves the scenario generalization capabilities of vision-based roadside 3D object detectors. The code is available here (https://github.com/yanglei18/SGV3D).

Learning-based techniques hold considerable promise in achieving human-like autonomous driving. However, one deployed policy encounters difficulties in satisfying the drivers’ diverse decision-making preferences simultaneously. Meanwhile, training personalized policies for each driver from scratch is time-consuming and resource-intensive. To address these challenges, this paper proposes a human-guided continual learning framework, wherein the human drivers could real-time take over a deployed policy when it performs unsatisfactorily, and the autonomous vehicle (AV) agent would automatically acquire human demonstrations and dynamically alter itself in accordance with personalized decision-making preference. Furthermore, a priority experience memory-enabled elastic weight consolidation (PEM-EWC) mechanism is developed to prevent the AV agent from overfitting to a limited number of human demonstrations and catastrophically forgetting its acquired fundamental driving abilities. Driver-in-the-loop simulations and real-world experiments are conducted in representative autonomous driving decision-making scenarios, and experimental results demonstrate the superior equilibrium of our proposed approach in terms of driving safety, human likeness, and training efficiency, compared to other baselines, which suggests that it provides a promising solution for personalized decision-making in autonomous driving. The supplementary video is available athttps://youtu.be/HKF0ayxMycc.

Autonomous driving (AD) will rely, either fully or partially, on data-driven approaches. As such, being aware of the algorithm limitation is crucial when implementing learning-based methods in such safety-critical contexts. A comprehensive AD monitor allows control authority to be transferred promptly to a contingency backup solution when the vehicle is recognized in impasses. To address this challenge, we propose MonitorGAN, a Bayesian discriminator trained within an adversarial framework, designed to recognize unknown traffic scenarios and monitor planning quality in open-world autonomous driving. Additionally, it is designed to be aware of its own limitations using a Bayesian approach. Unlike previous epistemic uncertainty estimation algorithms for self-driving, MonitorGAN is independent and planner-agnostic, capable of monitoring various types of planners without requiring real outlier exposure. MonitorGAN is trained exclusively on Argoverse 2 and tested through extensive cross-dataset experiments, including NGISM, HighD, RounD, and NuScenes, across three common planning schemes: learning-based, polynomial-based, and optimization-based, all of which use the same training dataset for interaction-aware planning. Both quantitative results and qualitative comparisons with other epistemic uncertainty estimation algorithms indicate that our approach can estimate the feasibility of the AD’s planning in a planner-agnostic manner and ensure safety.

Advancements in autonomous driving technologies continue to revolutionize transportation, yet the full realization of self-driving vehicles remains hampered by several critical challenges. Automated vehicles continue to encounter significant challenges in perception, prediction, and decision-making, while their low-level modules are relatively mature and robust. Conversely, humans surpass machines in terms of high-level intelligence but may suffer from control performance degradation. To coalesce the strength of the human and machine, a novel collaboration scheme is proposed to compensate for the prediction-decision uncertainty via human guidance while providing low-level control feedback to the driver. This approach generates multiple decision candidates and corresponding predictions for other road users using a transformer-based socially compliant generative adversarial network (SCGAN). The driver can assist in choosing the appropriate candidate using the context-understanding capability, while concurrently, the control projection of this chosen decision guides the driver to achieve the desired objective via haptic steering feedback. The haptic feedback can reflect the decision uncertainties enabled by the decision-control projection of the intention estimation of the ego vehicle. A Type-II fuzzy controller is utilized to determine the control authority to account for the complexity of the future movement. We verify the effectiveness of the proposed algorithm through a real-time human-in-the-loop experiment, including an ablation study and comparisons with other human-machine collaboration schemes. The results demonstrate that the proposed scheme can minimize human-machine conflicts while increasing system safety.

Urban air mobility (UAM) aims to revolutionize transportation in urban airspace by integrating advanced aerial vehicles for efficient, safe, and sustainable short-distance travel within cities. The growth in UAM operations significantly increases collision risks in dense, dynamic urban airspaces filled with buildings, obstacles, and temporary geo-fences triggered by crowded events or severe weather. Addressing the dynamic emergence of geo-fences in environments originally characterized by dense building structures presents significant challenges, and current solutions remain inadequate. In this study, we propose a novel Deep Deterministic Policy Gradient Learning Framework (DDPG), enhanced with Gated Recurrent Units (GRU), introduce a randomized layer after the neural network’s input observation and incorporate Feature Matching (FM) loss. This introduction of the randomized layer boosts the performance of autonomous aerial vehicles in avoiding dynamically emergent geo-fences in environment with dense obstacles and mitigate path deviation. Moreover the similar improvements can be found when AAV operates in unseen environments. We modeled our simulation environment using real spatial data from a residential area in Singapore, dividing it into areas for training and evaluation. Our results demonstrate that employing feature matching gated recurrent unit-deep deterministic policy gradient (FMGRU-DDPG) algorithms enables AAVs to achieve over a 90% success rate in reaching their destinations despite the presence of up to 10 dynamically appearing geo-fences during its flight. Additionally, the algorithm maintains more than a 60% success rate in unseen environments during training which outperforming both GRU-DDPG and DDPG algorithms.

Numerous real-world decision or control problems involve multiple conflicting objectives whose relative importance (preference) is required to be weighed in different scenarios. While Pareto optimality is desired, environmental uncertainties (e.g., environmental changes or observational noises) may mislead the agent into performing suboptimal policies. In this article, we present a novel multiobjective optimization paradigm, robust multiobjective reinforcement learning (RMORL) considering environmental uncertainties, to train a single model that can approximate robust Pareto-optimal policies across the entire preference space. To enhance policy robustness against environmental changes, an environmental disturbance is modeled as an adversarial agent across the entire preference space via incorporating a zero-sum game into a multiobjective Markov decision process (MOMDP). Additionally, we devise an adversarial defense technique against observational perturbations, which ensures that policy variations, perturbed by adversarial attacks on state observations, remain within bounds under any specified preferences. The proposed technique is assessed in five multiobjective environments with continuous action spaces, showcasing its effectiveness through comparisons with competitive baselines, which encompass classical and state-of-the-art schemes.

Reducing undesirable path crossings among tra jectories of different robots is vital in multi-robot navigation missions, which not only reduces detours and conflict scenarios, but also enhances navigation efficiency and boosts productivity. Despite recent progress in multi-robot path-crossing-minimal (MPCM) navigation, the majority of approaches depend on the minimal squared-distance reassignment of suitable desired points to robots directly. However, if obstacles occupy the passing space, calculating the actual robot-point distances becomes complex or intractable, which may render the MPCM navigation in obstacle environments inefficient or even infeasible. In this paper, the concurrent-allocation task execution (CATE) algorithm is presented to address this problem (i.e., MPCM navigation in obstacle environments). First, the path-crossing related elements in terms of (i) robot allocation, (ii) desired-point convergence, and (iii) collision and obstacle avoidance are en coded into integer and control barrier function (CBF) constraints. Then, the proposed constraints are used in an online constrained optimization framework, which implicitly yet effectively minimizes the possible path crossings and trajectory length in obstacle environments by minimizing the desired point allocation cost and slack variables in CBF constraints simultaneously. In this way, the MPCM navigation in obstacle environments can be achieved with flexible spatial orderings. Note that the feasibility of solutions and the asymptotic convergence property of the proposed CATE algorithm in obstacle environments are both guaranteed, and the calculation burden is also reduced by concurrently calculating the optimal allocation and the control input directly without the path planning process. Finally, extensive simulations and experiments are conducted to validate that the CATE algorithm (i) outperforms the existing state-of-the-art baselines in terms of feasibility and efficiency in obstacle environments, (ii) is effective in environments with dynamic obstacles and is adaptable for per forming various navigation tasks in 2D and 3D, (iii) demonstrates its efficacy and practicality by 2D experiments with a multi-AMR onboard navigation system, and (iv) provides a possible solution to evade deadlocks and pass through a narrow gap.

We present MM-AU, a novel dataset for Multi-Modal Accident video Understanding. MM-AU contains 11,727 in-the-wild ego-view accident videos, each with temporally aligned text descriptions. We annotate over 2.23 mil-lion object boxes and 58,650 pairs of video-based accident reasons, covering 58 accident categories. MM-AU supports various accident understanding tasks, particularly multimodal video diffusion to understand accident cause-effect chains for safe driving. With MM-AU, we present an Abductive accident Video unders tanding framework for Safe Driving perception (AdVersa-SD). AdVersa-SD performs video diffusion via an Object-Centric Video Diffusion (OAVD) method which is driven by an abductive CLIP model. This model involves a contrastive interaction loss to learn the pair co-occurrence of normal, near-accident, accident frames with the corresponding text descriptions, such as accident reasons, prevention advice, and accident categories. OAVD enforces the object region learning while fixing the content of the original frame background in video generation, to find the dominant objects for certain accidents. Extensive experiments verify the abductive ability of AdVersa-SD and the superiority of OAVD against the state-of-the-art diffusion models. Additionally, we provide care-ful benchmark evaluations for object detection and accident reason answering since AdVersa-SD relies on precise object and accident reason information.

Reinforcement learning (RL) is challenging in the common case of delays between events and their sensory perceptions. State-of-the-art (SOTA) state augmentation techniques either suffer from state space explosion or performance degeneration in stochastic environments. To address these challenges, we present a novel *Auxiliary-Delayed Reinforcement Learning (AD-RL)* method that leverages auxiliary tasks involving short delays to accelerate RL with long delays, without compromising performance in stochastic environments. Specifically, AD-RL learns a value function for short delays and uses bootstrapping and policy improvement techniques to adjust it for long delays. We theoretically show that this can greatly reduce the sample complexity. On deterministic and stochastic benchmarks, our method significantly outperforms the SOTAs in both sample efficiency and policy performance. Code is available at https://github.com/QingyuanWuNothing/AD-RL.

Motion prediction and cost evaluation are vital components in the decision-making system of autonomous vehicles. However, existing methods often ignore the importance of cost learning and treat them as separate modules. In this study, we employ a tree-structured policy planner and propose a differentiable joint training framework for both ego-conditioned prediction and cost models, resulting in a direct improvement of the final planning performance. For conditional prediction, we introduce a query-centric Transformer model that performs efficient ego-conditioned motion prediction. For planning cost, we propose a learnable context-aware cost function with latent interaction features, facilitating differentiable joint learning. We validate our proposed approach using the real-world nuPlan dataset and its associated planning test platform. Our framework not only matches state-of-the-art planning methods but outperforms other learning-based methods in planning quality, while operating more efficiently in terms of runtime. We show that joint training delivers significantly better performance than separate training of the two modules. Additionally, we find that tree-structured policy planning outperforms the conventional single-stage planning approach. Code is available: https://github.com/MCZhi/DTPP.

While existing Learning from intervention (LfI) methods within the human-in-the-loop reinforcement learning (HiL-RL) paradigm mainly operate on the assumption that human policies are homogeneous and deterministic with low variance, natural human driving behaviors are multimodal with intrinsic uncertainties, and hence, accommodating diverse human capabilities is significant for its practical applications. This work proposes an enhanced LfI approach for learning the optimal RL policy by leveraging multimodal human behaviors in the setting of N-driver concurrent interventions. Specifically, we first learn the N number of human digital drivers from the multi-human demonstration dataset, wherein each driver possesses its own policy distribution. Then, the post-trained drivers will be kept in the training loop of the RL algorithms, providing diverse driving guidance whenever the intervention is required. Additionally, to better utilize the provided guidance, we augment the RL regarding the fundamental architecture and optimization objectives to facilitate the proposed uncertainty-aware reinforcement learning (UnaRL) algorithm. The proposed approach, which won 2ndplace in the Alibaba Future Car Innovation Challenge 2022, is solidly compared in two challenging autonomous driving scenarios against state-of-the-art (SOTA) LfI baselines, and results of both simulation and real-world experiment confirm the superiority of our method in terms of learning robustness and driving performance. Videos and source code are provided.1

Recent advances in multi-agent reinforcement learning (MARL) have significantly enhanced cooperation capabilities within multi-robot teams. However, the application to heterogeneous teams poses the critical challenge of combinatorial generalization—adapting learned policies to teams with new compositions of varying sizes and robots capabilities. This challenge is paramount for dynamic real-world scenarios where teams must swiftly adapt to changing environmental and task conditions. To address this, we introduce a novel transformer-based MARL method for heterogeneous multirobot cooperation. Our approach leverages graph neural networks and self-attention mechanisms to effectively capture the intricate dynamics among heterogeneous robots, facilitating policy adaptation to team size variations. Moreover, by treating robot team decisions as sequential inputs, a capability-oriented decoder is introduced to generate actions in an auto-regressive manner, enabling decentralized decision-making that tailored each robot’s varying capabilities and heterogeneity type. Furthermore, we evaluate our method across two heterogeneous cooperation scenarios in both simulated and real-world environments, featuring variations in team number and robot capabilities. Comparative results reveal our method’s superior generalization performance compared to existing MARL methodologies, marking its potential for real-world multi-robot applications.

Simulation provides a fast, cost-effective, and secure environment for developing autonomous driving systems. However, mitigating the gap between simulation and reality is a challenging task as it demands a behavior simulation method that is human-like, diverse, controllable, socially consistent, and scalable. This work proposes a data-driven traffic agent simulation method to address the aforementioned challenges. Our approach centers around a graph-based scene representation and an encoding method, dividing the simulation into two stages: Multi-Agent Goal assignment (MAG) and Goal-Directed Planning (GDP). Firstly, we create joint goal sets for all agents involved in the scenario. Subsequently, we assign target centerlines (TCLs) to each agent based on their predicted goals. To account for any potential mismatch between the predicted joint goal sets and the road structure, we further align the goals of each agent with their respective assigned TCLs. These on-TCL goals serve as inputs for our interactive autoregressive Goal-Directed Planner (AR-GDP), constituting the second stage of our method that generates roll-outs for simulations. Evaluation results on the leaderboard of the Waymo Open Sim Agents Challenge (WOSAC) 2023 show the competitiveness of the proposed method.

In environments with delayed observation, state augmentation by including actions within the delay window is adopted to retrieve Markovian property to enable reinforcement learning (RL). Whereas, state-of-the-art (SOTA) RL techniques with Temporal-Difference (TD) learning frameworks commonly suffer from learning inefficiency, due to the significant expansion of the augmented state space with the delay. To improve the learning efficiency without sacrificing performance, this work novelly introduces Variational Delayed Policy Optimization (VDPO), reforming delayed RL as a variational inference problem. This problem is further modelled as a two-step iterative optimization problem, where the first step is TD learning in the delay-free environment with a small state space, and the second step is behaviour cloning which can be addressed much more efficiently than TD learning. We not only provide a theoretical analysis of VDPO in terms of sample complexity and performance, but also empirically demonstrate that VDPO can achieve consistent performance with SOTA methods, with a significant enhancement of sample efficiency (approximately 50\% less amount of samples) in the MuJoCo benchmark.

Ensuring safety and achieving human-level driving performance remain challenges for autonomous vehicles, especially in safety-critical situations. As a key component of artificial intelligence, reinforcement learning is promising and has shown great potential in many complex tasks; however, its lack of safety guarantees limits its real-world applicability. Hence, further advancing reinforcement learning, especially from the safety perspective, is of great importance for autonomous driving. As revealed by cognitive neuroscientists, the amygdala of the brain can elicit defensive responses against threats or hazards, which is crucial for survival in and adaptation to risky environments. Drawing inspiration from this scientific discovery, we present a fear-neuro-inspired reinforcement learning framework to realize safe autonomous driving through modeling the amygdala functionality. This new technique facilitates an agent to learn defensive behaviors and achieve safe decision making with fewer safety violations. Through experimental tests, we show that the proposed approach enables the autonomous driving agent to attain state-of-the-art performance compared to the baseline agents and perform comparably to 30 certified human drivers, across various safety-critical scenarios. The results demonstrate the feasibility and effectiveness of our framework while also shedding light on the crucial role of simulating the amygdala function in the application of reinforcement learning to safety-critical autonomous driving domains.

Accurate and robust risk prediction on curved roads can significantly reduce lane departure accidents and improve traffic safety. However, limited study has considered dynamic driver-related factors in risk prediction, resulting in poor algorithm adaptiveness to individual differences. This article presents a novel personalized risk prediction method with consideration of driver turning behavior and workload by using the predicted vehicle trajectory.First, driving simulation experiments are conducted to collect synchronized trajectory data, vehicle dynamic data, and eye movement data. The drivers are distracted by answering questions via a Bluetooth headset, leading to an increased cognitive workload. Secondly, thek-means clustering algorithm is utilized to extract two turning behaviors: driving toward the inner and outer side of a curved road. The turning behavior of each trajectory is then recognized using the trajectory data. In addition, the driver workload is recognized using the vehicle dynamic features and eye movement features. Thirdly, an extra personalization index is introduced to a long short-term memory encoder–decoder trajectory prediction network. This index integrates the driver turning behavior and workload information. After introducing the personalization index, the root-mean-square errors of the proposed network are reduced by 15.6%, 23.5%, and 29.1% with prediction horizons of 2, 3, and 4 s, respectively. Fourthly, the risk potential field theory is employed for risk prediction using the predicted trajectory data. This approach implicitly incorporates the driver's personalized information into risk prediction.

Human–robot hybrid disassembly cells provide a shared workspace that synergizes the strength of both humans and robots. These cells are characterized by adaptability and reconfigurability to accommodate the frequent changes stemming from diverse products, championing the mass personalization paradigm in Industry 5.0. Disassembly sequence planning assumes paramount importance within hybrid cells but proves to be a formidable challenge. Conventional methods prioritize the fulfillment of product-related constraints while neglecting the ergonomics considerations of operators. This oversight runs counter to the human-centric ethos central to Industry 5.0. This article proposes a personalized disassembly sequence planning approach for a human–robot hybrid disassembly cell. It presents a biobjective disassembly sequence planning model that concurrently addresses sequence scheduling and task allocation. Personal ergonomics are automatically assessed by analyzing joint angles within the operator's body skeleton. To yield disassembly plans that optimize both benefit and efficiency, a hybrid multiobjective ant lion optimizer is proposed featuring improved encoding/decoding mechanisms, updating strategies, and constraint satisfaction strategies. It adeptly addresses the discrete nature of disassembly sequences and the binary attributes associated with task execution and assignment. Personalized disassembly experiments are carried out to illustrate the feasibility and practicability of the proposed approach.

This article explores the challenge of achieving scalable and constrained consensus in general linear multiagent systems (MASs), where agents can occasionally join and leave the network. Two distributed model predictive control (DMPC)-based consensus methods are developed to tackle the scalability, performance, and constraint challenges. The first approach uses an innovative online DMPC optimization that integrates with a predesigned scalable consensus protocol, ensuring constraint satisfaction while achieving scalable consensus. The second method leverages tracking DMPC, enabling each agent to adhere to a locally evolving time-specific reference, which is continually updated through the utilization of the predicted state sequences from neighboring agents. Moreover, it is shown that the feasibility of the associated optimization problems can be recursively ensured with the suitably designed cost function and constraints. In addition, the scalable consensus property of the constrained MAS is guaranteed. Finally, the simulation results illustrate the effectiveness of the proposed algorithms.

Precise path tracking and agilely avoiding obstacles are essential for the stability and safety of autonomous driving. In this paper, we introduce a uniform safe path tracking control strategy that combines obstacle avoidance with path tracking via a barrier function. Unlike the conventional hierarchical collision avoidance methods, our approach employs an integral heuristic barrier function that addresses obstacle avoidance planning and reference trajectory tracking problems simultaneously. Via this, the complex safe trajectory following problem is simplified into a tractable yaw angle tracking problem. We then present a novel finite-time adaptive barrier function-based sliding mode controller that handles input saturation and enhances robustness. This ensures precise and robust yaw angle tracking within specified performance constraints. Moreover, the proposed approach achieves accelerated finite-time convergence compared to the exponential convergence rate. Finally, the Carsim-Simulink co-simulations and real-vehicle experiments validate the effectiveness and superiority of our method in addressing the path-tracking challenge, while upholding driving safety.

Accurate vision-based driver attention estimation is a challenging task due to the limitations of the visual sensor, and it is a critical and fundamental function of building a human-centered intelligent driving system. Unlike previous investigations which consider it a classification task, this study newly introduces scenario contextual information to improve the accuracy and obtain a fine-grained estimation. Therefore, a data-driven hybrid architecture for context-aware driver attention estimation is proposed to jointly model the scene and state of the driver during driving. A visual saliency map is typically assumed to highlight a distinct area that can capture human attention. To leverage this characteristic, a multi-hierarchy fusion network is proposed to extract effectively saliency features of a scene image. A gaze-tracking network is employed to estimate the potential focus zone of the driver, and this coarse estimation is optimized subsequently using the extracted saliency information to obtain a fine-grained estimation. Three related and commonly used task-agnostic and task-driven datasets are adopted to evaluate the proposed saliency estimation model, and experimental results show that it can achieve state-of-the-art performance. To verify the joint modeling methodology, two new driving attention datasets supplemented with driver information are collected based on the existing ones. The results of comparative experiments indicate that the consideration of saliency features can significantly improve the estimation performance of gaze fixation, demonstrating the feasibility and efficiency of the proposed method.

The learning from intervention (LfI) approach has been proven effective in improving the performance of RL algorithms; nevertheless, existing methodologies in this domain tend to operate under the assumption that human guidance is invariably devoid of risk, thereby possibly leading to oscillations or even divergence in RL training as a result of improper demonstrations. In this paper, we propose a safety-aware human-in-the-loop reinforcement learning (SafeHIL-RL) approach to bridge the abovementioned gap. We first present a safety assessment module based on the artificial potential field (APF) model that incorporates dynamic information of the environment under the Frenet coordinate system, which we call the Frenet-based dynamic potential field (FDPF), for evaluating the real-time safety throughout the intervention process. Subsequently, we propose a curriculum guidance mechanism inspired by the pedagogical principle of whole-to-part patterns in human education. The curriculum guidance facilitates the RL agent’s early acquisition of comprehensive global information through continual guidance while also allowing for fine-tuning local behavior through intermittent human guidance through a human-AI shared control strategy. Consequently, our approach enables a safe, robust, and efficient reinforcement learning process independent of the quality of guidance human participants provide. The proposed method is validated in two highway autonomous driving scenarios under highly dynamic traffic flows (https://github.com/OscarHuangWind/Safe-Human-in-the-Loop-RL). The experiments’ results confirm the superiority and generalization capability of our approach when compared to other state-of-the-art (SOTA) baselines, as well as the effectiveness of the curriculum guidance.

Despite some successful applications of goal-driven navigation, existing deep reinforcement learning (DRL)-based approaches notoriously suffers from poor data efficiency issue. One of the reasons is that the goal information is decoupled from the perception module and directly introduced as a condition of decision-making, resulting in the goal-irrelevant features of the scene representation playing an adversary role during the learning process. In light of this, we present a novel Goal-guided Transformer-enabled reinforcement learning (GTRL) approach by considering the physical goal states as an input of the scene encoder for guiding the scene representation to couple with the goal information and realizing efficient autonomous navigation. More specifically, we propose a novel variant of the Vision Transformer as the backbone of the perception system, namely Goal-guided Transformer (GoT), and pre-train it with expert priors to boost the data efficiency. Subsequently, a reinforcement learning algorithm is instantiated for the decision-making system, taking the goal-oriented scene representation from the GoT as the input and generating decision commands. As a result, our approach motivates the scene representation to concentrate mainly on goal-relevant features, which substantially enhances the data efficiency of the DRL learning process, leading to superior navigation performance. Both simulation and real-world experimental results manifest the superiority of our approach in terms of data efficiency, performance, robustness, and sim-to-real generalization, compared with other state-of-the-art (SOTA) baselines. The demonstration video (https://www.youtube.com/watch?v=aqJCHcsj4w0) and the source code (https://github.com/OscarHuangWind/DRL-Transformer-SimtoReal-Navigation) are also provided.

Autonomous vehicles necessitate prediction of future motions of surrounding traffic participants for safe navigation. However, prediction is challenging due to complex road structures and the multimodality of driving behaviors. Recent approaches usually output a fixed number of predicted trajectories, thus can hardly generalize to situations with more options and match modalities in different situations. They also rely on complex loss design and time-consuming training. This work proposes a novel map-adaptive multimodal trajectory predictor that links driving modalities, driver’s intentions, and a vehicle’s candidate centerlines (CCLs) together, rendering the predictor map-adaptive and the multimodality explainable. The predictor is derived by training an intention-aware unimodal trajectory predictor, which consists of aCCL-based goal predictorand agoal-directed trajectory completer, and aCCL scorerfor estimating the possibilities of a target vehicle (TV) choosing a CCL to follow. This decomposed approach simplifies the training process and reduces the computational resources required, thereby rendering it a faster and more cost-effective alternative. Additionally, the proposed predictor has demonstrated comparable or even superior performance to traditional multimodal predictors in specific applications. Overall, the unimodal predictor presents a promising approach for practical machine learning applications, particularly when computational resources are limited.

This paper presents an adaptive leading cruise control strategy for the automated vehicle (AV) and first considers its impact on the following human-driven vehicle (HDV) with diverse driving characteristics in the unified optimization framework for improved holistic energy efficiency. The car-following behaviors of HDV are statistically calibrated using the Next Generation Simulation dataset. In a typical single-lane car-following scenario where AVs and HDVs share the road, the longitudinal speed control of AVs can substantially reduce the energy consumption of the following HDV by avoiding unnecessary acceleration and braking. Moreover, apart from the objectives including car-following safety and traffic efficiency, the energy efficiencies of both AV and HDV are incorporated into the reward function of reinforcement learning (RL). The specific driving pattern of the following HDV is learned in real-time from historical speed information to predict its acceleration and power consumption in the optimization horizon. A comprehensive simulation is conducted to statistically verify the positive impacts of AV on the holistic energy efficiency of the mixed traffic flow with uncertain and diverse human driving behaviors. In freeway driving scenarios, simulation results indicate that the holistic energy efficiency is improved by an average of 6.03% and 6.41% compared to the reference control algorithms, specifically, RL without HDV consideration and model predictive control. These improvements highlight the significance of our approach in optimizing energy efficiency for mixed traffic on freeways.

This study addresses the challenges and solutions for achieving flexible and resilient platooning in Intelligent and Connected Vehicles (ICVs) under diverse constraints. We focus on enabling vehicles to freely join or leave the platoon and maintaining resilience against adversarial cyberattacks within the network. We propose a hierarchical distributed coordination framework that combines high-level event-driven cluster coordination with lower-level decoupled longitudinal and lateral control designs. Each normal vehicle updates its longitudinal state by solving a distributed optimization-based control problem, utilizing both itself and neighboring vehicles’ information. Meanwhile, the lateral control scheme employs a decentralized optimization algorithm to facilitate lane-changing coordination. Additionally, we develop a distributed attack detection algorithm that enables the identification and removal of adversarial vehicles from the platoon. The stability of the closed-loop system is proven, and simulation results validate the effectiveness of our framework in achieving flexible and resilient vehicle platooning.

Cooperative adaptive cruise control (CACC) is one of the most studied platooning algorithms for connected vehicles. Despite its popularity, available studies neglect an important practical aspect of CACC: because the control input is a desired acceleration, existing CACC algorithms require an acceleration-hold loop, fed by accelerometer measurements that are noisy in practice. This work proposes new classes of CACC strategies that, while avoiding any feedback from the accelerometer, guarantee the same properties of existing designs. Theoretical properties are proven in terms of stability and string stability. Numerical tests, also performed in the open-source CARLA platooning toolbox named OpenCDA (cooperative driving automation), are provided to validate the theoretical properties of the design and the improved performance against noisy measurements.

Due to the use of inter-vehicle wireless communication, vehicular platooning can be prone to attacks with corrupted data, as in false data injection (FDI) attacks. It is crucial to develop platooning protocols promoting resilience to injected false data. In this work we show that resilience can be attained by making use of a system-theoretic property known as disturbance decoupling. We first show how disturbance decoupling is obtained in nominal platooning protocols without attacks: then, in the presence of FDI attacks, we propose compensation strategies that guarantee to recover the nominal performance of the platoon. The proposed compensation strategies can cope with platoons of heterogeneous vehicles and are designed towards string stability. Numerical experiments, also performed in a SUMO-Veins co-simulation environment with different platooning scenarios under FDI attacks, validate the effectiveness of the proposed protocols in handling cyber-attacks and platoon heterogeneity.

Accurate recognition of driver distraction is significant for the design of human-machine cooperation driving systems. Existing studies mainly focus on classifying varied distracted driving behaviors, which depend heavily on the scale and quality of datasets and only detect the discrete distraction categories. Therefore, most data-driven approaches have limited capability of recognizing unseen driving activities and cannot provide a reasonable solution for downstream applications. To address these challenges, this paper develops a vision Transformer-enabled weakly supervised contrastive (W-SupCon) learning framework, in which distracted behaviors are quantified by calculating their distances from the normal driving representation set. The Gaussian mixed model (GMM) is employed for the representation clustering, which centralizes the distribution of the normal driving representation set to better identify distracted behaviors. A novel driver behavior dataset and the other three ones are employed for the evaluation, experimental results demonstrate that our proposed approach has more accurate and robust performance than existing methods in the recognition of unknown driver activities. Furthermore, the rationality of distraction levels for different driving behaviors is evaluated through driver skeleton poses. The constructed dataset and demo videos are available athttps://yanghh.io/Driver-Distraction-Quantification.

Driver drowsiness detection is of great significance in improving driving safety and has been widely studied in recent years. However, some existing methods have not fully utilized the drowsiness-related information, and some methods are susceptible to interference from the redundant information of input data. To address these issues, a video-based driver drowsiness detection method according to the key facial features including facial landmarks and local facial areas (VBFLLFA) is proposed in this paper. In order to fully utilize the key facial features related to drowsiness and exclude the interference of redundant information, the head movement information is obtained through facial landmark analysis and the movement information of eyes and mouth is acquired from the local facial areas. And the spatial filtering based on the common spatial pattern (CSP) algorithm is introduced to improve the discrimination of different classes of samples. To adequately extract the temporal and spatial features, a two-branch multi-head attention (TB-MHA) module is designed in this paper. Furthermore, the center loss with center vector distance penalty is introduced to further improve the discrimination of different classes of samples in the feature space. In addition to two public datasets, we specifically create a novel video-based driver drowsiness detection (VBDDD) dataset to evaluate the effectiveness of our method. The experimental results verify that our method can achieve very excellent performance in driver drowsiness detection tasks.

Interacting with the surrounding road users is crucial for autonomous vehicles (AV). However, the inherent multimodality and uncertainties associated with traffic participants (TP) pose challenges in AVs’ prediction and decision-making (PnD). A primary challenge is adapting predictors trained on static offline datasets to the dynamic, diverse data streams encountered in reality. Secondly, utilizing one single forecast trajectory with the highest probability for decision-making contains potential risks as it neglects that even a small probability represents a subset of TP behaviors. Based on the existing prediction backbone, we propose an online learning approach incorporating pseudo-labels inferred from partial feedback as compensation for conventional methodologies, considering both the commonsense and personalization facets of driving. Drawing inspiration from the second law of thermodynamics, we propose to minimize microscopic traffic entropy as an additional objective in decision-making. This objective aims to reduce the chaos of traffic scenes, thus achieving more predictable future interactions and, conversely, making future decisions easier. Through real-time human-in-the-loop experiments, we quantifiably and comparably reveal that adopting one single trajectory without online learning in PnD is risky. However, this reliability is verified to be significantly improved by our proposed techniques, and the efficacy is further analyzed in a subsequent qualitative study. A static experiment transferring the prediction algorithm trained exclusively on Argoverse 2 to datasets including NGSIM, HighD, RounD, and NuScenes is also conducted, demonstrating that the proposed correction can effectively mitigate the gap between the datasets and real-world scenarios.

Conditional Automated Driving (CAD) has attracted widespread attention due to the substantial gap in achieving fully autonomous driving, wherein an essential endeavor entails determining the transition timing between automated and manual driving modes. Driver cognitive workload serves as a crucial indicator for identifying transition timing, while its precise determination is challenging with discrete workload levels in previous studies. To address this issue, this work develops a dual-stage learning framework to quantify driver cognitive workload continuously. Specifically, a semi-supervised co-training strategy is first designed to approximate workload values, and then supervised contrastive learning is employed to align them with their feature representations in the latent space. A novel driver workload dataset is constructed for the evaluation, and experimental results demonstrate that our proposed approach outperforms other state-of-the-art baselines in estimation accuracy. Furthermore, the rationality of quantified cognitive workload is analyzed through the driver’ subjective assessment, indicating it is a more reliable solution for achieving the driving authority transition.

Predicting the future states of surrounding traffic participants and planning a safe, smooth, and socially compliant trajectory accordingly are crucial for autonomous vehicles (AVs). There are two major issues with the current autonomous driving system: the prediction module is often separated from the planning module, and the cost function for planning is hard to specify and tune. To tackle these issues, we propose a differentiable integrated prediction and planning (DIPP) framework that can also learn the cost function from data. Specifically, our framework uses a differentiable nonlinear optimizer as the motion planner, which takes as input the predicted trajectories of surrounding agents given by the neural network and optimizes the trajectory for the AV, enabling all operations to be differentiable, including the cost function weights. The proposed framework is trained on a large-scale real-world driving dataset to imitate human driving trajectories in the entire driving scene and validated in both open-loop and closed-loop manners. The open-loop testing results reveal that the proposed method outperforms the baseline methods across a variety of metrics and delivers planning-centric prediction results, allowing the planning module to output trajectories close to those of human drivers. In closed-loop testing, the proposed method outperforms various baseline methods, showing the ability to handle complex urban driving scenarios and robustness against the distributional shift. Importantly, we find that joint training of planning and prediction modules achieves better performance than planning with a separate trained prediction module in both open-loop and closed-loop tests. Moreover, the ablation study indicates that the learnable components in the framework are essential to ensure planning stability and performance. Code and Supplementary Videos are available at https://mczhi.github.io/DIPP/.

Stochastic exploration is the key to the success of the deep -network (DQN) algorithm. However, most existing stochastic exploration approaches either explore actions heuristically regardless of their values or couple the sampling with values, which inevitably introduce bias into the learning process. In this article, we propose a novel preference-guided -greedy exploration algorithm that can efficiently facilitate exploration for DQN without introducing additional bias. Specifically, we design a dual architecture consisting of two branches, one of which is a copy of DQN, namely, the branch. The other branch, which we call the preference branch, learns the action preference that the DQN implicitly follows. We theoretically prove that the policy improvement theorem holds for the preference-guided -greedy policy and experimentally show that the inferred action preference distribution aligns with the landscape of corresponding values. Intuitively, the preference-guided -greedy exploration motivates the DQN agent to take diverse actions, so that actions with larger values can be sampled more frequently, and those with smaller values still have a chance to be explored, thus encouraging the exploration. We comprehensively evaluate the proposed method by benchmarking it with well-known DQN variants in nine different environments. Extensive results confirm the superiority of our proposed method in terms of performance and convergence speed.

Reinforcement learning (RL) requires skillful definition and remarkable computational efforts to solve optimization and control problems, which could impair its prospect. Introducing human guidance into RL is a promising way to improve learning performance. In this article, a comprehensive human guidance-based RL framework is established. A novel prioritized experience replay mechanism that adapts to human guidance in the RL process is proposed to boost the efficiency and performance of the RL algorithm. To relieve the heavy workload on human participants, a behavior model is established based on an incremental online learning method to mimic human actions. We design two challenging autonomous driving tasks for evaluating the proposed algorithm. Experiments are conducted to access the training and testing performance and learning mechanism of the proposed algorithm. Comparative results against the state-of-the-art methods suggest the advantages of our algorithm in terms of learning efficiency, performance, and robustness.

Although deep reinforcement learning (DRL) methods are promising for making behavioral decisions in autonomous vehicles (AVs), their low training efficiency and difficulty to adapt to untrained cases hinder their applications. Introducing a human role in the DRL paradigm could improve training efficiency by using human prior knowledge and overcome untrained cases in deployment by online human takeover. In this study, a novel value-based DRL algorithm that leverages human guidance to improve its performance is proposed for addressing high-level decision-making problems in autonomous driving. We develop a new learning objective for DRL to increase the value of the human policy over the undertrained DRL policy so that the DRL agent can be encouraged to mimic human behaviors and thereby utilizing human guidance more efficiently. Our method can autonomously evaluate the importance of different human guidance, which makes it more robust for variation of human performance. The proposed DRL algorithm was used to address a challenging multiobjective lane-change decision-making problem. We collected human guidance from a human-in-the-loop driving experiment and evaluated our method in a high-fidelity simulator. Results validated the advantages of the proposed algorithm in terms of training efficiency and optimality in the decision-making problem compared to the baselines of state-of-the-art existing methods. Results also revealed the favorable fine-tuning ability of the proposed algorithm, which is promising for addressing the long-tail issue in DRL-based autonomous driving. Our methodology does not introduce additional domain knowledge so that it can be seamlessly applied to other similar issues. The supplementary video is available at https://youtu.be/Ec7WkqeLsB8.

Many intelligent robots use a combination of radar and camera sensors to capture environmental information. Robust and accurate perception highly relies on the result of multi-sensor calibration. Most current spatial calibration methods require a calibration board or a special marker as the target. In this paper, we provide a novel calibration method for RGBD camera and millimeter-wave radar, which automatically estimates the extrinsic parameters. Our proposed method includes the following two stages: rough extrinsic parameters are estimated by using object contours as geometric constraints, and meanwhile, the optimum is reached via optimizing based on the difference of velocity obtained from camera and radar. It only needs an object moving past sensors, but does not require for a calibration board. We validate our method through simulation experiments and real-world experiments. We construct a simulation environment in CARLA to verify the performance of our proposed method against different angles. Furthermore, different levels of zero mean Gaussian noise are added to evaluate the stability of our method. In addition, real-world experiments with different hardware setups are taken to verify the feasibility of our method in real-world conditions.

Reinforcement learning (RL) is a promising approach in unmanned ground vehicles (UGVs) applications, but limited computing resource makes it challenging to deploy a well-behaved RL strategy with sophisticated neural networks. Meanwhile, the training of RL on navigation tasks is difficult, which requires a carefully-designed reward function and a large number of interactions, yet RL navigation can still fail due to many corner cases. This shows the limited intelligence of current RL methods, thereby prompting us to rethink combining RL with human intelligence. In this paper, a human-guided RL framework is proposed to improve RL performance both during learning in the simulator and deployment in the real world. The framework allows humans to intervene in RL's control progress and provide demonstrations as needed, thereby improving RL's capabilities. An innovative human-guided RL algorithm is proposed that utilizes a series of mechanisms to improve the effectiveness of human guidance, including human-guided learning objective, prioritized human experience replay, and human intervention-based reward shaping. Our RL method is trained in simulation and then transferred to the real world, and we develop a denoised representation for domain adaptation to mitigate the simulation-to-real gap. Our method is validated through simulations and real-world experiments to navigate UGVs in diverse and dynamic environments based only on tiny neural networks and image inputs. Our method performs better in goal-reaching and safety than existing learning- and model-based navigation approaches and is robust to changes in input features and ego kinetics. Furthermore, our method allows small-scale human demonstrations to be used to improve the trained RL agent and learn expected behaviors online.

The specific restrictions are breached in the case of vehicle platoon faults and result in unacceptable system performance degradation. This paper proposed a novel prescribed time performance recovery fault tolerant control method to ensure nominal platoon performance under multiple faults, including actuator faults with deferred backup actuator switching and leader-follower link faults in consideration. A novel barrier function based prescribed time sliding mode controller is devised to assure platoon consensus errors and convergence time within prescribed constraints under normal conditions at first. Under multiple faults conditions, to tackle with leader-follower link faults problem, a novel distributed recursive estimator is proposed to estimate the leader’s states and recover the previous leader-follower platooning control protocol in a prescribed time. Besides, in the presence of actuator failures, the nominal constraints violated problem under faults is put into consideration. Owing to the unavoidable deferred actuator replacement time, the previous platoon consensus error constraints are violated and cause platoon performance degradation. Under such circumstances, by exploiting one novel barrier function-based sliding mode controller with an error shifting function, the unfavorable exceeding platoon consensus errors can be recovered into the nominal constraints domains within a prescribed time. Numerical simulations and hardware-in-loop (HIL) experiments are demonstrated to validate the effectiveness and superiority of our performance recovery fault tolerant control algorithms.

In this paper, a human-like driving system is designed for autonomous vehicles (AVs), which aims to make AVs better integrate into the human transportation systems and mitigate misunderstanding and conflicts when interacting with human-driven vehicles. Based on the analysis of the real world INTERACTION dataset, a driving aggressiveness estimation model is established with the fuzzy inference approach. In the human-like lane-change decision-making algorithm, the cost function is designed comprehensively considering driving safety and travel efficiency. Based on the cost function with multi-constraint, a dynamic game algorithm is developed to model the interactions and decision making between AV and human-driven vehicles. Additionally, to guarantee the safety during lane-change of AVs, an artificial potential field model is built for collision risk assessment. Further, a human-like driving model is designed, which integrates the brain emotional learning circuit model (BELCM) with a two-point preview model. Finally, the proposed algorithm is evaluated through human-in-the-loop experiments, and the results demonstrated the feasibility and effectiveness of the proposed method.

Reinforcement learning has demonstrated its potential in a series of challenging domains. However, many real-world decision making tasks involve unpredictable environmental changes or unavoidable perception errors that are often enough to mislead an agent into making suboptimal decisions and even cause catastrophic failures. In light of these potential risks, reinforcement learning with application in safety-critical autonomous driving domain remains tricky without ensuring robustness against environmental uncertainties (e.g., road adhesion changes or measurement noises). Therefore, this paper proposes a novel constrained adversarial reinforcement learning approach for robust decision making of autonomous vehicles at highway on-ramps. Environmental disturbance is modelled as an adversarial agent that can learn an optimal adversarial policy to thwart the autonomous driving agent. Meanwhile, observation perturbation is approximated to maximize the variation of the perturbed policy through a white-box adversarial attack technique. Furthermore, a constrained adversarial actor-critic algorithm is presented to optimize an on-ramp merging policy while keeping the variations of the attacked driving policy and action-value function within bounds. Finally, the proposed robust highway on-ramp merging decision making method of autonomous vehicles is evaluated in three stochastic mixed traffic flows with different densities, and its effectiveness is demonstrated in comparison with the competitive baselines.

Making safe and human-like decisions is an essential capability of autonomous driving systems, and learning-based behavior planning presents a promising pathway toward achieving this objective. Distinguished from existing learning-based methods that directly output decisions, this work introduces a predictive behavior planning framework that learns to predict and evaluate from human driving data. This framework consists of three components: a behavior generation module that produces a diverse set of candidate behaviors in the form of trajectory proposals, a conditional motion prediction network that predicts future trajectories of other agents based on each proposal, and a scoring module that evaluates the candidate plans using maximum entropy inverse reinforcement learning (IRL). We validate the proposed framework on a large-scale real-world urban driving dataset through comprehensive experiments. The results show that the conditional prediction model can predict distinct and reasonable future trajectories given different trajectory proposals and the IRL-based scoring module can select plans that are close to human driving. The proposed framework outperforms other baseline methods in terms of similarity to human driving trajectories. Additionally, we find that the conditional prediction model improves both prediction and planning performance compared to the non-conditional model. Lastly, we note that the learning of the scoring module is crucial for aligning the evaluations with human drivers.

Reliable path tracking control (PTC) method is essential for autonomous driving. However, existing PTC methods count on prior vehicle parameters to achieve good performance. This paper presents an optimal PTC method without requiring any prior vehicle parameters based on online modeling with strict parameter convergence ability. First, we build a virtual optimal control problem using adaptive dynamic programming (ADP) scheme to guide the data collection and solve two characteristic matrices containing parameter information. Then, the model construction method is derived using the solved matrices and the optimal PTC method is constructed using the constructed model. Finally, a fault-tolerant control scheme is further designed using the constructed model and the online modeling ability of the proposed method. The effectiveness of the proposed method is validated through co-simulation between Matlab/Simulink and high-fidelity vehicle dynamic simulation software CarSim® under both fault-free and fault-tolerant situations.

Accurate trajectory prediction is essential for safe and efficient autonomous driving in complex traffic environments. While artificial intelligence has shown great promise in improving prediction accuracy, its inherent uncertainty and lack of explainability may lead to unpredictable failures, creating challenges for safety-critical decision-making. This study aims to address these challenges by exploring the impact of traffic environment on prediction algorithms. The study proposes a trajectory prediction framework with epistemic uncertainty estimation ability to output high uncertainty when facing unforeseeable or unknown scenarios. The framework analyzes the environmental effect on the trajectory prediction by considering scenario features and shifts. Features are divided into kinematic features of a target agent, features of surrounding traffic participants, and other scenario features. Feature correlation and importance analyses are performed to study their influence on prediction error and epistemic uncertainty. The impact of unavoidable distributional shifts in the real world on trajectory predictions is investigated using multiple intersection datasets. The results indicate that deep ensemble-based methods have advantages in improving robustness while estimating epistemic uncertainty. Consistent conclusions were obtained from the correlation and importance analyses, indicating that kinematic features of the target agent have relatively strong effects on both prediction error and epistemic uncertainty. Finally, the study analyzes the accuracy deterioration caused by distributional shifts and the potential of the deep ensemble-based method. Through deep ensemble, the errors of the prediction methods based on GRIP++ and Trajectron++ have been improved by 6.4% and 10.8% in the same-dataset test, and 6.3% and 10.8% in the cross-dataset test.

Cooperative Adaptive Cruise Control (CACC) is an autonomous vehicle-following technology that allows groups of vehicles on the highway to form in tightly-coupled platoons. This is accomplished by exchanging inter-vehicle data through Vehicle-to-Vehicle (V2V) wireless communication networks. CACC increases traffic throughput and safety, and decreases energy consumption. However, the surge of vehicle connectivity has brought new security challenges as vehicular networks increasingly serve as new access points for adversaries trying to deteriorate the platooning performance or even cause collisions. In this manuscript, we propose a novel anomaly detection scheme that leverage real-time sensor/network data and physics-based mathematical models of vehicles in the platoon. Nevertheless, even the best detection scheme could lead to conservative detection results because of unavoidable modelling uncertainties, network effects (delays, quantization, communication dropouts), and noise. It is hard (often impossible) for any detector to distinguish between these different perturbation sources and actual attack signals. This enables adversaries to launch a range of attack strategies that can surpass the detection scheme by hiding within the system uncertainty. Here, we provide risk assessment tools (in terms of semi-definite programs) for Connected and Automated Vehicles (CAVs) to quantify the potential effect of attacks that remain hidden from the detector (referred here as stealthy attacks). A numerical case-study is presented to illustrate the effectiveness of our methods.

Since emotions generally affect driver’s behavior, judgment, and reaction time, accurately identifying driver’s emotions is of great significance to improve the safety and comfort of intelligent driving system. However, the gender, skin color, age, and appearance of different drivers often have big differences, which will greatly interfere with the emotional recognition process. Besides, light intensity inside the vehicle varies with different time, weather, and location, which will also pose a challenge to driver emotion recognition. In this paper, a robust driver emotion recognition method based on feature separation is proposed to overcome the interference of individual differences and illumination changes. In order to realize the separation of expression-related features and irrelevant features, we design a high-purity feature separation (HPFS) framework based on partial feature exchange and the constraints of multiple loss functions. To verify that the proposed method can overcome the interference of illumination changes, we specifically create a multiple light intensities driver emotion recognition (MLI-DER) dataset and conduct a great deal of experiments on the dataset. In addition, to further demonstrate that our method can largely alleviate the interference of individual difference, some cross-subject emotion recognition experiments are conducted on two famous facial expression recognition datasets FACES and Oulu-CASIA and the experimental results are compared with that of some state-of-the-art methods.

Deep reinforcement learning (DRL) is a promising way to achieve human-like autonomous driving. However, the low sample efficiency and difficulty of designing reward functions for DRL would hinder its applications in practice. In light of this, this article proposes a novel framework to incorporate human prior knowledge in DRL, in order to improve the sample efficiency and save the effort of designing sophisticated reward functions. Our framework consists of three ingredients, namely, expert demonstration, policy derivation, and RL. In the expert demonstration step, a human expert demonstrates their execution of the task, and their behaviors are stored as state-action pairs. In the policy derivation step, the imitative expert policy is derived using behavioral cloning and uncertainty estimation relying on the demonstration data. In the RL step, the imitative expert policy is utilized to guide the learning of the DRL agent by regularizing the KL divergence between the DRL agent's policy and the imitative expert policy. To validate the proposed method in autonomous driving applications, two simulated urban driving scenarios (unprotected left turn and roundabout) are designed. The strengths of our proposed method are manifested by the training results as our method can not only achieve the best performance but also significantly improve the sample efficiency in comparison with the baseline algorithms (particularly 60% improvement compared with soft actor-critic). In testing conditions, the agent trained by our method obtains the highest success rate and shows diverse and human-like driving behaviors as demonstrated by the human expert. We also find that using the imitative expert policy trained with the ensemble method that estimates both policy and model uncertainties, as well as increasing the training sample size, can result in better training and testing performance, especially for more difficult tasks. As a result, the proposed method has shown its potential to facilitate the applications of DRL-enabled human-like autonomous driving systems in practice. The code and supplementary videos are also provided. [https://mczhi.github.io/Expert-Prior-RL/].

Interest in autonomous driving (AD) and intelligent vehicles (IVs) is growing at a rapid pace due to the convenience, safety, and economic benefits. Although a number of surveys have reviewed research achievements in this field, they are still limited in specific tasks and lack systematic summaries and research directions in the future. Our work is divided into three independent articles and the first part is a survey of surveys (SoS) for total technologies of AD and IVs that involves the history, summarizes the milestones, and provides the perspectives, ethics, and future research directions. This is the second part (Part I for this technical survey) to review the development of control, computing system design, communication, high-definition map (HD map), testing, and human behaviors in IVs. In addition, the third part (Part II for this technical survey) is to review the perception and planning sections. The objective of this article is to involve all the sections of AD, summarize the latest technical milestones, and guide abecedarians to quickly understand the development of AD and IVs. Combining the SoS and Part II, we anticipate that this work will bring novel and diverse insights to researchers and abecedarians, and serve as a bridge between past and future.

Research on the shared control system for human-in-the-loop cooperative driving has grown steadily in the past decade. However, most proposed methodologies were focused on structural roads such as highway rather than the unstructured environment. This paper presents a novel potential hazard-aware shared steering approach for human-robot cooperative driving in unstructured environment. First, we propose a hierarchical Gaussian risk field (HGRF) to evaluate the potential hazard of the predicted path. Then an adaptive control authority allocation module is developed to engage the control in real-time. The control authority will be fully owned by the human driver when the vehicle drives in a safe manner. However, in the situation where the hazard level is predicted to be high, the authority of the human driver decrease, and the automation actively assists the driver by dynamically sharing the control authority to enhance safety and performance. The proposed methodology is experimentally verified with a steer-by-wire car-like mobile robot. The results show that our proposed approach can effectively engage and cooperatively control the vehicle in hazard cases, ensuring driving safety.

This paper presents a multi-objective strategy for trajectory tracking of intelligent electric vehicles incorporating tracking performance with the energy economy. A model predictive controller is built using the single-track vehicle dynamics model to predict planar motions. The optimization problem is formulated with a cost function involving tracking errors and motor efficiency. The proposed system is verified on a typical road in the co-simulation platform - CarSim/Simulink. Experimental results from the simulation demonstrate the expected performance of the developed system. Intelligent electric vehicles, trajectory tracking, multi-objective optimization, model predictive control

As typical applications of cyber–physical systems (CPSs), connected and automated vehicles (CAVs) are able to measure the surroundings and share local information with the other vehicles by using multimodal sensors and wireless networks. CAVs are expected to increase safety, efficiency, and capacity of our transportation systems. However, the increasing usage of sensors has also increased the vulnerability of CAVs to sensor faults and adversarial attacks. Anomalous sensor values resulting from malicious cyberattacks or faulty sensors may cause severe consequences or even fatalities. In this article, we increase the resilience of CAVs to faults and attacks by using multiple sensors for measuring the same physical variable to create redundancy. We exploit this redundancy and propose a sensor fusion algorithm for providing a robust estimate of the correct sensor information with bounded errors independent of the attack signals, and for attack detection and isolation. The proposed sensor fusion framework is applicable to a large class of security-critical CPSs. To minimize the performance degradation resulting from the usage of the estimation for control, we provide an$H_{\infty }$controller for cooperative adaptive cruise control-equipped CAVs. The designed controller is capable of stabilizing the closed-loop dynamics of each vehicle in the platoon while reducing the joint effect of estimation errors and communication channel noise on the tracking performance and string behavior of the vehicle platoon. Numerical examples are presented to illustrate the effectiveness of our methods.

Accurate trajectory prediction of surrounding vehicles enables lower risk path planning in advance for autonomous vehicles, thus promising the safety of automated driving. A low-risk and high-efficiency path planning approach is proposed for autonomous driving based on the high-performance and practical trajectory prediction method. A long short-term memory (LSTM) network is trained and tested using the highD dataset, and the validated LSTM is used to predict the trajectories of surrounding vehicles combining the information extracted from vehicle-to-vehicle (V2V) technology. A risk assessment and mitigation-based local path planning algorithm is proposed according to the information of predicted trajectories of surrounding vehicles. Two driving scenarios are extracted and reconstructed from the highD dataset for validation and evaluation, i.e., an active lane-change scenario and a longitudinal collision-avoidance scenario. The results illustrate that the risk is mitigated and the driving efficiency is improved with the proposed path planning algorithm comparing to the constant-velocity prediction and the prediction method of the nonlinear input–output (NIO) network, especially when the velocity and trajectory with sudden changes. Note to Practitioners—This article was motivated by the problem of promising the safety decision-making and path planning through accurate environment prediction. There are two main parts included in this article. First, this article proposed one pragmatic approach to predict the environment movement correctly based on the long short-term memory (LSTM) approach. The prediction performance of LSTM was compared with nonlinear input–output (NIO). The results showed that the LSTM approach has a significant advantage in motivation prediction of the surrounded vehicles during path planning. The second part of this article is to make the decision and realize local path planning based on the risk assessment. The potential field-based approach is implemented on the risk assessment based on these accurate predictions. Some primary results demonstrate that the decision-making algorithm performs better under the accurate prediction model. The results also show that the safety and driving efficiency of the ego vehicle were improved by tracking the trajectory, which was planned based on the risk assessment. The only concern for the real-time application is the computation time; in future, we will figure it out how to further reduce the computation time.

This article presents a structure constraint matrix factorization framework for different behavior segmentation of the human behavior sequential data. This framework is based on the structural information of the behavior continuity and the high similarity between neighboring frames. Due to the high similarity and high dimensionality of human behavior data, the high-precision segmentation of human behavior is hard to achieve from the perspective of application and academia. By making the behavior continuity hypothesis, first, the effective constraint regular terms are constructed. Subsequently, the clustering framework based on constrained non-negative matrix factorization is established. Finally, the segmentation result can be obtained by using the spectral clustering and graph segmentation algorithm. For illustration, the proposed framework is applied to the Weiz dataset, Keck dataset, mo_86 dataset, and mo_86_9 dataset. Empirical experiments on several public human behavior datasets demonstrate that the structure constraint matrix factorization framework can automatically segment human behavior sequences. Compared to the classical algorithm, the proposed framework can ensure consistent segmentation of sequential points within behavior actions and provide better performance in accuracy.

To address the coordination issue of connected automated vehicles (CAVs) at urban scenarios, a game-theoretic decision-making framework is proposed that can advance social benefits, including the traffic system efficiency and safety, as well as the benefits of individual users. Under the proposed decision-making framework, in this work, a representative urban driving scenario, i.e. the unsignalized intersection, is investigated. Once the vehicle enters the focused zone, it will interact with other CAVs and make collaborative decisions. To evaluate the safety risk of surrounding vehicles and reduce the complexity of the decision-making algorithm, the driving risk assessment algorithm is designed with a Gaussian potential field approach. The decision-making cost function is constructed by considering the driving safety and passing efficiency of CAVs. Additionally, decision-making constraints are designed and include safety, comfort, efficiency, control and stability. Based on the cost function and constraints, the fuzzy coalitional game approach is applied to the decision-making issue of CAVs at unsignalized intersections. Two types of fuzzy coalitions are constructed that reflect both individual and social benefits. The benefit allocation in the two types of fuzzy coalitions is associated with the driving aggressiveness of CAVs. Finally, the effectiveness and feasibility of the proposed decision-making framework are verified with three test cases.

To address the safety and efficiency issues of vehicles at multi-lane merging zones, a cooperative decision-making framework is designed for connected automated vehicles (CAVs) using a coalitional game approach. Firstly, a motion prediction module is established based on the simplified single-track vehicle model for enhancing the accuracy and reliability of the decision-making algorithm. Then, the cost function and constraints of the decision making are designed considering multiple performance indexes, i.e. the safety, comfort and efficiency. Besides, in order to realize human-like and personalized smart mobility, different driving characteristics are considered and embedded in the modeling process. Furthermore, four typical coalition models are defined for CAVS at the scenario of a multi-lane merging zone. Then, the coalitional game approach is formulated with model predictive control (MPC) to deal with decision making of CAVs at the defined scenario. Finally, testings are carried out in two cases considering different driving characteristics to evaluate the performance of the developed approach. The testing results show that the proposed coalitional game based method is able to make reasonable decisions and adapt to different driving characteristics for CAVs at the multi-lane merging zone. It guarantees the safety and efficiency of CAVs at the complex dynamic traffic condition, and simultaneously accommodates the objectives of individual vehicles, demonstrating the feasibility and effectiveness of the proposed approach.

Driver attention modeling is a crucial technique in building human-centric intelligent driving systems. Considering the human visual mechanism, this study leverages multi-level visual content, including low-level texture features, middle-level optical flows, and high-level semantic information, as the model input. Subsequently, a heterogeneous model is proposed to handle the multi-level input, which integrates the graph and convolutional neural networks. Distinguished from the existing studies that use semantic segmentation, our study directly leverages the objection detection information in an interpretable manner. To deal with the detected objects, in this work, a graph attention network is used to explicitly construct the semantic information, rather than handle the features extracted by convolutional modules for building the latent space features, which are used in existing studies. Further, a semantic attention module is proposed to integrate the non-Euclidean output of the graph network with the Euclidean feature maps of the convolutional neural networks. Finally, these integrated features are decoded to generate a driver attention map. Three typical datasets are used to validate the proposed method. A comprehensive comparison and analysis have proven the feasibility and validity of our proposed method, as well as its ability to achieve state-of-the-art performance.

This paper investigates a human-machine cooperative trajectory planning and tracking control approach for automated vehicles. The proposed method is developed based on a novel algorithm of cooperative human-machine rapidly-exploring random (HM-RRT) for path planning, together with the risk assessment of driver behavior. First, the driver’s behaviour is assessed according to the information of the predicted vehicle trajectory, the identified safe driving area and the driving risks evaluated in both lateral and longitudinal directions. Based on the driver’s expected driving task, when driving risks are identified by real-time assessment, then the human-machine cooperation is activated during trajectory planning. By HM-RRT, the newly developed safety assurance mechanism for path planning, the cooperative trajectory is then generated, which incorporates the driver’s desire and actions and automation’s corrective actions, to ensure the safety, stability and smoothness of the human-vehicle system. The simulation and experimental results show that the proposed HM-RRT algorithm can effectively improve the convergence rate and reduce the computation load, comparing to the conventional method. Beyond this, the proposed human-machine cooperation approach is able to simultaneously ensure the safety, stability and smoothness of the vehicle and largely reduce human-machine conflicts in real-time applications, demonstrating its feasibility and effectiveness.

Driving behavior modeling is of great importance for designing safe, smart, and personalized autonomous driving systems. In this paper, an internal reward function-based driving model that emulates the human’s decision-making mechanism is utilized. To infer the reward function parameters from naturalistic human driving data, we propose a structural assumption about human driving behavior that focuses on discrete latent driving intentions. It converts the continuous behavior modeling problem to a discrete setting and thus makes maximum entropy inverse reinforcement learning (IRL) tractable to learn reward functions. Specifically, a polynomial trajectory sampler is adopted to generate candidate trajectories considering high-level intentions and approximate the partition function in the maximum entropy IRL framework. An environment model considering interactive behaviors among the ego and surrounding vehicles is built to better estimate the generated trajectories. The proposed method is applied to learn personalized reward functions for individual human drivers from the NGSIM highway driving dataset. The qualitative results demonstrate that the learned reward functions are able to explicitly express the preferences of different drivers and interpret their decisions. The quantitative results reveal that the learned reward functions are robust, which is manifested by only a marginal decline in proximity to the human driving trajectories when applying the reward function in the testing conditions. For the testing performance, the personalized modeling method outperforms the general modeling approach, significantly reducing the modeling errors in human likeness (a custom metric to gauge accuracy), and these two methods deliver better results compared to other baseline methods. Moreover, it is found that predicting the response actions of surrounding vehicles and incorporating their potential decelerations caused by the ego vehicle are critical in estimating the generated trajectories, and the accuracy of personalized planning using the learned reward functions relies on the accuracy of the forecasting model.

Driver model adaptation (DMA) plays an essential role for driving behaviour modelling when there is a lack of sufficient data for training the new model. A new data-driven DMA method is proposed in this paper to realise the instance-level knowledge transfer between individual drivers. Using the importance-weighted transfer learning (IWTL), the data collected from one driver (source driver) can be directly used to train the model of another driver (target driver). Under the framework of IWTL, the relationship between two different drivers can be modelled by the importance weight (IW). Two estimation methods Kullback-Leibler (KL) Divergence and least-squares (LS), are used to estimate IW for each data instance by modelling the importance-weight function as a radial basis function (RBF). Experiments based on the driving simulator and real vehicle are carried out to test the performance of TL for steering behaviour adaptation during the overtaking manoeuvre. The experimental results show that the TL method can transfer the knowledge observed from one driver to another when training the new driver model without sufficient data by keeping the modelling error at a low level.

Simultaneous trajectory prediction for multiple heterogeneous traffic participants is essential for safe and efficient operation of connected automated vehicles under complex driving situations. Two main challenges for this task are to handle the varying number of heterogeneous target agents and jointly consider multiple factors that would affect their future motions. This is because different kinds of agents have different motion patterns, and their behaviors are jointly affected by their individual dynamics, their interactions with surrounding agents, as well as the traffic infrastructures. A trajectory prediction method handling these challenges will benefit the downstream decision-making and planning modules of autonomous vehicles. To meet these challenges, we propose a three-channel framework together with a novel Heterogeneous Edge-enhanced graph ATtention network (HEAT). Our framework is able to deal with the heterogeneity of the target agents and traffic participants involved. Specifically, agents’ dynamics are extracted from their historical states using type-specific encoders. The inter-agent interactions are represented with a directed edge-featured heterogeneous graph and processed by the designed HEAT network to extract interaction features. Besides, the map features are shared across all agents by introducing a selective gate-mechanism. And finally, the trajectories of multiple agents are predicted simultaneously. Validations using both urban and highway driving datasets show that the proposed model can realize simultaneous trajectory predictions for multiple agents under complex traffic situations, and achieve state-of-the-art performance with respect to prediction accuracy. The achieved final displacement error (FDE@3sec) is 0.66 meter under urban driving, demonstrating the feasibility and effectiveness of the proposed approach.

This paper investigates the cyber-security problem for autonomous vehicles under sensor attacks. In particular, a model-based framework is proposed which can detect sensor attacks and identify their sources in order to achieve the secure localization of self-driving vehicles. To ensure robustness of the vehicle against cyber-attacks, sensor redundancy is introduced, that is to deploy multiple sensors, each of which provides real-time pose observations of the vehicle. A bank of attack detectors is developed to capture anomalies in each sensor measurement, which is a combination of an extended Kalman filter (EKF) and a cumulative sum (CUSUM) discriminator. EKFs are employed to estimate the vehicle position and orientation recursively, while each CUSUM discriminator is designed to analyze the residual generated by its combined EKF to detect the possible deviation of the sensor measurement from the expected pose derived according to the mathematical model of the vehicle. To monitor the inconsistency amongst multiple sensor measurements, an auxiliary detector is introduced which fuses observations from multiple sensors. Based on the results of all the detectors, a rule-based isolation scheme is developed to identify the source anomalous sensor. The effectiveness of our proposed framework has been demonstrated on real vehicle data.

In this article, an improved short-term speed prediction method is proposed to predict short-term future speed and analyze future energy consumption of intelligent fuel cell vehicles. The short-term future speed is predicted by the proposed Inflated 3-D Inception long short-term memory (LSTM) network, which takes the spatiotemporal-vision information and vehicle motion states. Specifically, the spatiotemporal-vision-based deep neural network utilizes image sequences captured by a front-facing camera as environmental information and historical speed series as motion information to improve the prediction accuracy. Then, a case study of the proposed speed prediction method, with rule-based energy management strategy to calculate future energy consumption, is presented. The simulation results show that short-term speed prediction based on the Inflated 3-D Inception LSTM network can achieve high accuracy of speed prediction in various traffic densities, as well as low prediction errors of future energy consumption including the hydrogen consumption and state-of-charge attenuation.

Considering that human-driven vehicles and autonomous vehicles (AVs) will coexist on roads in the future for a long time, how to merge AVs into human drivers' traffic ecology and minimize the effect of AVs and their misfit with human drivers, are issues worthy of consideration. Moreover, different passengers have different needs for AVs, thus, how to provide personalized choices for different passengers is another issue for AVs. Therefore, a human-like decision making framework is designed for AVs in this paper. Different driving styles and social interaction characteristics are formulated for AVs regarding driving safety, ride comfort and travel efficiency, which are considered in the modeling process of decision making. Then, Nash equilibrium and Stackelberg game theory are applied to the noncooperative decision making. In addition, potential field method and model predictive control (MPC) are combined to deal with the motion prediction and planning for AVs, which provides predicted motion information for the decision-making module. Finally, two typical testing scenarios of lane change, i.e., merging and overtaking, are carried out to evaluate the feasibility and effectiveness of the proposed decision-making framework considering different human-like behaviors. Testing results indicate that both the two game theoretic approaches can provide reasonable human-like decision making for AVs. Compared with the Nash equilibrium approach, under the normal driving style, the cost value of decision making using the Stackelberg game theoretic approach is reduced by over 20%.

Connected automated driving technologies have shown tremendous improvement in recent years. However, it is still not clear how driving behaviors and energy consumption correlate with each other and to what extent these factors related to connected vehicles can influence the motion prediction performance. The precise recognition of driving behaviors and prediction of the vehicle motion is critical to the driving safety for connected automated vehicles (CAVs). Hence, in this study, an energy-aware driving pattern analysis and motion prediction system are proposed for CAVs using a deep learning-based time-series modeling approach. First, energy-aware longitudinal acceleration and deceleration behaviors and lateral lane-change behaviors are statistically analyzed. Then, a sliding standard deviation (SSD) test is applied to evaluate the smoothness of the trajectory and velocity signals considering different energy consumption levels. An energy-aware personalized joint time-series modeling (PJTSM) approach based on a deep recurrent neural network (RNN) and long short-term memory (LSTM) cell are proposed for accurate motion (trajectory and velocity) prediction of the leading vehicle. Finally, the differences in the prediction performance regarding different energy consumption levels are compared and discussed. It is shown that due to the higher randomness of the driving behaviors, the prediction accuracy for heavy energy users is the lowest among the three categories, which means it is harder to anticipate the driving behaviors of cars exhibiting heavy energy consumption. The personalized estimation of driving behaviors of CAVs will contribute to safer automated driving and transportation systems.

Predicting the future trajectory of a surrounding vehicle in congested traffic is one of the necessary abilities of an autonomous vehicle. In congestion, a vehicle's future movement is the result of its interaction with surrounding vehicles. A vehicle in congestion may have many neighbors in a relatively short distance, while only a small part of neighbors affect its future trajectory mostly. In this work, An interaction-aware method that predicts the future trajectory of an ego vehicle considering its interaction with eight surrounding vehicles is proposed. The dynamics of vehicles are encoded by LSTMs with shared weights, and the interaction is extracted with a simple CNN. The proposed model is trained and tested on trajectories extracted from the publicly accessible NGSIM US-101 dataset. Quantitative experimental results show that the proposed model outperforms previous models in root-mean-square error (RMSE). Results visualization shows that the model is able to predict future trajectory induced by lane change before the vehicle operates noticeable lateral movement to initiate lane changing.

The automated driving vehicles are experiencing a rapid development in worldwide recently. It is commonly believed that before the achievement of fully autonomous driving, the driver will always need to remain within the vehicle control loop. Hence, intelligent interaction and collaboration between the human driver and the automation will be an efficient solution for the improvement of road safety, traffic efficiency, and social acceptance to the automated vehicles. As a popular collaboration method, shared control has been widely studied in the past two decades. While it is still a challenging task to involve rich human factors into the shared control system to increase the driving experience and acceptance of the automation. In this study, a literature review on human-centered shared control is proposed towards solid research on driver-vehicle collaboration. First, the basic background and literature surveys on the human-machine collaboration (HMC) is proposed, and the important factors for efficient multi-agent collaboration and teaming are discussed. Then, different driver behavior and state modeling methods are reviewed. Based on the HMC schemes and driver behavior recognition techniques, literature surveys on human-centered shared control are proposed. Finally, challenges and future works on human-centered shared control are analyzed.

In this paper, a novel reference-free shared control system is designed for obstacle avoidance for automated vehicles. Rather than using a reference path to guide the driver, the proposed framework constrains the vehicle's status to guarantee the safety without scarifying the driver's freedom. The constrained Delaunay triangle method is introduced to identify the vehicle's position constraints and the constraints of obstacle avoidance, vehicle stability and physical limitations are investigated and unified. A nonlinear predictive control problem, which is constructed accounting nonlinear vehicle dynamics and given driver actions, is designed to optimize the steering and braking actions needed to keep the vehicle safe. The automation is supposed to correct the driver's steering or braking actions to prevent constraint violation and losing the control of vehicle. The simulation results show that the automation can assist the driver to avoid obstacles and guarantee the vehicle's stability with minimal control intervention.

This paper proposes a novel haptic shared control concept between human driver and autonomous controller for lane keeping in semi-autonomous vehicles. Based on the human-machine interaction during lane keeping, the level of cooperativeness for completion of driving task is identified. Using the identified level of cooperativeness along with the driver workload, the level of assistance required is determined based on an inverse U-shaped relationship. Subsequently based on the level of assistance required, a factor is developed to modulate the assistance torque generated by the autonomous controller. For the generation of the assistance torque, a new ℓ∞linear parameter varying (LPV) control technique is proposed to deal with large variations in the vehicle longitudinal speed and those in the modulation factor. The control architecture works on an integrated driver-in-the-loop model developed by considering vehicle yaw-slip dynamics, steering column, and neuromuscular human driver dynamics. Subsequent closed-loop control performance for dynamic road conditions with varying road curvatures is presented through extensive evaluations.

The mutual understanding between driver and vehicle is critically important to the design of intelligent vehicles and customized interaction interface. In this study, a deep learning-based joint driver behavior reasoning system toward multi-scale and multi-tasks behavior recognition is proposed. Specifically, a multi-scale driver behavior recognition system is designed to recognize both the driver's physical and mental states based on a deep encoder-decoder framework. The system jointly recognizes three driver behaviors, namely, mirror-checking, lane change intention, and emotions based on the shared encoder network. The encoder network is designed based on a deep convolutional neural network (CNN), and several decoders for different driver states estimation are proposed with fully connected (FC), and long short-term memory (LSTM) based recurrent neural networks (RNN), respectively. The proposed framework can be used as a solution to exploit the relationship between different driver states for intelligent vehicles towards an efficient driver-side understanding. The testing results on the Brain4Car dataset show accurate performance and outperform existing methods on driver postures, intention, and emotion recognition.

Predicting driver steering intention enables intelligent vehicles to optimize its assistance and collaborative strategies with the human driver in advance, which contribute to an intelligent mutual-understanding system for driver-vehicle collaboration. In this study, a deep time-series learning-enabled driver steering intention prediction system is developed based on the Electromyography (EMG) signal processing. Specifically, the connection between the upper limb EMG signals from different muscles and the steering torque is established using a deep bi-directional long short-term memory (BiLSTM) recurrent neural network (RNN). The deep time-series model is trained to predict the future steering torque with historical EMG signals, and the prediction horizon is selected as 200 ms in this study. Moreover, three different steering postures with different hand positions on the steering wheel are studied. A joint BiLSTM network with shared temporal pattern extraction layers is developed to investigate the impact of the hand positions on the steering intention prediction. It is found that based on the joint BiLSTM network, the most accurate steering intention can be achieved with both hands on 3-clock positions. The experiments are conducted on a driving simulator environment with 21 participants. The proposed system can be used for precise driver steering intention prediction system towards a better mutual-understanding module on the intelligent and automated driving vehicles.

Over the past decade, there has been a growing interest in large-scale and privacy-concerned machine learning, especially in the situation where the data cannot be shared due to privacy protection or cannot be centralized due to computational limitations. Parallel computation has been proposed to circumvent these limitations, usually based on the master-slave and decentralized topologies, and the comparison study shows that a decentralized graph could avoid the possible communication jam on the central agent but incur extra communication cost. In this brief, a consensus algorithm is designed to allow all agents over the decentralized graph to converge to each other, and the distributed neural networks with enough consensus steps could have nearly the same performance as the centralized training model. Through the analysis of convergence, it is proved that all agents over an undirected graph could converge to the same optimal model even with only a single consensus step, and this can significantly reduce the communication cost. Simulation studies demonstrate that the proposed distributed training algorithm for multi-layer neural networks without data exchange could exhibit comparable or even better performance than the centralized training model.

Human driver's intervention is still needed before achieving fully autonomous driving, and this poses a great challenge for ensuring safety and smoothness during control transition from automation to human driver. This paper addresses this challenge by proposing a novel predictive haptic takeover control method. First, a new human-machine interaction model considering two phases, namely machine dominance and human dominance, is developed to describe the interactive behaviors between human driver and automation system during takeover. Based on the model built, a novel haptic takeover controller is designed by using model predictive control approach in order to achieve a safe and smooth handover transition. The optimal steering input is derived by considering the expected paths of both human driver and automation system. The takeover controller generates predictive haptic steering torque based on driver's states, guiding and assisting human driver to gradually resume manual control. Simulation is conducted under a typical driving condition with proposed method. Simulation results indicate that the automation-to-human control transfer is completed safely and smoothly with the proposed haptic takeover controller, and meanwhile the vehicle performance and stability are also guaranteed.

In this paper, we address the problem of secure pose estimation of an autonomous vehicle (AV) under cyber attacks. An extended Kalman filter (EKF) is used to fuse measurements from multiple sensors including GPS, LIDAR, and IMU. To deal with the possible sensor attacks, we design a cumulative sum (CUSUM) detector to monitor the inconsistency between the predicted pose via mathematical model and the sensor measurement. An EKF reconfiguration scheme is proposed to mitigate the influence of sensor attacks once the compromised sensor is identified. The feasibility and effectiveness of the proposed secure pose estimation method are validated using a simulation platform built on Autoware and Gazebo.

This paper proposes an extensive overview of the tire-road interaction estimation issue as it relates to automated driving from the prospectives of sensor configuration, tire modeling, and estimation approaches. The tire-road interactions needed for estimation are first determined and classified. Then, the sensor configuration schemes of different types of tire-road interactions are presented and analyzed. The following introduces various types of tire models and provides the limitations and advantages of different estimation approaches based on categorizing and summarizing those techniques. Moreover, some interesting perspectives for future research are listed based on the extensive experience of the authors.

Autonomous driving presents an exciting new development in vehicle technology. It poses a new challenge in driver-automation collaboration particularly during handover transitions between human and machine. In order to deal with this problem, this paper proposes a novel control framework for the haptic take-over system. The high-level framework of the haptic take-over control system, which takes driver cognitive workload, neuromuscular dynamics and optimal trajectory planning into consideration, is developed. Under the proposed framework, the determination approach of the optimal input sequence is introduced. The model of the allowed driver take-over authority, which is associated with driver's cognitive workload, as well as muscle readiness during take- over, is investigated and developed. The haptic feedback torque controller is then designed so as to minimize the deviation between the allowed control authority and driver's current degree of participation. A handover process, along with the proposed take-over control method, is also simulated. The simulation results validate the feasibility and effectiveness of the proposed approach.

Drivers' decision and their corresponding behaviors are important aspects that can affect the driving safety, and it is necessary to understand the driver behaviors in real-time. In this study, an end-to-end driving-related tasks recognition system is proposed. Specifically, seven common driving activities are identified, which are normal driving, right mirror checking, rear mirror checking, left mirror checking, using in-vehicle video device, texting, and answering mobile phone. Among these, the first four activities are regarded as normal driving tasks, while the rest three are divided into distraction group. The images are collected using a consumer range camera, namely, Kinect. In total, five drivers are involved in the naturalistic data collection. Before training the identification model, the raw images are first segmented using a Gaussian mixture model (GMM) to extract the driver region from the background. Then, a pre-trained deep convolutional neural network (CNN) model is trained to classify the behaviors, which directly takes the processed RGB images as the input and outputs the identified label. In this work, the AIexNet is selected as the pre-trained CNN model. Then, to reduce the training cost, the transfer learning mechanism is applied to the CNN model. An average of 79% detection accuracy is achieved for the seven driving tasks. The proposed integration model can be used as a low-cost driver distraction and dangerous tasks recognition modeL.

As an important input for the simulation and design process of powertrains, a driving cycle needs to be representative of real-world driving behavior. For the purpose of reducing the time consumption in the simulation, a novel modeling method is required to get a representative short driving cycle from the driving datasets. In this paper, a stochastic model based driving cycle synthesis is introduced. The Markov Chain process is combined with a transition probability extracted from the input driving data to determine the next possible state of the vehicle. Specifically, the velocity and slope are generated simultaneously using a three-dimensional Markov Chain model. After the generation process, the result is validated by selected criteria. Furthermore, this synthesis can generate the driving cycle with the desired length to compress the original driving cycle. The results show that the successful compression of the driving cycle can be tested for the fuel economic in the powertrain simulation. At last, the standard deviation of acceleration is found that has a positive correlation of the compression capability of the driving cycle.

Driver decisions and behaviors regarding the surrounding traffic are critical to traffic safety. It is important for an intelligent vehicle to understand driver behavior and assist in driving tasks according to their status. In this paper, the consumer range camera Kinect is used to monitor drivers and identify driving tasks in a real vehicle. Specifically, seven common tasks performed by multiple drivers during driving are identified in this paper. The tasks include normal driving, left-, right-, and rear-mirror checking, mobile phone answering, texting using a mobile phone with one or both hands, and the setup of in-vehicle video devices. The first four tasks are considered safe driving tasks, while the other three tasks are regarded as dangerous and distracting tasks. The driver behavior signals collected from the Kinect consist of a color and depth image of the driver inside the vehicle cabin. In addition, 3-D head rotation angles and the upper body (hand and arm at both sides) joint positions are recorded. Then, the importance of these features for behavior recognition is evaluated using random forests and maximal information coefficient methods. Next, a feedforward neural network (FFNN) is used to identify the seven tasks. Finally, the model performance for task recognition is evaluated with different features (body only, head only, and combined). The final detection result for the seven driving tasks among five participants achieved an average of greater than 80% accuracy, and the FFNN tasks detector is proved to be an efficient model that can be implemented for real-time driver distraction and dangerous behavior recognition.

As a typical cyber-physical system (CPS), electrified vehicle becomes a hot research topic due to its high efficiency and low emissions. In order to develop advanced electric powertrains, accurate estimations of the unmeasurable hybrid states, including discrete backlash nonlinearity and continuous half-shaft torque, are of great importance. In this paper, a novel estimation algorithm for simultaneously identifying the backlash position and half-shaft torque of an electric powertrain is proposed using a hybrid system approach. System models, including the electric powertrain and vehicle dynamics models, are established considering the drivetrain backlash and flexibility, and also calibrated and validated using vehicle road testing data. Based on the developed system models, the powertrain behavior is represented using hybrid automata according to the piecewise affine property of the backlash dynamics. A hybrid-state observer, which is comprised of a discrete-state observer and a continuous-state observer, is designed for the simultaneous estimation of the backlash position and half-shaft torque. In order to guarantee the stability and reachability, the convergence property of the proposed observer is investigated. The proposed observer are validated under highly dynamical transitions of vehicle states. The validation results demonstrates the feasibility and effectiveness of the proposed hybrid-state observer.

The evaluation of driver's steering comfort, which is mainly concerned with the haptic driver-vehicle interaction, is important for the optimization of advanced driver assistance systems. The current approaches to investigating steering comfort are mainly based on the driver's subjective evaluation, which is time-consuming, expensive, and easily influenced by individual variations. This paper makes some tentative investigation of objective evaluation, which is based on the electromyogram (EMG) and movement trajectory of the driver's upper limbs during steering maneuvers. First, a steering experiment with 21 subjects is conducted, and EMG and movement trajectories of the driver's upper limbs are measured, together with their subjective evaluation of steering comfort. Second, five evaluation indices including EMG and movement information are defined based on the measurements from the first step. Correlation analyses are conducted between each evaluation index and steering comfort rating (SCR), and the results show that all of the indices have significant correlations with SCR. Then, an artificial neural network model is devised based on the aforementioned indices and its predicting performance of SCR is demonstrated as acceptable. The results reveal that it may be feasible to establish an objective evaluation approach for vehicle steering comfort.

The papers in this special section focus on cyber-physical systems in green transportation. Ground mobility is being in a paradigm shift toward more efficient and green transportation. Wireless networking, sensing, computing, and control advances have significantly changed the way the society interacts with the physical world. In the context of cyber-physical systems (CPS), transportation systems become highly multidisciplinary. They require an ever-increasing integration of mechanical, electrical/ electronic, control, and information disciplines. Emerging innovative technologies, such as automated driving and electrified vehicles, are also profoundly promoting connection, automation, and electrification of the current transportation sector.

As an important safety-critical cyber-physical system (CPS), the braking system is essential to the safe operation of the electric vehicle. Accurate estimation of the brake pressure is of great importance for automotive CPS design and control. In this paper, a novel probabilistic estimation method of brake pressure is developed for electrified vehicles based on multilayer artificial neural networks (ANNs) with Levenberg-Marquardt backpropagation (LMBP) training algorithm. First, the high-level architecture of the proposed multilayer ANN for brake pressure estimation is illustrated. Then, the standard backpropagation (BP) algorithm used for training of the feed-forward neural network (FFNN) is introduced. Based on the basic concept of BP, a more efficient training algorithm of LMBP method is proposed. Next, real vehicle testing is carried out on a chassis dynamometer under standard driving cycles. Experimental data of the vehicle and the powertrain systems are collected, and feature vectors for FFNN training collection are selected. Finally, the developed multilayer ANN is trained using the measured vehicle data, and the performance of the brake pressure estimation is evaluated and compared with other available learning methods. Experimental results validate the feasibility and accuracy of the proposed ANN-based method for braking pressure estimation under real deceleration scenarios.

In order to develop an advanced haptic take-over system for highly automated vehicles, research into the driver's neuromuscular dynamics is needed. In this paper a dynamic model of drivers' neuromuscular interaction with a steering wheel is firstly established. The transfer function and the natural frequency of the systems are analysed. In order to identify the key parameters of the driver-steering-wheel coupled system and investigate the system properties under different situations, experiments with drive-in-the-loop are carried out. For each test subject, two steering tasks, namely the passive and active steering tasks, are instructed to be completed. Furthermore, during the experiments, subjects manipulated the steering wheel with two distinct postures and three different hand positions. Based on the test results, key parameters of the transfer function and system properties are identified and investigated. The data and characteristics of the driver neuromuscular system are discussed and compared with respect to different steering tasks, hand positions and driver postures. These test results identified system properties that provide a good foundation for the development of a haptic take-over control system for automated vehicles.

Equipped with the regenerative braking system, electric vehicle coordinates friction braking and regenerative braking appropriately in normal braking conditions and activates anti-lock braking system (ABS) in emergency braking conditions. This paper mainly focuses on the control strategy of electric-hydraulic blended brake for ABS control of an electric passenger car. According to the variation of the adhesion coefficient under different roads, the maximum adhesion force and the optimal slip ratio are calculated in real-time. Then, the control strategy of electric-hydraulic hybrid ABS, in which regenerative braking and hydraulic braking are coordinated in order to obtain the maximum available road adhesion and guarantee vehicle's braking stability, is proposed. Based on the control strategy developed, simulations and test-bench experiments are carried out. Simulation and test results indicate that braking stability and control performance of vehicle on different roads are guaranteed by the proposed hybrid ABS control, validating the feasibility and the effectiveness of the algorithms. Compared with conventional hydraulic ABS, the electric-hydraulic hybrid ABS, ensuring better braking performance on various road surfaces, provides a good solution to active safety control of EVs.