Driver distraction recognition is a critical component of human-machine collaborative driving systems. Accurately identifying driver distraction behaviors is of great significance for improving road traffic safety. However, previous studies often focus on single experimental settings, neglecting the significant variations in data distribution caused by factors such as camera angles, lighting conditions, and experimental subjects across different environments. This leads to significant challenges in model generalization across domains and in diverse and uncertain real-world scenarios. To address these issues, this paper proposes a novel hybrid unsupervised domain adaptation framework. The proposed method achieves hybrid-aligned domain adaptation by minimizing subdomain-level feature distribution discrepancies between the source and target domains, while simultaneously reducing the divergence between the logits of classifiers that exhibit prediction discrepancies on the target domain. This enhances the accuracy of driver distraction behavior recognition in the target domain, improving the model’s generalization ability and robustness. Extensive experiments on four distracted driving datasets demonstrate that our proposed strategy outperforms previous methods, especially when dealing with datasets with imbalanced class distributions, which are more representative of real-world scenarios.

End-to-end autonomous driving has emerged as a promising paradigm for directly mapping sensor inputs to planning maneuvers using learning-based modular integrations. However, existing imitation learning (IL)-based models suffer from generalization to hard cases, and a lack of corrective feedback loop under post-deployment. While reinforcement learning (RL) offers a potential solution to tackle hard cases with optimality, it is often hindered by overfitting to specific driving cases, resulting in catastrophic forgetting of generalizable knowledge and sample inefficiency. To overcome these challenges, we propose Reinforced Refinement with Self-aware Expansion (R2SE), a novel learning pipeline that constantly refines hard domain while keeping generalizable driving policy for model-agnostic end-to-end driving systems. Through reinforcement fine-tuning and policy expansion that facilitates continuous improvement, R2SE features three key components: 1) Generalist Pretraining with hard-case allocation trains a generalist imitation learning (IL) driving system while dynamically identifying failure-prone cases for targeted refinement; 2) Residual Reinforced Specialist Fine-tuning optimizes residual corrections using reinforcement learning (RL) to improve performance in hard case domain while preserving global driving knowledge; 3) Self-aware Adapter Expansion dynamically integrates specialist policies back into the generalist model, enhancing continuous performance improvement. Experimental results in closed-loop simulation and real-world datasets demonstrate improvements in generalization, safety, and long-horizon policy robustness over state-of-the-art E2E systems, highlighting the effectiveness of reinforce refinement for scalable autonomous driving.

Driver monitoring systems face critical challenges in modern transportation, including limited multitasking capabilities and a lack of interpretability. These limitations hinder the accurate and comprehensive assessment of driver states such as distraction, drowsiness, and emotions, which are essential to ensure road safety. This paper introduces visual language models for multitask domain adaptation in driver monitoring (VLM-DM), a novel framework that addresses these challenges by leveraging advanced visual language models for the simultaneous execution of multiple driver monitoring tasks. By employing parameter-efficient training methods such as Low-Rank Adaptation (LoRA) and integrating dynamic prompt tuning, VLM-DM achieves superior performance compared to state-of-the-art methods. Our experiments on three benchmark datasets across different driver states, demonstrating significant improvements in multitask accuracy and interpretability. This work highlights the potential of advanced multitask and multimodal architectures in developing robust, scalable, and interpretable driver monitoring systems for real-world applications.

Autonomous driving systems require a comprehensive understanding and accurate prediction of the surrounding environment to facilitate informed decision-making in complex scenarios. Recent advances in learning-based systems have highlighted the importance of integrating prediction and planning. However, this integration poses significant alignment challenges through consistency between prediction patterns, to interaction between future prediction and planning. To address these challenges, we introduce a Hybrid-Prediction integrated Planning (HPP) framework, which operates through three novel modules collaboratively. First, we introduce marginal-conditioned occupancy prediction to align joint occupancy with agent-specific motion forecasting. Our proposed MS-OccFormer module achieves spatial-temporal alignment with motion predictions across multiple granularities. Second, we propose a game-theoretic motion predictor, GTFormer, to model the interactive dynamics among agents based on their joint predictive awareness. Third, hybrid prediction patterns are concurrently integrated into the Ego Planner and optimized by prediction guidance. The HPP framework establishes state-of-the-art performance on the nuScenes dataset, demonstrating superior accuracy and safety in end-to-end configurations. Moreover, HPP's interactive open-loop and closed-loop planning performance are demonstrated on the Waymo Open Motion Dataset (WOMD) and CARLA benchmark, outperforming existing integrated pipelines by achieving enhanced consistency between prediction and planning.

Learning-based techniques hold considerable promise in achieving human-like autonomous driving. However, one deployed policy encounters difficulties in satisfying the drivers’ diverse decision-making preferences simultaneously. Meanwhile, training personalized policies for each driver from scratch is time-consuming and resource-intensive. To address these challenges, this paper proposes a human-guided continual learning framework, wherein the human drivers could real-time take over a deployed policy when it performs unsatisfactorily, and the autonomous vehicle (AV) agent would automatically acquire human demonstrations and dynamically alter itself in accordance with personalized decision-making preference. Furthermore, a priority experience memory-enabled elastic weight consolidation (PEM-EWC) mechanism is developed to prevent the AV agent from overfitting to a limited number of human demonstrations and catastrophically forgetting its acquired fundamental driving abilities. Driver-in-the-loop simulations and real-world experiments are conducted in representative autonomous driving decision-making scenarios, and experimental results demonstrate the superior equilibrium of our proposed approach in terms of driving safety, human likeness, and training efficiency, compared to other baselines, which suggests that it provides a promising solution for personalized decision-making in autonomous driving. The supplementary video is available athttps://youtu.be/HKF0ayxMycc.

Autonomous driving (AD) will rely, either fully or partially, on data-driven approaches. As such, being aware of the algorithm limitation is crucial when implementing learning-based methods in such safety-critical contexts. A comprehensive AD monitor allows control authority to be transferred promptly to a contingency backup solution when the vehicle is recognized in impasses. To address this challenge, we propose MonitorGAN, a Bayesian discriminator trained within an adversarial framework, designed to recognize unknown traffic scenarios and monitor planning quality in open-world autonomous driving. Additionally, it is designed to be aware of its own limitations using a Bayesian approach. Unlike previous epistemic uncertainty estimation algorithms for self-driving, MonitorGAN is independent and planner-agnostic, capable of monitoring various types of planners without requiring real outlier exposure. MonitorGAN is trained exclusively on Argoverse 2 and tested through extensive cross-dataset experiments, including NGISM, HighD, RounD, and NuScenes, across three common planning schemes: learning-based, polynomial-based, and optimization-based, all of which use the same training dataset for interaction-aware planning. Both quantitative results and qualitative comparisons with other epistemic uncertainty estimation algorithms indicate that our approach can estimate the feasibility of the AD’s planning in a planner-agnostic manner and ensure safety.

Accurate recognition of driver distraction is significant for the design of human-machine cooperation driving systems. Existing studies mainly focus on classifying varied distracted driving behaviors, which depend heavily on the scale and quality of datasets and only detect the discrete distraction categories. Therefore, most data-driven approaches have limited capability of recognizing unseen driving activities and cannot provide a reasonable solution for downstream applications. To address these challenges, this paper develops a vision Transformer-enabled weakly supervised contrastive (W-SupCon) learning framework, in which distracted behaviors are quantified by calculating their distances from the normal driving representation set. The Gaussian mixed model (GMM) is employed for the representation clustering, which centralizes the distribution of the normal driving representation set to better identify distracted behaviors. A novel driver behavior dataset and the other three ones are employed for the evaluation, experimental results demonstrate that our proposed approach has more accurate and robust performance than existing methods in the recognition of unknown driver activities. Furthermore, the rationality of distraction levels for different driving behaviors is evaluated through driver skeleton poses. The constructed dataset and demo videos are available athttps://yanghh.io/Driver-Distraction-Quantification.

Driver drowsiness detection is of great significance in improving driving safety and has been widely studied in recent years. However, some existing methods have not fully utilized the drowsiness-related information, and some methods are susceptible to interference from the redundant information of input data. To address these issues, a video-based driver drowsiness detection method according to the key facial features including facial landmarks and local facial areas (VBFLLFA) is proposed in this paper. In order to fully utilize the key facial features related to drowsiness and exclude the interference of redundant information, the head movement information is obtained through facial landmark analysis and the movement information of eyes and mouth is acquired from the local facial areas. And the spatial filtering based on the common spatial pattern (CSP) algorithm is introduced to improve the discrimination of different classes of samples. To adequately extract the temporal and spatial features, a two-branch multi-head attention (TB-MHA) module is designed in this paper. Furthermore, the center loss with center vector distance penalty is introduced to further improve the discrimination of different classes of samples in the feature space. In addition to two public datasets, we specifically create a novel video-based driver drowsiness detection (VBDDD) dataset to evaluate the effectiveness of our method. The experimental results verify that our method can achieve very excellent performance in driver drowsiness detection tasks.

Conditional Automated Driving (CAD) has attracted widespread attention due to the substantial gap in achieving fully autonomous driving, wherein an essential endeavor entails determining the transition timing between automated and manual driving modes. Driver cognitive workload serves as a crucial indicator for identifying transition timing, while its precise determination is challenging with discrete workload levels in previous studies. To address this issue, this work develops a dual-stage learning framework to quantify driver cognitive workload continuously. Specifically, a semi-supervised co-training strategy is first designed to approximate workload values, and then supervised contrastive learning is employed to align them with their feature representations in the latent space. A novel driver workload dataset is constructed for the evaluation, and experimental results demonstrate that our proposed approach outperforms other state-of-the-art baselines in estimation accuracy. Furthermore, the rationality of quantified cognitive workload is analyzed through the driver’ subjective assessment, indicating it is a more reliable solution for achieving the driving authority transition.

Although deep reinforcement learning (DRL) methods are promising for making behavioral decisions in autonomous vehicles (AVs), their low training efficiency and difficulty to adapt to untrained cases hinder their applications. Introducing a human role in the DRL paradigm could improve training efficiency by using human prior knowledge and overcome untrained cases in deployment by online human takeover. In this study, a novel value-based DRL algorithm that leverages human guidance to improve its performance is proposed for addressing high-level decision-making problems in autonomous driving. We develop a new learning objective for DRL to increase the value of the human policy over the undertrained DRL policy so that the DRL agent can be encouraged to mimic human behaviors and thereby utilizing human guidance more efficiently. Our method can autonomously evaluate the importance of different human guidance, which makes it more robust for variation of human performance. The proposed DRL algorithm was used to address a challenging multiobjective lane-change decision-making problem. We collected human guidance from a human-in-the-loop driving experiment and evaluated our method in a high-fidelity simulator. Results validated the advantages of the proposed algorithm in terms of training efficiency and optimality in the decision-making problem compared to the baselines of state-of-the-art existing methods. Results also revealed the favorable fine-tuning ability of the proposed algorithm, which is promising for addressing the long-tail issue in DRL-based autonomous driving. Our methodology does not introduce additional domain knowledge so that it can be seamlessly applied to other similar issues. The supplementary video is available at https://youtu.be/Ec7WkqeLsB8.

Reinforcement learning (RL) is a promising approach in unmanned ground vehicles (UGVs) applications, but limited computing resource makes it challenging to deploy a well-behaved RL strategy with sophisticated neural networks. Meanwhile, the training of RL on navigation tasks is difficult, which requires a carefully-designed reward function and a large number of interactions, yet RL navigation can still fail due to many corner cases. This shows the limited intelligence of current RL methods, thereby prompting us to rethink combining RL with human intelligence. In this paper, a human-guided RL framework is proposed to improve RL performance both during learning in the simulator and deployment in the real world. The framework allows humans to intervene in RL's control progress and provide demonstrations as needed, thereby improving RL's capabilities. An innovative human-guided RL algorithm is proposed that utilizes a series of mechanisms to improve the effectiveness of human guidance, including human-guided learning objective, prioritized human experience replay, and human intervention-based reward shaping. Our RL method is trained in simulation and then transferred to the real world, and we develop a denoised representation for domain adaptation to mitigate the simulation-to-real gap. Our method is validated through simulations and real-world experiments to navigate UGVs in diverse and dynamic environments based only on tiny neural networks and image inputs. Our method performs better in goal-reaching and safety than existing learning- and model-based navigation approaches and is robust to changes in input features and ego kinetics. Furthermore, our method allows small-scale human demonstrations to be used to improve the trained RL agent and learn expected behaviors online.

Reinforcement learning has demonstrated its potential in a series of challenging domains. However, many real-world decision making tasks involve unpredictable environmental changes or unavoidable perception errors that are often enough to mislead an agent into making suboptimal decisions and even cause catastrophic failures. In light of these potential risks, reinforcement learning with application in safety-critical autonomous driving domain remains tricky without ensuring robustness against environmental uncertainties (e.g., road adhesion changes or measurement noises). Therefore, this paper proposes a novel constrained adversarial reinforcement learning approach for robust decision making of autonomous vehicles at highway on-ramps. Environmental disturbance is modelled as an adversarial agent that can learn an optimal adversarial policy to thwart the autonomous driving agent. Meanwhile, observation perturbation is approximated to maximize the variation of the perturbed policy through a white-box adversarial attack technique. Furthermore, a constrained adversarial actor-critic algorithm is presented to optimize an on-ramp merging policy while keeping the variations of the attacked driving policy and action-value function within bounds. Finally, the proposed robust highway on-ramp merging decision making method of autonomous vehicles is evaluated in three stochastic mixed traffic flows with different densities, and its effectiveness is demonstrated in comparison with the competitive baselines.

Since emotions generally affect driver’s behavior, judgment, and reaction time, accurately identifying driver’s emotions is of great significance to improve the safety and comfort of intelligent driving system. However, the gender, skin color, age, and appearance of different drivers often have big differences, which will greatly interfere with the emotional recognition process. Besides, light intensity inside the vehicle varies with different time, weather, and location, which will also pose a challenge to driver emotion recognition. In this paper, a robust driver emotion recognition method based on feature separation is proposed to overcome the interference of individual differences and illumination changes. In order to realize the separation of expression-related features and irrelevant features, we design a high-purity feature separation (HPFS) framework based on partial feature exchange and the constraints of multiple loss functions. To verify that the proposed method can overcome the interference of illumination changes, we specifically create a multiple light intensities driver emotion recognition (MLI-DER) dataset and conduct a great deal of experiments on the dataset. In addition, to further demonstrate that our method can largely alleviate the interference of individual difference, some cross-subject emotion recognition experiments are conducted on two famous facial expression recognition datasets FACES and Oulu-CASIA and the experimental results are compared with that of some state-of-the-art methods.

Accurate recognition of driver braking intensity is of great importance for intelligent braking system. In this paper, the braking intensity is classified into four clusters based on an unsupervised Gaussian mixture model (GMM). Then, the architecture of an adaptive-network-based fuzzy inference system (ANFIS) is proposed for braking intensity prediction. A batch learning rule that combines the recursive least squares and gradient descent method used for training ANFIS is adopted to improve the generalization capability. The training data are collected from a hybrid vehicle under real driving conditions. In addition, co-simulation with the software of MATLAB/Simulink and Hardware-in-the-Loop (HiL) tests for an Electronic-Hydraulic Brake (EHB) system are carried out. In comparison to other typical learning methods, the simulation and experimental results demonstrate the effectiveness and accuracy of the proposed hybrid learning approach for braking intensity recognition in different braking scenarios.