Safe and executable trajectory planning in urban environments requires jointly considering traffic-related elements, ego behavior, and vehicle kinematics, posing challenges to the real-time performance and convergence of optimization-based methods. This paper proposes a behavior-guided optimization framework that structures the solution space at the spatiotemporal drivable domain level to facilitate fast and stable convergence. The planning space is partitioned into modular spatiotemporal domains, termed Behavior Cells (BCs), which encode ego motion feasibility and traffic-induced decisions. Feasible high-level behaviors are systematically enumerated through structured BC combinations and evaluated via a finite-horizon Markov decision process. The selected BC combination defines a continuous, behavior-consistent solution space, within which a dynamic two-stage optimization progressively restores the full planning formulation, enabling efficient and robust trajectory generation. Extensive simulations across diverse traffic scenarios demonstrate consistent reliability and real-time performance under varying traffic densities. On-road experiments further validate effectiveness in real-world urban environments. More detailed results are available at:https://lshasd123.github.io/Behavior-Cells/

Accurate assessment of human trust in driver assistance systems is crucial for enhancing user acceptance and system safety. While prior research has explored physiological signals like skin conductance and heart rate to gauge trust, the link between these signals and trust remains insufficiently understood. Here, we present a novel approach to interpreting driver trust in intelligent warning systems by integrating subjective measures from questionnaires with objective electroencephalography (EEG) data. We develop TrustNet, a model leveraging separable convolution to capture the spatiotemporal dynamics of EEG signals and class activation mapping (CAM) to identify trust-relevant features. TrustNet achieves superior performance in trust assessment and classification, with accuracy and F1 score both exceeding 87%. CAM analysis reveals that EEG beta- and gamma-wave changes in the occipital and frontal regions are strongly associated with trust dynamics. Misclassification analysis highlights sensor noise and individual differences in response variability as key factors affecting performance. These findings demonstrate the feasibility of EEG-based trust assessment, offering new avenues for adaptive driver assistance systems responsive to human trust.

Advanced driver assistance systems require a comprehensive understanding of the driver’s mental/physical state and traffic context but existing works often neglect the potential benefits of joint learning between these tasks. This paper proposes MMTL-UniAD, a unified multi-modal multitask learning framework that simultaneously recognizes driver behavior (e.g., looking around, talking), driver emotion (e.g., anxiety, happiness), vehicle behavior (e.g., parking, turning), and traffic context (e.g., traffic jam, traffic smooth). A key challenge is avoiding negative transfer between tasks, which can impair learning performance. To address this, we introduce two key components into the framework: one is the multi-axis region attention network to extract global context-sensitive features, and the other is the dual-branch multimodal embedding to learn multi-modal embeddings from both task-shared and task-specific features. The former uses a multi-attention mechanism to extract task-relevant features, mitigating negative transfer caused by task-unrelated features. The latter employs a dual-branch structure to adaptively adjust task-shared and task-specific parameters, enhancing cross-task knowledge transfer while reducing task conflicts. We assess MMTL-UniAD on the AIDE dataset, using a series of ablation studies, and show that it outperforms state-of-the-art methods across all four tasks. The code is available on https://github.com/Wenzhuo-Liu/MMTL-UniAD.

Multi-task learning (MTL) can advance assistive driving by exploring inter-task correlations through shared representations. However, existing methods face two critical limitations: single-modality constraints limiting comprehensive scene understanding and inefficient architectures impeding real-time deployment. This paper proposes TEM3-Learning (Time-Efficient Multimodal Multi-task Learning), a novel framework that jointly optimizes driver emotion recognition, driver behavior recognition, traffic context recognition, and vehicle behavior recognition through a two-stage architecture. The first component, the mamba-based multi-view temporal-spatial feature extraction subnetwork (MTS-Mamba), introduces a forward-backward temporal scanning mechanism and global-local spatial attention to efficiently extract low-cost temporal-spatial features from multi-view sequential images. The second component, the MTL-based gated multimodal feature integrator (MGMI), employs task-specific multi-gating modules to adaptively highlight the most relevant modality features for each task, effectively alleviating the negative transfer problem in MTL. Evaluation on the AIDE dataset, our proposed model achieves state-of-the-art accuracy across all four tasks, maintaining a lightweight architecture with fewer than 6 million parameters and delivering an impressive 142.32 FPS inference speed. Rigorous ablation studies further validate the effectiveness of the proposed framework and the independent contributions of each module. The code is available on https://github.com/Wenzhuo-Liu/TEM3-Learning.

In recent years, researchers have focused on identifying tasks related to driver state, traffic environment, and others to enhance the safety of autonomous driving assistance systems. However, current research on these tasks is conducted independently, neglecting the interconnections between the driver, traffic environment, and vehicle. In this paper, we propose a Unified Multi-task Assistive Driving Network Based on Multimodal Fusion (UMD-Net), the first unified model capable of recognizing four tasks simultaneously by utilizing multimodal data: driver behavior recognition, driver emotion recognition, traffic context recognition, and vehicle behavior recognition. In order to better enhance the synergistic effects between multiple tasks, we designed the position-sensitive multi-directional attention feature extraction subnetwork and recursive dynamic feature fusion module. The former captures the key features of multi-view images by different directions of attention mechanism to improve the generalization of the model across multiple tasks. The latter dynamically adjusts the fusion weight according to the multimodal features to enhance the representation ability of important features in multi-task learning. Our model was evaluated on the public dataset AIDE, achieving the best performance across all four tasks and a high accuracy of 95.31% in the traffic context recognition task, demonstrating the superiority of our approach. The code is available on https://github.com/Wenzhuo-Liu/UMD-Net.

Effective driving style analysis is critical to developing human-centered intelligent driving systems that consider drivers’ preferences. However, the approaches and conclusions of most related studies are diverse and inconsistent because no unified datasets tagged with driving styles exist as a reliable benchmark. The absence of explicit driving style labels makes verifying different approaches and algorithms difficult. This paper provides a new benchmark by constructing a natural dataset of Driving Style (100-DrivingStyle) tagged with the subjective evaluation of 100 drivers’ driving styles. In this dataset, the subjective quantification of each driver’s driving style is from themselves and an expert according to the Likert-scale questionnaire. The testing routes are selected to cover various driving scenarios, including highways, urban, highway ramps, and signalized traffic. The collected driving data consists of lateral and longitudinal manipulation information, including steering angle, steering speed, lateral acceleration, throttle position, throttle rate, brake pressure, etc. This dataset is the first to provide detailed manipulation data with driving-style tags, and we demonstrate its benchmark function using six classifiers. The 100-DrivingStyle dataset is available via https://github.com/chaopengzhang/100-DrivingStyle-Dataset

Multi-agent-based cooperation of autonomous vehicles(AVs) holds the potential to improve road safety, reduce emissions, and increase transport efficiency. However, the presence of uncertainties stemming from various sources poses a risk to the communication network and can alter the network topology, potentially causing instability in the multi-vehicle system. These uncertainties originate from two main sources: internal multi-vehicle system and external traffic environment. Time delays and packet losses contribute to uncertainties within the internal multi-vehicle system due to the uncontrollability of communication quality. Additionally, the dynamic nature of traffic environments introduces uncertainties related to the number of vehicles, interaction relationships, tasks, and destinations, thereby affecting communication resources and network topologies. Consequently, it is imperative to study the uncertainties faced by the multi-agent system and explore consensus methods for addressing these uncertainties. Notably, this study represents the first comprehensive review of consensus methods for both platooning and broader multi-agent cooperation in the presence of uncertain networks. Furthermore, a systematic summary of multi-agent consensus methods is presented, explicitly addressing two aspects of network uncertainty: imperfect communication transmission and the intricacies of traffic dynamics. The conclusion provides insights into open research issues, paving the way for future studies aimed at enhancing overall multi-vehicle system performance, including aspects such as convergence rate, robustness, and resilience.

Humans are experts at making decisions for challenging driving tasks with uncertainties. Many efforts have been made to model the decision-making process of human drivers at the behavior level. However, limited studies explain how human drivers actively make trustworthy sequential decisions to complete interactive driving tasks in an uncertain environment. This paper argues that human drivers intently search for actions to reduce the uncertainty of their perception of the environment, i.e., perceptual uncertainty, to a low level that allows them to make a trustworthy decision easily. This paper provides a proof-of-concept framework to empirically reveal that human drivers’ perceptual uncertainty decreases when executing interactive tasks with uncertainties. We first introduce an explainable-artificial intelligence approach (i.e., SHapley Additive exPlanation, SHAP) to determine the salient features on which human drivers base decisions. Then, we use entropy-based measures to quantify the drivers’ perceptual changes in these ranked salient features across the decision-making process, reflecting the changes in uncertainties. The validation and verification of our proposed method are conducted in the highway on-ramp merging scenario with congested traffic using the INTERACTION dataset. Experimental results support that human drivers intentionally seek information to reduce their perceptual uncertainties in the number and rank of salient features of their perception of environments to make a trustworthy decision.

Driving style is usually used to characterize driving behavior for a driverora group of drivers. However, it remains unclear how one individual’s driving style shares certain common grounds with other drivers. Our insight is that driving behavior is a sequence of responses to the weighted mixture of latent driving styles that are shareablewithinandbetweenindividuals. To this end, this paper develops a hierarchical latent model to learn the relationship between driving behavior and driving styles. We first propose a fragment-based approach to represent complex sequential driving behavior in a low-dimension feature space. Then, we provide an analytical formulation for the interaction of driving behavior and shareable driving styles through a hierarchical latent model. This model successfully extracts latent driving styles from extensive driving behavior data without the need for manual labeling, offering an interpretable statistical structure. Through real-world testing involving 100 drivers, our developed model is validated, demonstrating a subjective-objective consistency exceeding 90%, outperforming the benchmark method. Experimental results reveal that individuals share driving styles within and between them. We also found that individuals inclined towards aggressiveness only exhibit a higher proportion of such behavior rather than persisting consistently to be aggressive.

Potential fields have been integrated with local path-planning algorithms for autonomous vehicles (AVs) to tackle challenging scenarios with dense and dynamic obstacles. Most existing potential fields are isotropic without considering the traffic agent’s geometric shape and could cause failures due to local minima. We propose a three-dimensional potential field (TriPField) model to overcome this drawback by integrating an ellipsoid potential field with a Gaussian velocity field (GVF). Specifically, we model the surrounding vehicles as ellipsoids in corresponding ellipsoidal coordinates, where the formulated Laplace equation is solved with boundary conditions. Meanwhile, we develop a nonparametric GVF to capture the multi-vehicle interactions and then plan the AV’s velocity profiles, reducing the path search space and improving computing efficiency. Finally, a local path-planning framework with our TriPField is developed by integrating model predictive control to consider the constraints of vehicle kinematics. Our proposed approach is verified in three typical scenarios, i.e., active lane change, on-ramp merging, and car following. Experimental results show that our TriPField-based planner obtains a shorter, smoother local path with a slight jerk during control, especially in the scenarios with dense traffic flow, compared with traditional potential field-based planners. Our proposed TriPField-based planner can perform emergent obstacle avoidance for AVs with a high success rate even when the surrounding vehicles behave abnormally.

Urban digital twin (UDT) technologies offer new opportunities for intelligent road inspection (IRI). This paper first reviews the state-of-the-art algorithms used in the two key components of UDT-based IRI systems: (1) multi-temporal, multi-dimension, multi-score, and heterogeneous road data acquisition, and (2) road distress detection. This paper then summarizes the UDTIRI competition, organized in conjunction with IEEE Bigdata 2022. More details on our competition are available at sites.google.com/view/udtiri-workshop/bigdata-2022.

Humans make daily routine decisions based on their internal states in intricate interaction scenarios. This article presents a probabilistically reconstructive learning approach to identify the internal states of multivehicle sequential interactions when merging at highway on-ramps. We treated the merging task’s sequential decision as a dynamic, stochastic process and then integrated the internal states into a hidden Markov model (HMM)-Gaussian mixture regression (GMR) model, a probabilistic combination of an extended GMR and HMM. We also developed a variant of the expectation–maximization (EM) algorithm to estimate the model parameters and verified it based on a real-world dataset. Experiment results reveal that three interpretable internal states can semantically describe the interactive merge procedure at highway on-ramps. This finding provides a basis for developing an efficient model-based decision-making algorithm for autonomous vehicles (AVs) in a partially observable environment. Note to Practitioners—Model-based learning approaches have obtained increasing attention in decision-making design due to their stability and interpretability. This article was built upon two facts: 1) intelligent agents can only receive partially observable environmental information directly through their equipped sensors in the real world and 2) humans mainly utilize the internal states and associated dynamics inferred from observations to make proper decisions in complex environments. Similarly, autonomous vehicles (AVs) need to understand, infer, anticipate, and exploit the internal states of dynamic environments. Applying probabilistic decision-making models to AVs requires updating the internal states’ beliefs and associated dynamics after getting new observations. The designed and verified emission model in hidden Markov model (HMM)-Gaussian mixture regression (GMR) provides a modifiable functional module for online updates of the associated internal states. Experiment results based on the real-world driving dataset demonstrate that the internal states extracted using HMM-GMR can represent the dynamic decision-making process semantically and make an accurate prediction.

Driver model adaptation (DMA) plays an essential role for driving behaviour modelling when there is a lack of sufficient data for training the new model. A new data-driven DMA method is proposed in this paper to realise the instance-level knowledge transfer between individual drivers. Using the importance-weighted transfer learning (IWTL), the data collected from one driver (source driver) can be directly used to train the model of another driver (target driver). Under the framework of IWTL, the relationship between two different drivers can be modelled by the importance weight (IW). Two estimation methods Kullback-Leibler (KL) Divergence and least-squares (LS), are used to estimate IW for each data instance by modelling the importance-weight function as a radial basis function (RBF). Experiments based on the driving simulator and real vehicle are carried out to test the performance of TL for steering behaviour adaptation during the overtaking manoeuvre. The experimental results show that the TL method can transfer the knowledge observed from one driver to another when training the new driver model without sufficient data by keeping the modelling error at a low level.

Merging at highway on-ramps while interacting with other human-driven vehicles is challenging for autonomous vehicles (AVs). An efficient route to this challenge requires exploring and exploiting knowledge of the interaction process from demonstrations by humans. However, it is unclear what information (or environmental states) is utilized by the human driver to guide their behavior throughout the whole merging process. This paper provides quantitative analysis and evaluation of the merging behavior at highway on-ramps with congested traffic in a volume of time and space. Two types of social interaction scenarios are considered based on the social preferences of surrounding vehicles:courteousandrude. The significant levels of environmental states for characterizing the interactive merging process are empirically analyzed based on the real-world INTERACTION dataset. Experimental results reveal two fundamental mechanisms in the merging process: 1) Human drivers select different states to make sequential decisions at different moments of task execution; and 2) the social preference of surrounding vehicles can impact variable selection for making decisions. It implies that efficient decision-making design should filter out irrelevant information while considering social preference to achieve comparable human-level performance. These essential findings shed light on developing new decision-making approaches for AVs.

Understanding driver interaction behavioral semantics has potential benefits to autonomous car’s decision-making design. This article presents a framework of analyzing various encountering behaviors through decomposing driving encounter sequential data into small building blocks, called traffic primitives, using a Bayesian nonparametric learning (BNPL) approach. This framework offers a flexible way to gain semantic insights into complex driving encounters without any prerequisite knowledge of interaction behavior categories. Its effectiveness is then validated using 976 naturalistic driving encounters from which more than 4000 traffic primitives were learned with the BNPL approach. After that, a dynamic time warping method integrated with$k$-means clustering is then developed to cluster all these extracted traffic primitives into groups. Experimental results identify 20 kinds of traffic primitives capable of representing the essential components of driving encounters in our database. Based on the results, we conclude that the proposed primitive-based analysis could prove useful for autonomous vehicle applications.

Accurate speed prediction is practically critical to eco-safe driving for intelligent vehicles. Existing research only makes vehicles adapt to the dynamic driving environment while rarely considering the influence of human driving preferences. This paper proposes a learning-based model to leverage human driving preferences into speed prediction. We first designed an Oriented Hidden Semi-Markov Model (Oriented-HSMM) to learn and predict the driver’s driving preference sequences while considering traffic flow influence. Then, we developed an optimal speed prediction algorithm to retrieve the smooth speed trajectories with maximal likelihood based on the estimated driving preferences. Finally, we evaluated the proposed model using the Next Generation Simulation (NGSIM) data compared to its counterparts that do not consider driving preferences. Experimental results demonstrate that our proposed Oriented-HSMM method reaches the best results and achieves a satisfying performance with a low mean absolute error (4.16 km/h) and root mean square error (5.08 km/h) at a 200 m prediction horizon.

Interpretation of common-yet-challenging inter- action scenarios can benefit well-founded decisions for autonomous vehicles. Previous research achieved this using their prior knowledge of specific scenarios with predefined models, limiting their adaptive capabilities. This paper describes a Bayesian nonparametric approach that leverages continuous (i.e., Gaussian processes) and discrete (i.e., Dirichlet processes) stochastic processes to reveal underlying interaction patterns of the ego vehicle with other nearby vehicles. Our model relaxes dependency on the number of surrounding vehicles by developing an acceleration-sensitive velocity field based on Gaussian processes. The experiment results demonstrate that the velocity field can represent thespatialinteractions between the ego vehicle and its surroundings. A discrete Bayesian nonparametric model, integrating Dirichlet processes and hidden Markov models, is developed to learn the interaction patterns over thetemporalspace by segmenting and clustering the sequential interaction data into interpretable granular patterns automatically. We then evaluate our approach in the highway discretionary lane-change scenarios using the highD dataset collected from real-world settings. Results demonstrate that our proposed Bayesian nonparametric approach provides an insight into the complicated discretionary lane-change interactions of the ego vehicle with multiple surrounding traffic participants based on the interpretable interaction patterns and their transition properties in temporal relationships. Our proposed approach sheds light on efficiently analyzing other kinds of multi-agent interactions, such as vehicle-pedestrian interactions. View associated demos via:https://chengyuan-zhang.github.io/Multivehicle-Interaction.

Reliable representation of multi-vehicle interactions in urban traffic is pivotal but challenging for autonomous vehicles due to the volatility of the traffic environment, such as roundabouts and intersections. This paper describes a semi-stochastic potential field approach to represent multi-vehicle interactions by integrating a deterministic field approach with a stochastic one. First, we conduct a comprehensive evaluation of potential fields for representing multi-agent intersections from the deterministic and stochastic perspectives. For the former, the estimates at each location in the region of interest (ROI) are deterministic, which is usually built using a family of parameterized exponential functions directly. For the latter, the estimates are stochastic and specified by a random variable, which is usually built based on stochastic processes such as the Gaussian process. Our proposed semi-stochastic potential field, combining the best of both, is validated based on the INTERACTION dataset collected in complicated real-world urban settings, including intersections and roundabout. Results demonstrate that our approach can capture more valuable information than either the deterministic or stochastic ones alone. This work sheds light on the development of algorithms in decision-making, path/motion planning, and navigation for autonomous vehicles in the cluttered urban settings.

Generating multi-vehicle interaction scenarios can benefit motion planning and decision making of autonomous vehicles when on-road data is insufficient. This paper presents an efficient approach to generate varied multi-vehicle interaction scenarios that can both adapt to different road geometries and inherit the key interaction patterns in real-world driving. Towards this end, the available multi-vehicle interaction scenarios are temporally segmented into several interpretable fundamental building blocks, called traffic primitives, via the Bayesian nonparametric learning. Then, the changepoints of traffic primitives are transformed into the desired road to generate collision-free interaction trajectories through a sampling-based path planning algorithm. The Gaussian process regression is finally introduced to control the variance and smoothness of the generated multi-vehicle interaction trajectories. Experiments with simulation results of three multi-vehicle trajectories at different road conditions are carried out. The experimental results demonstrate that our proposed method can generate a bunch of human-like multi-vehicle interaction trajectories that can fit different road conditions remaining the key interaction patterns of agents in the provided scenarios, which is import to the development of autonomous vehicles.

To safely and efficiently change lanes among human drivers, autonomous vehicles (AVs) should make human-like decisions and seamlessly cooperate with surrounding vehicles. Both overaggressive and over-conservative cut-in maneuvers will have adverse effects on traffic efficiency and safety. However, it is still not entirely clear how much influence of the AV's cut-in behavior would lay on the surrounding drivers in urban traffic. To investigate this question, we design a series of driving scenarios and analyze the impact of different cut-in maneuvers performed by the human-like AV on the surrounding drivers' comfort. Ten volunteer drivers participate in our experiment and take a series of trials in a driving simulator. The experimental results demonstrate that the relative distance between the AV and the target car on the adjacent lane has a more significant effect on the surrounding drivers' comfort than the relative speed does. In addition, different parameters should be considered with different cut-in scenarios. This conclusion could provide practical support to make a friendly cut-in decision for the AVs.

Generating multi-vehicle trajectories from existing limited data can provide rich resources for autonomous vehicle development and testing. This paper introduces a multi-vehicle trajectory generator (MTG) that can encode multi-vehicle interaction scenarios (called driving encounters) into an interpretable representation from which new driving encounter scenarios are generated by sampling. The MTG consists of a bi-directional encoder and a multi-branch decoder. A new disentanglement metric is then developed for model analyses and comparisons in terms of model robustness and the independence of the latent codes. Comparison of our proposed MTG with β-VAE and InfoGAN demonstrates that the MTG has stronger capability to purposely generate rational vehicle-to-vehicle encounters through operating the disentangled latent codes. Thus the MTG could provide more data for engineers and researchers to develop testing and evaluation scenarios for autonomous vehicles.

Fast recognition of a driver's decision-making style when changing lanes plays a pivotal role in a safety-oriented and personalized vehicle control system design. This article presents a time-efficient recognition method by integrating k-means clustering (k-MC) with the K-nearest neighbor (KNN) algorithm, called kMC-KNN. Mathematical morphology is implemented to automatically label the decision-making data into three styles (moderate, vague, and aggressive), while the integration of k-MC and the KNN algorithm helps to improve the recognition speed and accuracy. Our developed mathematical-morphology-based clustering algorithm is then validated by a comparison with agglomerative hierarchical clustering. Experimental results demonstrate that the developed kMC-KNN method, in comparison with the traditional KNN algorithm, can shorten the recognition time by more than 72.67% with a recognition accuracy of 90-98%. In addition, our developed kMCKNN method also outperforms a support vector machine in terms of recognition accuracy and stability. The developed time-efficient recognition approach would have great application potential for in-vehicle embedded solutions with restricted design specifications.

Estimating a driver's lane-change (LC) intent is very important so as to avoid traffic accidents caused by improper LC maneuvers. This paper proposes a lane-change Bayesian network (LCBN) incorporated with a Gaussian mixture model (GMM), termed as LCBN-GMM, to estimate a driver's LC intent considering a driver's driving style over varying scenarios. According to the scores made by participates with a behavioral-psychological questionnaire, three driving styles are classified. In order to get more effective labeled LC and lane-keep (LK) data for model training, we propose a gaze-based labeling (GBL) method by monitoring a drivers's gaze behavior, instead of using a time-window labeling method. The capability of LCBN-GMM to estimate a driver's lane-change intent is evaluated in different LC scenarios and driving styles, in comparison to support vector machine and Naive Bayes. Data are collected in a seat-box-based driving simulator where 32 drivers, consisting of 9 aggressive, 15 neutral, and 8 conservative drivers, participated. Experimental results demonstrate that the LCBN-GMM with GBL achieves the best performance, estimating a driver's LC intent an average of 4.5 s ahead of actual LC maneuvers with 78.2% accuracy considering both driving style and contextual traffic, compared with other approaches.

Driving style analysis plays a pivotal role in intelligent vehicle design. This paper presents a novel framework for driving style analysis based on primitive driving patterns. To this end, a Bayesian nonparametric approach based on a hidden semi-Markov model (HSMM) is introduced to extract the primitive driving patterns from muti-dimensional time-series driving data without prior knowledge of these driving patterns. For the Bayesian nonparametric approach, a hierarchical Dirichlet process (HDP) is applied to learn the unknown smooth dynamical modes in the HSMM, called primitive driving patterns. Two other types of Bayesian nonparametric approaches (HDP-HMM and sticky HDP-HMM) are developed as comparatives in order to show the advantages of the HDP-HSMM. The naturalistic car-following data of 18 drivers are collected from the University of Michigan Safety Pilot Model Deployment database. For each driver, 75 primitive driving patterns are semantically predefined according to their physical and psychological perception thresholds. The individual driving styles are then semantically analyzed based on the distribution over primitive driving patterns, and the similarity of driving styles among drivers is then evaluated. Experimental results demonstrate that the utilization of driving primitive pattern provides a semantically interpretable way to analyze driver's behavior and driving style.

Deep learning techniques have been widely used in autonomous driving community for the purpose of environment perception. Recently, it starts being adopted for learning end-to-end controllers for complex driving scenarios. However, the complexity and nonlinearity of the network architecture limits its interpretability to understand driving scenarios and judge the importance of certain visual regions in sensory scenes. In this paper, based on the convolutional neural network (CNN), we propose two complementary frameworks to automatically determine the most contributive regions of the input scenes, offering intuitive knowledge of how a trained end-to-end autonomous vehicle controller understands driving scenarios. In the first framework, a feature map-based method is proposed by leveraging current progress in CNN visualization, in which the deconvolution approach recovers the feature maps to extract features that contribute most to understand driving scenes. In the second framework, the importance level of regions is ranked using the error map between the labeled and predicted control inputs generated by occluding different parts of input scenes, thus providing a pixel-wise rank of importance. Test data sets with extracted contributive regions are input to the CNN controller. Then, different CNN controllers trained with the new data sets preprocessed using our proposed frameworks are verified via closed-loop tests. Results show that both the features identified from the first framework and the regions identified from the second framework are of crucial importance to scene understanding for the controller and can significantly affect the performance of CNN controllers.

A new driver model adaptation (DMA) method is proposed in this paper to help the model adaptation between different individual drivers. This method is based on transfer learning which can improve the DMA process at data level. The Gaussian mixture model (GMM)-based method is used to model the steering behaviour of drivers during the overtaking manoeuvre. Based on the GMM model, an alignment-based transfer learning technique named local Procrustes analysis (LPA) is modified to formulate the transfer learning problem for driver steering behaviour. A series of experiments based on the data collected from a driving simulator are carried out to evaluate the proposed modified LPA (MLPA). The experimental results verify the ability of MLPA for knowledge transfer. Compared with the GMM-only method and LPA, MLPA shows better performance on the prediction accuracy with much lower predicting errors in most cases.

Learning knowledge from driving encounters could help self-driving cars make appropriate decisions when driving in complex settings with nearby vehicles engaged. This paper develops an unsupervised classifier to group naturalistic driving encounters into distinguishable clusters by combining an auto-encoder with k-means clustering (AE-kMC). The effectiveness of AE-kMC was validated using the data of 10,000 naturalistic driving encounters which were collected by the University of Michigan, Ann Arbor in the past five years. We compare our developed method with the k-means clustering methods and experimental results demonstrate that the AE-kMC method outperforms the original k-means clustering method.

To safely, efficiently change lanes among human drivers, autonomous vehicles (AV) should act as close as possible to human to decide when to friendly cut in and seamlessly cooperate with surrounding vehicles. However, it is still not fully clear about how AV cut-in behavior would influence on human drivers. This paper comprehensively analyzes the influence of AV cut-behavior on human drivers in terms of comfort levels over different cut-in scenarios in a driving simulator. The experiment results demonstrate that the relative distance has great influence on the comfort level of human drivers compared to relative speed; and the integrated influence of relative distance and speed between AV and the behind target vehicle have an important role in the influence. These findings could provide an empirical basis for the decision-making design of autonomous vehicles.

Deep learning-based approaches have been widely used for training controllers for autonomous vehicles due to their powerful ability to approximate nonlinear functions or policies. However, the training process usually requires large labeled data sets and takes a lot of time. In this paper, we analyze the influences of features on the performance of controllers trained using the convolutional neural networks (CNNs), which gives a guideline of feature selection to reduce computation cost. We collect a large set of data using The Open Racing Car Simulator (TORCS) and classify the image features into three categories (sky-related, roadside-related, and road-related features). We then design two experimental frameworks to investigate the importance of each single feature for training a CNN controller. The first framework uses the training data with all three features included to train a controller, which is then tested with data that has one feature removed to evaluate the feature's effects. The second framework is trained with the data that has one feature excluded, while all three features are included in the test data. Different driving scenarios are selected to test and analyze the trained controllers using the two experimental frameworks. The experiment results show that (1) the road-related features are indispensable for training the controller, (2) the roadside-related features are useful to improve the generalizability of the controller to scenarios with complicated roadside information, and (3) the sky-related features have limited contribution to train an end-to-end autonomous vehicle controller.

Evaluating the effectiveness and benefits of driver assistance systems is essential for improving the system performance. In this paper, we propose an efficient evaluation method for a semi-autonomous lane departure correction system. To achieve this, we apply a bounded Gaussian mixture model to describe drivers' stochastic lane departure behavior learned from naturalistic driving data, which can regenerate departure behaviors to evaluate the lane departure correction system. In the stochastic lane departure model, we conduct a dimension reduction to reduce the computation cost. Finally, to show the advantages of our proposed evaluation approach, we compare steering systems with and without lane departure assistance based on the stochastic lane departure model. The simulation results show that the proposed method can effectively evaluate the lane departure correction system.

Supervised learning approaches are widely used for driving style classification; however, they often require a large amount of labeled training data, which is usually scarce in a real-world setting. Moreover, it is time-consuming to manually label huge amounts of driving data due to uncertainties of driver behavior and variances among the data analysts. To address this problem, a semisupervised approach, a semisupervised support vector machine (S3VM), is employed to classify drivers into aggressive and normal styles based on a few labeled data points. First, a few data clusters are selected and manually labeled using a k-means clustering method. Then, a specific differentiable surrogate of a loss function is developed, which makes it feasible to use standard optimization tools to solve the nonconvex optimization problem. One of the most popular quasi-Newton algorithms is then used to assign the optimal label to all of the training data. Finally, we compare the S3VM method with a support vector machine method for classifying driving styles from different amounts of labeled data. Experiments show that the S3VM method can improve the classification accuracy by about 10% and reduce the labeling effort by using only a few labeled data clusters among huge amounts of unlabeled data.

To improve vehicle path-following performance and to reduce driver workload, a human-centered feed-forward control (HCFC) system for a vehicle steering system is proposed. To be specific, a novel dynamic control strategy for the steering ratio of vehicle steering systems that treats vehicle speed, lateral deviation, yaw error, and steering angle as the inputs and a driver's expected steering ratio as the output is developed. To determine the parameters of the proposed dynamic control strategy, drivers are classified into three types according to the level of sensitivity to errors, i.e., low, middle, and high. The proposed HCFC system offers a human-centered steering system (HCSS) with a tunable steering gain, which can assist drivers in tracking a given path with smaller steering wheel angles and change rate of the angle by adaptively adjusting steering ratio according to driver's path-following characteristics, reducing the driver's workload. A series of experiments of tracking the centerline of double lane change (DLC) are conducted in CarSim and three different types of drivers are subsequently selected to test in a portable driving simulator under a fixed-speed condition. The simulation and experiment results show that the proposed HCSS with the dynamic control strategy, as compared with the classical control strategy of steering ratio, can improve task performance by about 7% and reduce the driver's physical workload and mental workload by about 35% and 50%, respectively, when following the given path.