In human-machine (autonomous vehicles and robots) interaction scenarios, pedestrians often appear in groups. Pedestrian groups provide richer information compared to individuals, which helps address occlusion problems in pedestrian-machine interactions. However, the randomness and spatiotemporal complexity of pedestrian activity make pedestrian group activity recognition (PGAR) a highly challenging task. This article provides a detailed description of the PGAR task. For the first time, a definition of pedestrian group and activity for autonomous vehicles and robots is provided. Existing datasets and methods are systematically summarized. Furthermore, the unique challenges and trends in PGAR for autonomous vehicles and robots are outlined. Although some related surveys have been published, there has not yet been a survey specifically focused on PGAR in autonomous driving and robotics scenarios. Therefore, the goal of this article is to narrow the gap in this topic and provide a comprehensive reference for researchers in this field.

Accurate prediction of cyclist trajectories is essential for safe and reliable autonomous driving and intelligent transportation systems (ITSs) in complex traffic scenarios. To address the challenges posed by cyclists’ diverse intentions and non-linear motion patterns, we propose PhysiCycle, a novel multi-task learning framework that jointly predicts future trajectories and turning intentions. This framework integrates interpretable physical modeling with deep learning to enhance both prediction accuracy and behavioral consistency. Our model integrates a dual-path encoder to extract temporal motion cues and behavioral features, an intention classifier module, and a physically consistent decoder with bicycle kinematics consistency constraints. Experimental results on a real-world cyclist action dataset demonstrate that our method significantly outperforms baseline models in both intention classification and trajectory accuracy, achieving strong physical plausibility and generalization performance.

The application of autonomous vehicles (AVs) requires a safe and efficient decision-making approach for diverse and complex traffic environments. Most existing methodologies focus on specific scenarios or tasks, and are not sufficiently effective for real-world driving situations. This paper proposes a Safety Interval Reserve (SIR) model to quantify safety time margin, which is inspired by human driving behavior. Concurrently, a SIR network is developed to parametrically delineate the changes of SIR under dynamic traffic conditions. Consequently, a SIR network based decision-making approach (DMA-SIR) is designed to unify the macro path planning and micro behavior decision-making. The macro layer optimizes the global path with considering driving efficiency, while the micro layer generates local driving behavior to ensure safety during decision-making. The DMA-SIR facilitates multi-task management through a unified model and is grounded in motion mechanism. Besides, dataset validation demonstrates the anthropomorphic characteristics of DMA-SIR. As a result, it offers an interpretable method that effectively avoid the black box problem. Finally, extensive simulations experiments are conducted using 48 complex scenarios to verify the performance of DMA-SIR. The results show that DMA-SIR can efficiently handle multi-vehicle scenarios and generate driving trajectory with significant greater efficiency, safety and comfort compared to other methods.

Automated vehicles (AVs) are transforming urban transportation systems, as end-to-end autonomous driving models show great promise in enhancing traffic safety and operational efficiency. Despite these advances, their performance in highly interactive driving scenarios remains limited due to insufficient decision-making diversity and the absence of explicit safety guarantees. To address these challenges, we propose NavDrive, a safety-enhanced end-to-end autonomous driving framework that formulates planning as a multi-modal generative process.Specifically, NavDrive integrates navigation-based guidance into a diffusion policy. To focus on decision-critical information, a Decision-Aware Channel Fusion (DCF) module adaptively emphasizes regions involving key interactions between the ego vehicle and surrounding agents. Furthermore, a safety-aware generative planner refines trajectory samples toward feasible regions via the Target-Prior Diffusion Transformer (TDiT), which explicitly embeds physical constraints to ensure safe and human-aligned driving behaviors. Extensive experiments on the NAVSIM and nuScenes benchmarks demonstrate that NavDrive consistently outperforms existing baselines, delivering substantial gains in planning quality, safety, and robustness under complex and adverse conditions. The details will be available athttps://github.com/zgchongbo/NavDrive

A game-based cooperative steering control (GCSC) approach is introduced to facilitate effective collaboration between human drivers and automation, incorporating the neuromuscular delay inherent in human responses. In this framework, a dynamic coordination between driver and automation goals is achieved through the establishment of a game equilibrium in instances of driver and automation conflict. In response to the challenges posed by frequent modifications in driving weights and their subsequent burden on human drivers, this article proposes a strategy that integrates fixed initial weights with dynamic adjustments to driver–automation driving weights. Moreover, a comprehensive evaluation method including subjective and objective evaluation indexes is proposed. Different drivers are invited to perform virtual driving experiments, and the experimental results are analyzed by the proposed evaluation method. It is concluded that the driver’s driving weight should be kept at a high level during cooperative steering control when the driver’s intention cannot be perfectly obtained, and the determination of the driving weight should also consider the driver’s driving skills.

Trajectory planning and obstacle avoidance technologies for unmanned aerial vehicles (UAVs) are widely applied in Internet of Things-based intelligent urban management, data collection, and related fields, and are increasingly becoming a global research hotspot. However, uncertainties in trajectory planning caused by factors such as sensor measurement noise, model mismatch, and environmental disturbances can compromise the safety and robustness of UAV flights. While existing optimization-based methods build complex nonlinear models, they are often computationally expensive and inefficient. Learning-based methods, on the other hand, demand substantial computational resources. In this paper, we develop a nonlinear chance-constrained trajectory planning model that explicitly accounts for uncertainties, enabling autonomous obstacle avoidance and landing of UAVs on a dynamic platform. We derive the robust equivalent form of the chance constraints to address the solvability of models that include uncertainty factors. We develop a method that combines lossless convexification with the sequential convex programming (SCP) algorithm to achieve low complexity and high-efficiency solutions. Additionally, a real-time planning framework is proposed to address uncertain dynamic environments. We validate the robustness and safety of the proposed algorithm under various dynamic and uncertain scenarios, including different levels of disturbance, moving platforms, and unpredictable obstacles.

Virtual coupling (VC) is perceived to be promising in raising rail traffic capacity. In a train-to-train (T2T) based VC system, a communication network that ensures high quality of service (QoS) plays a critical role in enhancing both the coupling efficiency and the safety of the train platoon. However, unreliable communication environments characterized by issues such as time delays, packet loss, and network attacks present significant security risks to virtually coupled train sets (VCTS). How to cope with the impact caused by unstable communication and realize safe and stable VCTS formation are an important challenge for the VC system. In this paper, we propose a model predictive control (MPC) based distributed bidirectional control (DBC) strategy to tackle these challenges. We propose a control framework that integrates MPC with linear feedback-feedforward control to achieve real-time optimal control of the VC system, utilizing a bidirectional communication topology. To stabilize the VCTS, we derive local and string stability conditions to be satisfied by the controller parameters under asymmetric time-lagged unreliable networks, and utilize them as real-time constraints for the MPC controller. Furthermore, an analysis of the scalability of the proposed strategy has been conducted to improve its adaptability. Simulation results demonstrate that the proposed MPC-based DBC strategy significantly reduces the VCTS formation time and the maximum fluctuation of VCTS by 28.57% to 41.86%, and 28.84% to 52.10%, respectively, across various unreliable communication scenarios.

Vision-centric motion prediction concentrates on accurately determining the instance mask and its future trajectory from surround-view cameras, which manifests inherent merits such as holistic perspective and fully-differentiable spirit. Nonetheless, it is still impeded by sparse bird’s-eye view (BEV) representation and unfavorable temporal context across frames, resulting in a sub-optimal solution to decision-making and vehicle navigation. In this work, we propose a novelDifference-guideMotionPrediction for vision-centric autonomous driving, that is DMP, where it integrates BEV map refinement with spatial-temporal relation modeling in a hierarchical manner. Specifically, a bidirectional view projection strategy is introduced for the complementary BEV feature generation via depth-consistency correction. To promote spatiotemporal context aggregation, we design a difference-guided motion approach by offset approximation to align motion-aware cues between adjacent frames, and a dual-stream pyramid module is further developed for historical information fusion and future instance segmentation during specific durations. Extensive experiments on the large-scale nuScenes dataset demonstrate that it outperforms the baselines by a remarkable margin and delivers competitive motion prediction across diverse scenarios and range settings, suggesting its effectiveness and superiority. The details will be available athttps://github.com/pupu-chenyanyan/DMP-VAD.

The task of making lane change decisions for autonomous vehicles in mixed traffic is intricate and challenging due to the uncertainty of surrounding vehicles. The uncertainty exists in terms of the diverse social driving preferences and unpredictable driving behavior of human drivers. To address these challenges, the decision-making process for changing lanes is represented as an incomplete information game, where the driver characteristics of surrounding vehicles are unknown during the interaction. To eliminate the uncertainty of the driving environment, the concept of driver aggressiveness is proposed to quantify the social driving preferences based on the Risk-Response (R-R) diagram in an explainable manner. Then the predicted trajectory is utilized to calculate the driving risks using Gaussian Mixture Model (GMM) that is trained by the naturalistic driving data in the interactive lane change scenarios extracted from the highD dataset. To make the simulation environment more diverse and realistic, the data-driven motion model social Intelligent Driver Model (SIDM) is constructed based on car-following data obtained from cut-in scenarios in the highD dataset. The simulations are conducted by setting up the environment vehicles equipped with SIDM model with diverse social driving preferences. The findings indicate that the proposed decision-making model can recognize the category of surrounding vehicles, and in realistic interactive driving scenarios, it can produce adaptive and human-like driving decisions.

With increasing emphasis on vehicular automation and traffic efficiency, the management and coordination of platoon-based systems have become important. This research introduces a unique control framework based on a behavioral stability strategy, designed to enhance the cohesion of vehicle platoons and improve their ability to resist disturbances. Our approach integrates a vehicle scheduling system with a real-time platoon control mechanism to enhance the behavioral stability, robustness, and safety of the platoon. Given the heterogeneous nature of vehicles, we propose an optimal platoon formation model. This model strategically determines the number of platoons, arranges the sequence of vehicles within each platoon, and selects optimal cruising speeds to maximize platoon cohesion. To further enhance system robustness, a centralized robust model predictive controller is deployed for each platoon, ensuring stability against stochastic perturbations in vehicle dynamics and guaranteeing platoon safety. Finally, we conduct a simulation study involving multiple platoons with 20 heterogeneous vehicles to validate the effectiveness of the multi-layer optimization model.

Recently emerged vehicle-to-everything (V2X) perception has revealed great potential to overcome the limitation of single-vehicle intelligence aided by vigorous interaction among on-road agents, while prior endeavors are practically developed on parameter-specific simulation or configuration-dynamic real-world setting, overlooking the transferability across various scenarios. In this article, we propose unsupervised domain-adaptive vehicle-to-everything collaboration framework dubbed CUDA-X, which is built on top of a de facto collective model with key-point information exchange and instance adaptation. Specifically, collaborative knowledge transfer (CKT) is responsible for domain-agnostic feature reconstruction from nearby car or infrastructure by spatial-channel pooling operation in an elementwise manner. To promote the candidate alignment, a brand-new bin-based location correction (BLC) provides an auxiliary supervision for cross-dataset box refinement via residual coordinate encoding (RCE), and category-aware pooling alignment (CPA) is further designed for pulling the category-specific instance closer between source and target samples. We benchmark CUDA-X against the counterparts on four prevalent cooperative perception datasets, i.e., OPV2V, V2X-Sim, V2V4Real, and DAIR-V2X: it establishes the new state-of-the-art vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) performances regardless of simulation or reality. We expect that this appealing attempt would provide an in-depth insight into domain generalization in the context of multiagent perception, and the code is publicly available soon.

Articulating crane (AC) is used in various industrial activities. The articulated multisection arm exacerbates nonlinearities and uncertainties, making the precise tracking control challenging. This study proposes an adaptive prescribed performance tracking control (APPTC) for AC to robustly fulfill the task of precise tracking control, with adaptation to resist time-variant uncertainties, whose bounds are unknown but lie in prescribed fuzzy sets. Particularly, a state transformation is applied to simultaneously track the desired trajectory and satisfy the prescribed performance. Adopting the fuzzy set theory to describe uncertainties, APPTC does not invoke any IF-THEN fuzzy rules. There is no linearizations, or nonlinear cancelation for APPTC, thus making it approximation free. The performance of the controlled AC is twofold. First, deterministic performance in fulfilling the control task is ensured by the Lyapunov analysis using uniform boundedness and uniform ultimate boundedness. Second, fuzzy-based performance is further improved by an optimal design, which seeks the optima of control parameters by formulating a two-player Nash game. The existence of Nash equilibrium is theoretically proved, and its acquisition process is given. The simulation results are provided for validations. This is the first endeavor that explores the precise tracking control for fuzzy AC.

Multi-agent-based cooperation of autonomous vehicles(AVs) holds the potential to improve road safety, reduce emissions, and increase transport efficiency. However, the presence of uncertainties stemming from various sources poses a risk to the communication network and can alter the network topology, potentially causing instability in the multi-vehicle system. These uncertainties originate from two main sources: internal multi-vehicle system and external traffic environment. Time delays and packet losses contribute to uncertainties within the internal multi-vehicle system due to the uncontrollability of communication quality. Additionally, the dynamic nature of traffic environments introduces uncertainties related to the number of vehicles, interaction relationships, tasks, and destinations, thereby affecting communication resources and network topologies. Consequently, it is imperative to study the uncertainties faced by the multi-agent system and explore consensus methods for addressing these uncertainties. Notably, this study represents the first comprehensive review of consensus methods for both platooning and broader multi-agent cooperation in the presence of uncertain networks. Furthermore, a systematic summary of multi-agent consensus methods is presented, explicitly addressing two aspects of network uncertainty: imperfect communication transmission and the intricacies of traffic dynamics. The conclusion provides insights into open research issues, paving the way for future studies aimed at enhancing overall multi-vehicle system performance, including aspects such as convergence rate, robustness, and resilience.

For personalized human-machine cooperative (HMC) control, traditional model-driven approaches, which rely on predefined driver-vehicle-road (DVR) models, often struggle to adapt to individual driver differences. To address this, a data-learning shared control strategy based on game output regulation and adaptive dynamic programming (ADP) is presented. Firstly, considering the differences in driver’s characteristics, vehicle-road dynamics and human-machine interaction, an uncertain DVR system is established. Subsequently, robust output regulation (ROR) is utilized to handle road curvature perturbations and ensure closed-loop system stability. Subsequently, a dynamic game framework between the front-wheel steering system (AFS) and the active rear-wheel steering system (ARS) is further developed to ensure both vehicle stability and path-tracking accuracy in complex environments. Finally, the AFS-ARS optimal control strategies are iteratively learned and updated by ADP, using online DVR system data, without requiring prior knowledge of specific drivers or vehicles. Through driver-in-the-loop experiments, it is demonstrated that the presented method exhibits good adaptability to different drivers.

Virtual coupling is expected to primarily improve the capacity of a railway system. Virtual coupled systems are affected by multi-source disturbances due to the complex operating environment. However, existing research only partially considers the effects of structural or external disturbances, which limits the stability and robustness of the virtually coupled train set (VCTS). In this paper, we aim to tackle the challenges arising from both structural and external disturbances in virtual coupling. We specifically propose a distributed robust model predictive control (DRMPC) solution based on a linearized model by joining linear feedback and feedforward control into a model predictive control (MPC) framework with a discrete Kalman filter (DKF). We also theoretically derive and prove a set of sufficient conditions for both local and string stabilities under structural uncertainty. The stability conditions are incorporated into the constraint space of the distributed MPC framework in order to guarantee system stability in the presence of structural and external uncertainties. The simulation results validate that our proposed control method can stabilize train platooning under both structural and external disturbances. Our control method particularly reduces the spacing and velocity tracking errors by approximately 97.55% and 99.97% on average, respectively, as compared to several baselines.

Optimization-based methods are commonly applied in autonomous driving trajectory planners, which transform the continuous-time trajectory planning problem into a finite nonlinear program with constraints imposed at finite collocation points. However, potential violations between adjacent collocation points can occur. To address this issue thoroughly, we propose a safety-guaranteed collision-avoidance model to mitigate collision risks within optimization-based trajectory planners. This model introduces an “embodied footprint”, an enlarged representation of the vehicle’s nominal footprint. If the embodied footprints do not collide with obstacles at finite collocation points, then the ego vehicle’s nominal footprint is guaranteed to be collision-free at any of the infinite moments between adjacent collocation points. According to our theoretical analysis, we define the geometric size of an embodied footprint as a simple function of vehicle velocity and curvature. Particularly, we propose a trajectory optimizer with the embodied footprints that can theoretically set an appropriate number of collocation points prior to the optimization process. We conduct this research to enhance the foundation of optimization-based planners in robotics. Comparative simulations and field tests validate the completeness, solution speed, and solution quality of our proposal.

Perception systems play a crucial role in autonomous driving by reading the sensory data and providing meaningful interpretation of the operating environment for decision-making and planning. Guaranteeing a safe perception performance is the foundation for high-level autonomy, so that we can hand over the driving and monitoring tasks to the machine with ease. With the motivation of improving the perception systems’ safety, this survey analyzes and reviews the current achievements of safety-related standards and definitions, sensory modeling, and metrics for perception tasks in autonomous driving applications. Furthermore, it covers the generic categorization of potential failures and causal analysis in perception tasks, correlates the effect with the scenario modelling choices, and highlights major triumphs and noted limitations encountered by current research efforts. The new safety challenges laid out by the information exchange stage of the connected autonomous vehicle application have also been summarized. The open research questions and future directions are outlined to welcome researchers and practitioners to this exciting domain.

Well-trained visual object detectors are generally confronted with a severe performance decline when deployed in a novel driving scenario due to the impact of domain shift. Despite excellent improvements in unsupervised domain adaptive object detection achieved by adversarial training, those approaches fail to capture the transfer core underlying the holistic scenes. To solve this problem, we propose a progressive critical region transfer framework for cross-domain visual object detection. Specifically, we exploit a potential foreground mining (PFM) module and a semantic-specific RoI aggregation (SRA) module to improve the robustness of the cross-domain detection framework. Upon the critical regions in the broad sense, the PFM module first highlights the foreground regions by reweighting the hierarchical feature maps in sequence, and then modifies location biases at the downstream position of the backbone network for more accurate upstream predictions. Deep into the critical regions in the narrow sense, the SRA module concentrates on establishing an appropriate matching between batch-wise RoIs and all semantic centers, and further strengthens the aggregation of cross-domain identical semantic with the complement of context references. Together these modules are obligated to transform the adaptation importance from the whole scope to the latent foreground areas, and afterward to the informative regions of interest along the detection pipeline. Experiments show that our progressive critical region transfer framework achieves a state-of-the-art performance in adverse weather, camera configuration, and complicated scene adaptation, which outperforms the baselines by 19.4%, 5.0%, and 6.1%, respectively.

Reducing traffic accidents and associated casualties is a growing concern for modern human society. The secondary or even chain collisions for an unstable vehicle after an initial impact can result in more hazards and fatalities. Passive safety systems such as airbags and seat belts only provide limited level of protection for vehicle occupants, but cannot prevent collision accidents, while active safety systems usually work before the initial collision. Therefore, it is of great significance to develop dedicated post-impact stability control systems to help vehicles quickly restore stability to mitigate and/or avoid secondary collisions. However, the loss of original nonholonomic constraint property and the nonlinearity and saturation of tire forces due to post-impact sideslip, over-spinning, and drifting motions pose great challenges in controller design. Moreover, how to simulate and analyze the collision process and to further construct a simulation environment is the primary problem to solve for enabling controller development. Also, exploring repeatable, effective and low-cost experiment methods lays the foundation for controller verification. This paper aims to provide an overview of the latest technological advancements in collision modeling, control synthesis, and experimental procedures for post-impact stability control. The advantages and disadvantages of different modeling, control and experimental approaches are compared in succession. Finally, the paper discusses the challenges encountered in existing research and the prospects for post-impact active safety control systems.

Multi-vehicle cooperative perception has recently emerged for facilitating long-range and large-scale perception ability of connected automated vehicles (CAVs). Nonetheless, enormous efforts formulate collaborative perception as LiDAR-only 3D detection paradigm, neglecting the significance and complementary of dense image. In this work, we construct the first multi-modal vehicle-to-vehicle cooperative perception framework dubbed as V2VFormer++, where individual camera-LiDAR representation is incorporated with dynamic channel fusion (DCF) at bird’s-eye-view (BEV) space and ego-centric BEV maps from adjacent vehicles are aggregated by global-local transformer module. Specifically, channel-token mixer (CTM) with MLP design is developed to capture global response among neighboring CAVs, and position-aware fusion (PAF) further investigate the spatial correlation between each ego-networked map in a local perspective. In this manner, we could strategically determine which CAVs are desirable for collaboration and how to aggregate the foremost information from them. Quantitative and qualitative experiments are conducted on both publicly-available OPV2V and V2X-Sim 2.0 benchmarks, and our proposed V2VFormer++ reports the state-of-the-art cooperative perception performance, demonstrating its effectiveness and advancement. Moreover, ablation study and visualization analysis further suggest the strong robustness against diverse disturbances from real-world scenarios.

Recently, thanks to the introduction of human feedback, Chat Generative Pre-trained Transformer (ChatGPT) has achieved remarkable success in the language processing field. Analogically, human drivers are expected to have great potential in improving the performance of autonomous driving under real-world traffic. Therefore, this study proposes a novel framework for evolutionary decision-making and planning by developing a hybrid augmented intelligence (HAI) method to introduce human feedback into the learning process. In the framework, a decision-making scheme based on interactive reinforcement learning (Int-RL) is first developed. Specifically, a human driver evaluates the learning level of the ego vehicle in real-time and intervenes to assist the learning of the vehicle with a conditional sampling mechanism, which encourages the vehicle to pursue human preferences and punishes the bad experience of conflicts with the human. Then, the longitudinal and lateral motion planning tasks are performed utilizing model predictive control (MPC), respectively. The multiple constraints from the vehicle’s physical limitation and driving task requirements are elaborated. Finally, a safety guarantee mechanism is proposed to ensure the safety of the HAI system. Specifically, a safe driving envelope is established, and a safe exploration/exploitation logic based on the trial-and-error on the desired decision is designed. Simulation with a high-fidelity vehicle model is conducted, and results show the proposed framework can realize an efficient, reliable, and safe evolution to pursue higher traffic efficiency of the ego vehicle in both multi-lane and congested ramp scenarios.

Vehicular edge computing (VEC) is emerging as a promising architecture of vehicular networks (VNs) by deploying the cloud computing resources at the edge of the VNs. However, efficient resource management and task offloading in the VEC network is challenging. In this work, we first present a hierarchical framework that coordinates the heterogeneity among tasks and servers to improve the resource utilization for servers and service satisfaction for vehicles. Moreover, we formulate a joint resource allocation and task offloading problem (JRATOP), aiming to jointly optimize the intra-VEC server resource allocation and inter-VEC server load-balanced offloading by stimulating the horizontal and vertical collaboration among vehicles, VEC servers, and cloud server. Since the formulated JRATOP is NP-hard, we propose a cooperative resource allocation and task offloading algorithm named BARGAIN-MATCH, which consists of a bargaining-based incentive approach for intra-server resource allocation and a matching method-based horizontal-vertical collaboration approach for inter-server task offloading. Besides, BARGAIN-MATCH is proved to be stable, weak Pareto optimal, and polynomial complex. Simulation results demonstrate that the proposed approach achieves superior system utility and efficiency compared to the other methods, especially when the system workload is heavy.

In this article, a new dataset, the driver emotion facial expression (DEFE) dataset for drivers’ spontaneous emotions analysis is introduced. The dataset includes facial expression recordings from 60 participants during driving. After watching a selected video-audio clip to elicit a specific emotion, each participant completed the driving tasks in the same driving scenario and rated his/her emotional responses during the driving processes from the aspects of dimensional emotion method and discrete emotion method. The study also conducted classification experiments to recognize the scales of arousal, valence, dominance, as well as the emotion category and intensity to establish baseline results for the proposed dataset. Furthermore, this paper compared emotion recognition results difference through facial expressions between dynamic driving and static life scenarios. The results showed that dynamic driving and static life datasets were different in emotion recognition results. To further explore the reasons for the difference in emotion recognition results, the analysis from the AU (action unit) presence perspective was studied. The results showed significant differences in the AUs presence of facial expressions between dynamic driving and static life scenarios, indicating that drivers’ facial expressions may be affected by the driving task to influence the recognition of drivers’ emotions through facial expressions. Therefore, to accurately recognize the drivers’ emotions to establish a reliable emotion-aware human-machine interaction system, thereby improving driving safety and comfort, publishing a human emotion dataset specifically for the driver is necessary. The proposed dataset will be publicly available so that researchers worldwide can use it to develop and examine their driver emotion analysis methods. To the best of our knowledge, this is currently the only public driver facial expression dataset.

Detecting driver inattentive behaviors is crucial for driving safety in a driver monitoring system (DMS). Recent works treat driver distraction detection as a multiclass action recognition problem or a binary anomaly detection problem. The former approach aims to classify a fixed set of action classes. Although specific distraction classes can be predicted, this approach is inflexible to detect unknown driver anomalies. The latter approach mixes all distraction actions into one class: anomalous driving. Because the objective focuses on finding the difference between safe and distracted driving, this approach has better generalization in detecting unknown driver distractions. However, a detailed classification of the distraction is missing from the predictions, meaning that the downstream DMS can only treat all distractions with the same severity. In this work, we propose a two-phase anomaly proposal and classification framework [driver anomaly detection and classification network (DADCNet)] robust for open-set anomalies while maintaining high-level distraction understanding. DADCNet makes efficient allocation of multimodal and multiview inputs. The anomaly proposal network first utilizes a subset of the available modalities and views to suggest suspicious anomalous driving behavior. Then, the classification network employs more features to verify the anomaly proposal and classify the proposed distraction action. Through extensive experiments in two driver distraction datasets, our approach significantly reduces the total amount of computation during inference time while maintaining high anomaly detection sensitivity and robust performance in classifying common driver distractions.

Accurate forecasting of future traffic is a critical contemporary problem for transportation research. However, it is difficult to understand the feature patterns of traffic events due to the complexity of the traffic environment, heterogeneous factors, and lack of abnormal samples. This article proposes a framework to integrate the social traffic data and use the TabNet model to facilitate the representation learning task in traffic event prediction. With the tabular learning and model interpretability analysis, the importance of common traffic external factors toward traffic events is studied. The study has practical significance for regulating traffic planning and the development of the operational boundary for autonomous driving systems.

Remote sensing semantic segmentation plays a significant role in various applications such as environmental monitoring, land use planning, and disaster response. CNNs have been dominating remote sensing semantic segmentation. However, due to the limitations of convolution operations, CNNs cannot effectively model global context. The success of Transformers in the NLP domain provides a new solution for global context modeling. Inspired by Swin Transformer, we propose a novel remote sensing semantic segmentation model called CSTUNet. This model employs a dual-encoder structure consisting of a CNN-based main encoder and a Swin Transformer-based auxiliary encoder. We first utilize a detail-structure preservation module (DPM) to mitigate the loss of detail and structure information caused by Swin Transformer downsampling. Then we introduce a spatial feature enhancement module (SFE) to collect contextual information from different spatial dimensions. Finally, we construct a position-aware attention fusion module (PAFM) to fuse contextual and local information. Our proposed model obtained 70.75% MIoU on the ISPRS-Vaihingen dataset and 77.27% MIoU on the ISPRS-Potsdam dataset.

Multi-vehicle motion planning (MVMP) has become an emerging paradigm in connected and automated vehicles (CAVs). The cooperative lane-change movements with the coexistence of platoons and CAVs are typical scenarios on the muti-lane roads. This paper proposes an optimal control framework with the advantages of completeness and universality for platoons and CAVs’ cooperative lane-change motion planning in different task scenarios. Two typical cooperative scenarios are designed for the subsequent study of optimal modeling. The platoons’ reconfiguration and original shape maintenance are considered to reflect the universality of moving objects and the diversity of cooperative tasks. Approximately geometric contour models, dynamic externally tangent rectangle and inflated rectangle, are utilized to describe the platoon’s profile. Analytical complete collision avoidance constraints among different motion objects are constructed effectively. Other necessary constraints and the weighted cost function that minimizes lane-change time and motion energy are comprehensively considered. The optimal control models are established for the desired scenarios. Moreover, a numerical solution method combined with the simultaneous direct collocation method based on the trapezoidal rule and the barrier function method is proposed to obtain the optimal schemes. Simulation and contrast experiments are conducted for two scenarios. The results indicate that the cost function’s weight coefficients and specific lane-change tasks influence the cooperative motion planning effects and verify that the proposed optimal control framework is of reasonability, effectiveness, and unification.

Benefiting from the advancement of intelligent transportation systems (ITSs), intelligent connected vehicles (ICVs) are ushering in a once-in-a-generation development opportunity. Considering the widespread presence of heterogeneous vehicles with disturbances and uncertain dynamics in actual platoon scenarios as well as the multimodel switching produced by unavoidable interruptions in the communication process, this paper proposes a data–mechanism adaptive switched predictive (DASP) control strategy. The characteristics of the mechanism model are mapped based on state data to more accurately describe the system’s dynamic characteristics and improve the interpretability of variables. The introduction of Givens rotations and switching criteria enables online adaptive switching of the controller. A robustness analysis of heterogeneous platoon switching control under bounded disturbance is presented, and sufficient conditions for$\mathcal {L}_{2}$string stability are provided. Finally, CarSim simulations and real-time bench experiments are reported to demonstrate the effectiveness of the DASP algorithm for heterogeneous multivehicle regulation with communication interruptions.

Recent advances on road damage detection relies on a large amount of labeled data, whilst collecting pavement image is labor-intensive and time-consuming. Unsupervised Domain Adaptation (UDA) provides a promising solution to adapt a source domain to the target domain, however, cross-domain crack detection is still an open problem. In this paper, we propose domain adaptive road damage detection termed as DA-RDD, by incorporating image-level with instance-level feature alignment for domain-invariant representation learning in an adversarial manner. Specifically, importance weighting is introduced to evaluate the intermediate samples for image-level alignment between domains, and we aggregate RoI-wise feature with multi-scale contextual information to recover the crack details for progressive domain alignment at instance level. Additionally, a large-scale road damage dataset (based on Road Damage Dataset 2020 (RDD2020)) named as RDD2021 is constructed with$100k$synthetic labeled distress images. Extensive experimental results on damage detection across different countries demonstrate the universality and superiority of DA-RDD, and empirical studies on RDD2021 further claim its effectiveness and advancement. To our best knowledge, it is the first time to investigate domain adaptative pavement crack detection, and we expect the contributions in this work would facilitate the development of generalized road damage detection in the future.

Cooperative on-ramp merging control for connected and automated vehicles (CAVs) can effectively improve traffic throughput and vehicle fuel efficiency at highway on-ramp merging bottlenecks. However, in the mixed traffic scenario where CAVs and human-driven vehicles (HDVs) coexist, the uncertain maneuvers of human drivers pose a major challenge to merging control in terms of safety and flexibility. To this end, this paper proposes a hierarchical cooperative on-ramp merging control strategy for CAVs to optimize flexible trajectories with safety guarantees in mixed traffic. First, the on-ramp merging control problem for CAVs is considered in the case of a three-vehicle coordination, resulting in an optimal control problem (OCP) coordinating on-ramp and main-lane CAVs for efficient operation while satisfying multiple safety-critical constraints. Second, a two-level hierarchical control architecture is developed to solve the OCP with mixed state-control constraints. The upper-level planner solves an unconstrained OCP with Pontryagin’s Minimum Principle to calculate an expected merging position, which is embedded in the variable time headway of safe merging constraints in the lower-level controller. Then, the controller converts the nonlinear OCP with safety-critical constraints to a quadratic programing (QP) problem by exploiting Control Barrier Functions (CBFs) and Control Lyapunov Functions (CLFs). By solving the QP efficiently, the time and energy efficient trajectory for each CAV is obtained. In addition, a receding horizon control framework is employed, which enables CAVs to determine flexible merging opportunity and tackle the disturbances caused by HDVs. Finally, comprehensive simulation results show that the proposed cooperative on-ramp merging strategy has potential in enabling merging flexibility, improving traffic efficiency and energy economy in real time.

Reinforcement learning (RL) has been widely used in the decision-making of autonomous vehicles (AVs) in recent studies. However, existing RL methods generally find the optimal policy by maximizing the expectation of future returns, which lacks distributional treatments of risky situations. Additionally, various uncertainties arising from the environment could also cause unreliable decisions, particularly in some complex urban environments. In this paper, the fully parameterized quantile network (FPQN) is utilized to estimate the full return distribution. Then, the conditional value-at-risk (CVaR) is utilized with the return distribution information to generate uncertainty-aware driving behavior. Additionally, an uncontrolled four-way intersection is developed by the Simulation of Urban Mobility (SUMO) simulation platform, which considers both the surrounding vehicles (SVs) and pedestrians. More specifically, to simulate the real-world traffic environment, the uncertainty arising from the occlusion, and the behavior uncertainty of surrounding traffic participants are also considered. The experiment results suggest that the proposed method outperforms the baseline methods in terms of safety. Furthermore, the results also indicate that the proposed method can make reasonable decisions in some challenging driving cases in the presence of uncertainty.

Autonomous driving is considered one of the revolutionary technologies shaping humanity’s future mobility and quality of life. However, safety remains a critical hurdle in the way of commercialization and widespread deployment of autonomous vehicles on public roads. Safety concerns require the autonomous driving system to handle uncertainties from multiple sources that are either preexisting, e.g., the stochastic behavior of traffic participants or scenario occlusion, or introduced as a result of processing, e.g., the application of neural networks. Thus, it is crucial to analyze the sources of uncertainties and quantify the risks associated with them, including the propagated risks that accumulate in the decision-making system. In this context, this paper provides an overview of uncertainty challenges and state-of-the-art techniques for mitigating these challenges. We argue that the uncertainties mainly originate from two aspects: 1) the external traffic environment, and 2) the internal autonomous driving system. Specifically, this paper first analyzes the safety challenges caused by the uncertainties and summarizes their sources. In addition, the corresponding techniques that mitigate and quantify the risk of uncertainties are presented. Finally, research perspectives are highlighted to facilitate future studies for guaranteeing the safety of autonomous vehicles.

Recent advances in cross-modal 3D object detection rely heavily on anchor-based methods, and however, intractable anchor parameter tuning and computationally expensive postprocessing severely impede an embedded system application, such as autonomous driving. In this work, we develop an anchor-free architecture for efficient camera-light detection and ranging (LiDAR) 3D object detection. To highlight the effect of foreground information from different modalities, we propose a dynamic fusion module (DFM) to adaptively interact images with point features via learnable filters. In addition, the 3D distance intersection-over-union (3D-DIoU) loss is explicitly formulated as a supervision signal for 3D-oriented box regression and optimization. We integrate these components into an end-to-end multimodal 3D detector termed 3D-DFM. Comprehensive experimental results on the widely used KITTI dataset demonstrate the superiority and universality of 3D-DFM architecture, with competitive detection accuracy and real-time inference speed. To the best of our knowledge, this is the first work that incorporates an anchor-free pipeline with multimodal 3D object detection.

Interest in autonomous driving (AD) and intelligent vehicles (IVs) is growing at a rapid pace due to the convenience, safety, and economic benefits. Although a number of surveys have reviewed research achievements in this field, they are still limited in specific tasks and lack systematic summaries and research directions in the future. Our work is divided into three independent articles and the first part is a survey of surveys (SoS) for total technologies of AD and IVs that involves the history, summarizes the milestones, and provides the perspectives, ethics, and future research directions. This is the second part (Part I for this technical survey) to review the development of control, computing system design, communication, high-definition map (HD map), testing, and human behaviors in IVs. In addition, the third part (Part II for this technical survey) is to review the perception and planning sections. The objective of this article is to involve all the sections of AD, summarize the latest technical milestones, and guide abecedarians to quickly understand the development of AD and IVs. Combining the SoS and Part II, we anticipate that this work will bring novel and diverse insights to researchers and abecedarians, and serve as a bridge between past and future.

A growing interest in autonomous driving (AD) and intelligent vehicles (IVs) is fueled by their promise for enhanced safety, efficiency, and economic benefits. While previous surveys have captured progress in this field, a comprehensive and forward-looking summary is needed. Our work fills this gap through three distinct articles. The first part, a “survey of surveys” (SoS), outlines the history, surveys, ethics, and future directions of AD and IV technologies. The second part, “Milestones in AD and IVs Part I: Control, Computing System Design, Communication, high-definition map (HD map), Testing, and Human Behaviors” delves into the development of control, computing system, communication, HD map, testing, and human behaviors in IVs. This part, the third part, reviews perception and planning in the context of IVs. Aiming to provide a comprehensive overview of the latest advancements in AD and IVs, this work caters to both newcomers and seasoned researchers. By integrating the SoS and Part I, we offer unique insights and strive to serve as a bridge between past achievements and future possibilities in this dynamic field.

Effective road maintenance can not only achieve a balance between limited resources and long-term high-efficiency performance of road but also reduce the loss of life and property caused by road damage to vehicles and pedestrians. Due to the lack of a multidimensional dynamic monitoring system and enough extremely special data, the existing road maintenance system cannot accurately assess the road surface condition and provide timely early warning of sudden road damage. In this article, the M-RM system is proposed, that is, a metaverse-enabled road maintenance system based on cyber–physical–social systems (CPSSs), which fully utilizes the social and artificial system information of CPSS, as well as the simulation, monitoring, diagnosis and prediction functions of road systems in the virtual world of the metaverse. Then, in the road damage detection of system model in the virtual world, for the virtual data of the core assets of the metaverse, we propose an adaptive and information-preserving data augmentation (AIDA) algorithm-based nonclassical receptive field suppression and enhancement, an algorithm developed from human visual cognition. This algorithm enables the generation of a large amount of scarce fidelity data and avoids the introduced noise from impairing the performance of nonaugmented data. Finally, a crack detection algorithm named pay attention twice (PAT) is proposed, which uses the generated virtual data for training, and achieves secondary attention to high-frequency targets by fusing frequency-division convolution and mixed-domain attention mechanism. The detection performance of small targets in uncertain environments is enhanced. The metaverse system built in the current research can not only be used for road maintenance but also empower the traffic metaverse by using the traffic flow prediction module embedded in the algorithm. Experimental results demonstrate that the proposed algorithm can be applied to the road damage detection task under different noise and weather conditions, and the performance outweighs other state-of-the-art algorithms.

Replacing traffic signals with roadside vehicle-to-infrastructure systems in the era of connected and autonomous vehicles (CAVs) is promising. Managing CAVs in a signal-free intersection, known as autonomous intersection management (AIM), controls the driving behavior of each intersection-traverse CAV to maximize the throughput. Although AIM improves the gross throughput, the fairness of each individual vehicle in its right of way is not seriously considered. This study sets up an AIM system in the cyber–physical–social space to trade traverse priorities quantitatively and fairly. To that end, one needs an AIM method that is optimal and stable, otherwise no convincing trades of traverse priorities could be made. This study proposes a near-optimal lane-free AIM method based on numerical optimal control, wherein log-exp functions are deployed to convexify nondifferentiable collision-avoidance constraints. Besides that, a parameterized social force model (SFM) is proposed to provide a tunable initial guess for numerical optimal control. By tuning the urgency weights in SFM, one may get cooperative trajectories in different homotopy classes, which are further utilized to decide the amount of virtual currency to reward those CAVs who tend to share their traverse priorities. The overall method improves the traverse throughput with individual fairness respected. In experiencing this system, passengers learn how to behave with politeness when they drive manually. Experiments show the efficiency and robustness of the AIM method and also show the efficacy of the overall priority-sharing system.

Advances in technologies, such as intelligent connected vehicles and the metaverse are driving the rapid development of automotive intelligent cockpits. From the perspective of the cyber–physical–social system (CPSS), this study proposed the intelligent cockpit composition framework which includes three layers of perception, cognition and decision, and interaction. Meanwhile, we also describe the relationship between the intelligent cockpit framework and the outside environment. The framework can dynamically perceive and understand humans, and provide feedback on the understanding results, which is beneficial to provide a safe, efficient, and enjoyable experience for humans in the intelligent cockpit. In the cognition and decision layers of the proposed framework, we design a case study of active empathetic auditory regulation of driver anger, focusing on improving road traffic safety. We conducted an in-depth interview experiment and designed two auditory regulation materials of active empathy speech and text-to-speech (TTS) speech. Next, 30 participants were recruited, and they completed a total of 240 anger-regulated driving experiments in the straight and obstacle avoidance scenarios. Finally, we quantitatively analyzed and compared the participants’ subjective feelings, physiological changes, driving behaviors, and driving risks, as well as validated the driver anger regulation quality of AES and TTS. The proposed research methods results are beneficial to the design of future intelligent cockpit emotion regulation systems, toward a better intelligent cockpit.

In intelligent vehicle cooperative systems, the mismatch in driving characteristics between a human and a machine and the driver misoperation caused by this mismatch result in human–machine conflicts, which significantly affect driving safety. Therefore, an intelligent vehicle human–machine cooperative steering torque control method is proposed herein. To adapt the intelligent system to the varying previewing characteristics of a human, a time-varying previewing driver model is constructed, and a penalty factor for human–machine intervention is designed based on fuzzy rules to assign driving control rights by assessing the driver’s state. Consequently, a human–vehicle–road model with driver preview time and penalty factor as varying parameters is established. Based on gain-scheduling control, a human–machine cooperative steering torque controller is designed to adapt to the varying previewing characteristics of a human and the change in human–machine intervention. The stability and robustness of the entire parameter space are guaranteed by constraining the poles in a certain region. Finally, the proposed human–machine cooperative control scheme demonstrates the effective alleviation of conflicts between the driver and the intelligent driving system.

In ice and snow-rutted conditions, it is challenging to design a vehicle stability controller to simultaneously resolve the conflict between the accuracy of the system model and the easy implementation of the controller. To this end, the application of a data-driven control method for vehicle stability control represents a novel, feasible opportunity. This article introduces Givens rotation and forgetting factors to efficiently update the subspace prediction equation with online data. An online data-driven predictive control (ODPC) method is proposed on this basis. To address the problem that persistently excited (PE) condition will cause fluctuations in the steady-state response of ODPC, a data-driven switched predictive control strategy (DSPCS) employing attenuated excitation (AE) signals and hysteresis comparisons based on posterior prediction errors is proposed. In addition, an implementation method involving the Laguerre function (LF) parameterization of the control input is proposed to improve the computational efficiency further. Numerical simulation results show that both the ODPC method and the DSPCS can effectively track given yaw rate and sideslip angle reference under the influence of ruts. Furthermore, the DSPCS can effectively reduce steady-state response fluctuations. In addition, the LF parameterization is superior regarding computational time.

Autonomous driving datasets have played an important role in validating the advancement of intelligent vehicle algorithms including localization, perception and prediction in academic areas. However, current existing datasets pay more attention to the structured urban road, which hampers the exploration on unstructured special scenarios. Moreover, the open-pit mine is one of the typical representatives for them. Therefore, we introduce the Autonomous driving dataset on the Mining scene (AutoMine) for positioning and perception tasks in this paper. The AutoMine is collected by multiple acquisition platforms including an SUV, a wide-body mining truck and an ordinary mining truck, depending on the actual mine operation scenarios. The dataset consists of 18+ driving hours, 18K annotated lidar and image frames for 3D perception with various mines, time-of-the-day and weather conditions. The main contributions of the AutoMine dataset are as follows: I.The first autonomous driving dataset for perception and localization in mine scenarios. 2.There are abundant dynamic obstacles of 9 degrees of freedom with large dimension difference (mining trucks and pedestrians) and extreme climatic conditions (the dust and snow) in the mining area. 3.Multi-platform acquisition strategies could capture mining data from multiple perspectives that fit the actual operation. More details can be found in our website(https://automine.cc).

As a critical component of the Internet of Things, connected automated vehicles (CAVs) are progressively gaining attention for their benefits in terms of increased safety and reduced traffic congestion. In this article, a novel distributed data-driven model-predictive control (DDMPC) approach including feedforward for disturbance is proposed for cruise control of a hybrid platoon with a combination of human-operated and autonomous vehicles. By employing a predictor constructed from input/output data, predictive controllers are obtained without depending on the characteristic information of the system. A robustness analysis is performed with a combination of the input-to-state stability (ISS) theory with the sampled-data systems theory, and the$\mathcal {L}_{2}$-norm string stability is ensured by strict mathematical proof. In addition, we also discuss the asymptotic stability when the controller switches. CarSim simulation and bench experiment results verify that the DDMPC for connected vehicles can be robust to velocity disturbances and achieve satisfactory performance in ensuring string stability.

Distracted driving has been recognized as a major challenge to traffic safety improvement. This article presents a novel driving distraction detection method that is based on a new deep network. Unlike traditional methods, the proposed method uses both temporal information and spatial information of electroencephalography (EEG) signals as model inputs. Convolutional techniques and gated recurrent units were adopted to map the relationship between drivers’ distraction status and EEG signals in the time domain. A driving simulation experiment was conducted to examine the effectiveness of the proposed method. Twenty-four healthy volunteers participated and three types of secondary tasks (i.e., cellphone operation task, clock task, and 2-back task) were used to induce distraction during driving. Drivers’ EEG responses were measured using a 32-channel electrode cap, and the EEG signals were preprocessed to remove artifacts and then split into short EEG sequences. The proposed deep-network-based distraction detection method was trained and tested on the collected EEG data. To evaluate its effectiveness, it was also compared with the networks using temporal or spatial information alone. The results showed that our proposed distraction detection method achieved an overall binary (distraction versus nondistraction) classification accuracy of 0.92. In terms of task-specific distraction detection, its accuracy was 0.88. Further analysis on the individual difference in detection performance showed that drivers’ EEG performance differed across individuals, which suggests that adaptive learning for each individual driver would be needed when developing in-vehicle distraction detection applications. Note to Practitioners—Driver distraction detection is crucial for safety enhancement to avoid crashes caused by nondriving-related activities, such as calling and texting while driving. Related previous studies mainly focus on detection by monitoring head and eye movement using computer vision technologies or by extracting indicators from driving performance measures for driver state inference. However, complex traffic environments (e.g., dynamically changing light distribution on driver’s face and nighttime driving with low illumination) strongly limit the effectiveness of computer vision technologies, and the driving performance characteristics may also be caused by factors other than distraction (e.g., fatigue). To solve these problems, this article seeks to develop a deep learning-based approach to map the unique relationship between driver distraction and the bioelectric electroencephalography (EEG) signals that are not affected by traffic environments. The proposed method can be integrated into the driver assistance systems and autonomous vehicles to deal with emergency situations that need drivers to handle. The timely detection of distraction by our method will significantly facilitate its practical applications in collision avoidance or danger mitigation in the handover process.

Accurate trajectory prediction of surrounding vehicles enables lower risk path planning in advance for autonomous vehicles, thus promising the safety of automated driving. A low-risk and high-efficiency path planning approach is proposed for autonomous driving based on the high-performance and practical trajectory prediction method. A long short-term memory (LSTM) network is trained and tested using the highD dataset, and the validated LSTM is used to predict the trajectories of surrounding vehicles combining the information extracted from vehicle-to-vehicle (V2V) technology. A risk assessment and mitigation-based local path planning algorithm is proposed according to the information of predicted trajectories of surrounding vehicles. Two driving scenarios are extracted and reconstructed from the highD dataset for validation and evaluation, i.e., an active lane-change scenario and a longitudinal collision-avoidance scenario. The results illustrate that the risk is mitigated and the driving efficiency is improved with the proposed path planning algorithm comparing to the constant-velocity prediction and the prediction method of the nonlinear input–output (NIO) network, especially when the velocity and trajectory with sudden changes. Note to Practitioners—This article was motivated by the problem of promising the safety decision-making and path planning through accurate environment prediction. There are two main parts included in this article. First, this article proposed one pragmatic approach to predict the environment movement correctly based on the long short-term memory (LSTM) approach. The prediction performance of LSTM was compared with nonlinear input–output (NIO). The results showed that the LSTM approach has a significant advantage in motivation prediction of the surrounded vehicles during path planning. The second part of this article is to make the decision and realize local path planning based on the risk assessment. The potential field-based approach is implemented on the risk assessment based on these accurate predictions. Some primary results demonstrate that the decision-making algorithm performs better under the accurate prediction model. The results also show that the safety and driving efficiency of the ego vehicle were improved by tracking the trajectory, which was planned based on the risk assessment. The only concern for the real-time application is the computation time; in future, we will figure it out how to further reduce the computation time.

Driver’s emotion recognition is vital to improving driving safety, comfort, and acceptance of intelligent vehicles. This article presents a cognitive-feature-augmented driver emotion detection method that is based on emotional cognitive process theory and deep networks. Different from the traditional methods, both the driver’s facial expression and cognitive process characteristics (age, gender, and driving age) were used as the inputs of the proposed model. Convolutional techniques were adopted to construct the model for driver’s emotion detection simultaneously considering the driver’s facial expression and cognitive process characteristics. A driver’s emotion data collection was carried out to validate the performance of the proposed method. The collected dataset consists of 40 drivers’ frontal facial videos, their cognitive process characteristics, and self-reported assessments of driver emotions. Another two deep networks were also used to compare recognition performance. The results prove that the proposed method can achieve well detection results for different databases on the discrete emotion model and dimensional emotion model, respectively.

Autonomous vehicles were experiencing rapid development in the past few years. However, achieving full autonomy is not a trivial task, due to the nature of the complex and dynamic driving environment. Therefore, autonomous vehicles are equipped with a suite of different sensors to ensure robust, accurate environmental perception. In particular, the camera-LiDAR fusion is becoming an emerging research theme. However, so far there has been no critical review that focuses on deep-learning-based camera-LiDAR fusion methods. To bridge this gap and motivate future research, this article devotes to review recent deep-learning-based data fusion approaches that leverage both image and point cloud. This review gives a brief overview of deep learning on image and point cloud data processing. Followed by in-depth reviews of camera-LiDAR fusion methods in depth completion, object detection, semantic segmentation, tracking and online cross-sensor calibration, which are organized based on their respective fusion levels. Furthermore, we compare these methods on publicly available datasets. Finally, we identified gaps and over-looked challenges between current academic researches and real-world applications. Based on these observations, we provide our insights and point out promising research directions.

This paper proposes a novel driver-automation shared control based on a potential-field-driven model predictive controller (PF-MPC) and a two-layer fuzzy strategy (TLFS) to address driver-automation conflicts and control authority allocation issues. The PF-MPC approach based on the driver-vehicle model is introduced to deal with obstacles avoidance and driver-automation conflicts. The potential field is constructed to evaluate the driving risk by considering the driving environment and vehicle states, meanwhile, it is also involved in the optimized objective in the PF-MPC controller for obstacle avoidance. The tuning weight is designed to adjust the trade-off between the motion planning-related cost and driver-related cost to reduce driver-automation conflicts. To further alleviate the conflict and control authority allocation between the human driver and PF-MPC controller, the TLFS for shared control is designed based on the evaluation of the driving risk level and conflict situation, and the values of the tuning weight and cooperative coefficient are determined using the fuzzy control method. Moreover, comparative studies are conducted to verify the driving safety and conflict management performance of the proposed shared control method on a straight road and a curvy road. The results show that the proposed shared control method can help drivers avoid obstacles safely and alleviate the driver-automation conflicts in different driving conditions.

The intersection is a critical traffic problem from the perspective of safety and traffic efficiency. As wireless communication technology advances, vehicle infrastructure cooperative approaches have received increased attention. In this paper, we propose a cooperative critical turning point method to help the cooperation between vehicles and infrastructures to improve the traffic efficiencies. The idea of cooperative critical turning point improves the cooperation between connected automated vehicles, the surrounding traffic and roadside infrastructures in order to provide high efficiencies of the intersection. An intersection management system using such a method is implemented based on the framework of the connected automated vehicle highway system. Such system can efficiently allow a roadside infrastructure receives state information from vehicles, reserve the associated intersection time-space occupancy, and then provide decision-making and planning feedback to the vehicles. The vehicles covered by the system then adjust their trajectories to meet their assigned time slot. The study validates the proposed system that considers the uncertainties of the driving environment by formulating the problem into a POMDP problem and solves it using an online solver. Based on preliminary simulation experiments, the proposed strategy can significantly reduce travel delays, decrease stops and improve the sustainability of the traffic system.

Camera-LiDAR 3D object detection has been extensively investigated due to its significance for many real-world applications. However, there are still of great challenges to address the intrinsic data difference and perform accurate feature fusion among two modalities. To these ends, we propose a two-stream architecture termed as CL3D, that integrates with point enhancement module, point-guided fusion module with IoU-aware head for cross-modal 3D object detection. Specifically, pseudo LiDAR is firstly generated from RGB image, and point enhancement module (PEM) is then designed to enhance the raw LiDAR with pseudo point. Moreover, point-guided fusion module (PFM) is developed to find image-point correspondence at different resolutions, and incorporate semantic with geometric features in a point-wise manner. We also investigate the inconsistency between localization confidence and classification score in 3D detection, and introduce IoU-aware prediction head (IoU Head) for accurate box regression. Comprehensive experiments are conducted on publicly available KITTI dataset, and CL3D reports the outstanding detection performance compared to both single- and multi-modal 3D detectors, demonstrating its effectiveness and competitiveness.

Driver model adaptation (DMA) plays an essential role for driving behaviour modelling when there is a lack of sufficient data for training the new model. A new data-driven DMA method is proposed in this paper to realise the instance-level knowledge transfer between individual drivers. Using the importance-weighted transfer learning (IWTL), the data collected from one driver (source driver) can be directly used to train the model of another driver (target driver). Under the framework of IWTL, the relationship between two different drivers can be modelled by the importance weight (IW). Two estimation methods Kullback-Leibler (KL) Divergence and least-squares (LS), are used to estimate IW for each data instance by modelling the importance-weight function as a radial basis function (RBF). Experiments based on the driving simulator and real vehicle are carried out to test the performance of TL for steering behaviour adaptation during the overtaking manoeuvre. The experimental results show that the TL method can transfer the knowledge observed from one driver to another when training the new driver model without sufficient data by keeping the modelling error at a low level.

Object detection is becoming increasingly significant for autonomous-driving system. However, poor accuracy or low inference performance limits current object detectors in applying to autonomous driving. In this work, a fast and accurate object detector termed as SA-YOLOv3, is proposed by introducing dilated convolution and self-attention module (SAM) into the architecture of YOLOv3. Furthermore, loss function based on GIoU and focal loss is reconstructed to further optimize detection performance. With an input size of$512\times 512$, our proposed SA-YOLOv3 improves YOLOv3 by 2.58 mAP and 2.63 mAP on KITTI and BDD100K benchmarks, with real-time inference (more than 40 FPS). When compared with other state-of-the-art detectors, it reports better trade-off in terms of detection accuracy and speed, indicating the suitability for autonomous-driving application. To our best knowledge, it is the first method that incorporates YOLOv3 with attention mechanism, and we expect this work would guide for autonomous-driving research in the future.

Accurate and fast 3D object detection from point clouds is a key task in autonomous driving. Existing one-stage 3D object detection methods can achieve real-time performance, however, they are dominated by anchor-based detectors which are inefficient and require additional post-processing. In this paper, we eliminate anchors and model an object as a single point—the center point of its bounding box. Based on the center point, we propose an anchor-free CenterNet3D network that performs 3D object detection without anchors. Our CenterNet3D uses keypoint estimation to find center points and directly regresses 3D bounding boxes. However, because inherent sparsity of point clouds, 3D object center points are likely to be in empty space which makes it difficult to estimate accurate boundaries. To solve this issue, we propose an extra corner attention module to enforce the CNN backbone to pay more attention to object boundaries. Besides, considering that one-stage detectors suffer from the discordance between the predicted bounding boxes and corresponding classification confidences, we develop an efficient keypoint-sensitive warping operation to align the confidences to the predicted bounding boxes. Our proposed CenterNet3D is non-maximum suppression free which makes it more efficient and simpler. We evaluate CenterNet3D on the widely used KITTI dataset and more challenging nuScenes dataset. Our method outperforms all state-of-the-art anchor-based one-stage methods and has comparable performance to two-stage methods as well. It has an inference speed of 20 FPS and achieves the best speed and accuracy trade-off. Our source code will be released athttps://github.com/wangguojun2018/CenterNet3d.

To improve the accuracy and efficiency of 3D LiDAR mapping, real-time cooperative SLAM has been considered to explore large and complex areas. To merge the individual maps from multiple robots, it is crucial to identify the common areas and obtain alternative matches between them. However, data transmission, especially in sparse networks with narrow bandwidth and limited range, is a challenging issue for the above problem. Since the distribution manner is suitable for limited communication, we proposed a common framework of 3D real-time distributed cooperative SLAM to fill the community gap. Assuming that each robot can communicate with others, the presented framework consists of four key modules: place recognition, relative pose estimation, distributed graph optimization, and communication. Meanwhile, we developed a complete real-time distributed cooperative SLAM system, called RDC-SLAM, by integrating state-of-the-art components into the framework. For computation and data transmission efficiency, descriptor-based registration is used instead of the conventional point cloud matching. An intensity-based descriptor is developed to perform the place recognition and obtain the alternative matches, while an eigenvalue-based segment descriptor is applied to further refine the relative pose estimations between these alternative matches. A distributed graph optimization method is utilized to obtain the maximum likelihood of multi-trajectory estimation. A communication protocol is also designed to associate data among robots that are easy to deploy and have low network requirements. The RDC-SLAM is validated by real-world experiments and exhibits superior performance concerning accuracy, computation efficiency, and data efficiency.

Platoon-based vehicular cyber-physical systems have gained increasing attention due to their potentials in improving traffic efficiency, capacity, and saving energy. However, external uncertain disturbances arising from mismatched model errors, sensor noises, communication delays and unknown environments can impose a great challenge on the constrained control of vehicle platooning. In this paper, we propose a closed-loop min-max model predictive control (MPC) with causal disturbance feedback for vehicle platooning. Specifically, we first develop a compact form of a centralized vehicle platooning model subject to external disturbances, which also incorporates the lower-level vehicle dynamics. We then formulate the uncertain optimal control of the vehicle platoon as a worst-case constrained optimization problem and derive its robust counterpart by semidefinite relaxation. Thus, we design a causal disturbance feedback structure with the robust counterpart, which leads to a closed-loop min-max MPC platoon control solution. Even though the min-max MPC follows a centralized paradigm, its robust counterpart can keep the convexity and enable the efficient and practical implementation of current convex optimization techniques. We also derive a linear matrix inequality (LMI) condition for guaranteeing the recursive feasibility and input-to-state practical stability (ISpS) of the platoon system. Finally, simulation results are provided to verify the effectiveness and advantage of the proposed MPC in terms of constraint satisfaction, platoon stability and robustness against different external disturbances.

This article presents a framework to determine driving style and design a driver steering model considering driver characteristics. First, principal component analysis (PCA) and$K$-means clustering are utilized to classify 30 participants into cautious, moderate, and aggressive drivers. Subsequently, a generic steering model is established based on the model predictive control method. Thereafter, the maximum lateral acceleration is extracted as a crucial indicator to represent driver characteristics, and it is calibrated through probabilistic models using the dataset, which consists of the classified drivers. Besides, point estimation model and interval estimation model are leveraged to determine driving style and adjust constraints in the stochastic programming-based steering model. Finally, simulation experiments present the variations of actual output trajectories between the aggressive drivers and the cautious drivers.

There is an increasing awareness of the need to reduce the traffic accidents and fatality rates due to vehicle rollover incidents. The accurate detection of impending rollover is necessary to effectively implement vehicle rollover prevention. To this end, a real-time rollover index and a rollover tendency evaluation system are needed. These should give high accuracy and be of a low application cost. In this article, we propose a rollover evaluation system taking lateral load transfer ratio (LTR) as the rollover index with inertial measurement unit as the system input. A nonlinear suspension model and a rolling plane vehicle model are established for the state and parameter estimation. An adaptive extended Kalman filter is utilized to estimate the roll angle and rate, which adjusts noise covariance matrices to accommodate the nonlinear model characteristic and the unknown noise characteristic. In the meantime, the forgetting factor recursive least squares method is utilized to identify the height of the center of gravity. The Butterworth filter is used to filter out the high-frequency noise of the acceleration signal and the index of LTR is accordingly calculated based on the estimation results. The proposed scheme is verified and compared through hardware-in-loop tests. The results show that the developed scheme performs well in a variety of operating conditions.

Vehicle and pedestrian detection is one of the critical tasks in autonomous driving. Since heterogeneous techniques have been proposed, the selection of a detection system with an appropriate balance among detection accuracy, speed and memory consumption for a specific task has become very challenging. To deal with this issue and to provide guidance for model selection, this paper analyzes several mainstream object detection architectures, including Faster R-CNN, R-FCN, and SSD, along with several typical feature extractors, such as ResNet50, ResNet101, MobileNet_V1, MobileNet_V2, Inception_V2 and Inception_ResNet_V2. By conducting extensive experiments using the KITTI benchmark, which is a commonly used street dataset, we demonstrate that Faster R-CNN ResNet50 obtains the best average precision (AP) (58%) for vehicle and pedestrian detection, with a speed of 8.6 FPS. Faster R-CNN Inception_V2 performs best for detecting cars and detecting pedestrians respectively (74.5% and 47.3%). ResNet101 consumes the highest memory (9907 MB) and has the largest number of parameters (64.42 millions), and Inception_ResNet_V2 is the slowest model (3.05 FPS). SSD MobileNet_V2 is the fastest model (70 FPS), and SSD MobileNet_V1 is the lightest model in terms of memory usage (875 MB), both of which are suitable for applications on mobile and embedded devices.

Recently, the advancement of deep learning (DL) in discriminative feature learning from 3-D LiDAR data has led to rapid development in the field of autonomous driving. However, automated processing uneven, unstructured, noisy, and massive 3-D point clouds are a challenging and tedious task. In this article, we provide a systematic review of existing compelling DL architectures applied in LiDAR point clouds, detailing for specific tasks in autonomous driving, such as segmentation, detection, and classification. Although several published research articles focus on specific topics in computer vision for autonomous vehicles, to date, no general survey on DL applied in LiDAR point clouds for autonomous vehicles exists. Thus, the goal of this article is to narrow the gap in this topic. More than 140 key contributions in the recent five years are summarized in this survey, including the milestone 3-D deep architectures, the remarkable DL applications in 3-D semantic segmentation, object detection, and classification; specific data sets, evaluation metrics, and the state-of-the-art performance. Finally, we conclude the remaining challenges and future researches.

Autonomous urban driving navigation is still an open problem and has ample room for improvement in unknown complex environments and terrible weather conditions. In this paper, we propose a two-stage framework, called IPP-RL, to handle these problems. IPP means an Imitation learning method fusing visual information with the additional steering angle calculated by Pure-Pursuit (PP) method, and RL means using Reinforcement Learning for further training. In our IPP model, the visual information captured by camera can be compensated by the calculated steering angle, thus it could perform well under bad weather conditions. However, imitation learning performance is limited by the driving data severely. Thus we use a reinforcement learning method-Deep Deterministic Policy Gradient (DDPG)-in the second stage training, which shares the learned weights from pretrained IPP model. In this way, our IPP-RL can lower the dependency of imitation learning on demonstration data and solve the problem of low exploration efficiency caused by randomly initialized weights in reinforcement learning. Moreover, we design a more reasonable reward function and use the n-step return to update the critic-network in DDPG. Our experiments on CARLA driving benchmark demonstrate that our IPP-RL is robust to lousy weather conditions and shows remarkable generalization capability in unknown environments on navigation task.

The mutual understanding between driver and vehicle is critically important to the design of intelligent vehicles and customized interaction interface. In this study, a deep learning-based joint driver behavior reasoning system toward multi-scale and multi-tasks behavior recognition is proposed. Specifically, a multi-scale driver behavior recognition system is designed to recognize both the driver's physical and mental states based on a deep encoder-decoder framework. The system jointly recognizes three driver behaviors, namely, mirror-checking, lane change intention, and emotions based on the shared encoder network. The encoder network is designed based on a deep convolutional neural network (CNN), and several decoders for different driver states estimation are proposed with fully connected (FC), and long short-term memory (LSTM) based recurrent neural networks (RNN), respectively. The proposed framework can be used as a solution to exploit the relationship between different driver states for intelligent vehicles towards an efficient driver-side understanding. The testing results on the Brain4Car dataset show accurate performance and outperform existing methods on driver postures, intention, and emotion recognition.

Predicting driver steering intention enables intelligent vehicles to optimize its assistance and collaborative strategies with the human driver in advance, which contribute to an intelligent mutual-understanding system for driver-vehicle collaboration. In this study, a deep time-series learning-enabled driver steering intention prediction system is developed based on the Electromyography (EMG) signal processing. Specifically, the connection between the upper limb EMG signals from different muscles and the steering torque is established using a deep bi-directional long short-term memory (BiLSTM) recurrent neural network (RNN). The deep time-series model is trained to predict the future steering torque with historical EMG signals, and the prediction horizon is selected as 200 ms in this study. Moreover, three different steering postures with different hand positions on the steering wheel are studied. A joint BiLSTM network with shared temporal pattern extraction layers is developed to investigate the impact of the hand positions on the steering intention prediction. It is found that based on the joint BiLSTM network, the most accurate steering intention can be achieved with both hands on 3-clock positions. The experiments are conducted on a driving simulator environment with 21 participants. The proposed system can be used for precise driver steering intention prediction system towards a better mutual-understanding module on the intelligent and automated driving vehicles.

Internet of Things (IoT) is becoming increasingly prevalent in transportation systems. The traffic system depends on safer, faster, and more intelligent vehicles. Vehicular networks [vehicle-to-vehicle (V2V) and vehicle-to- Infrastructure (V2I)] and automated driving technique are two of the cornerstone technologies enabling the construction of the future-generation highly functional and intelligent transportation system. The IoT-based transportation system can provide enormous connections of devices and sensors for the networked automated vehicles. The capacity of connected autonomous vehicles (CAVs) is expected to be dramatically enhanced by employing IoT techniques.

This article proposes a new solution for end-to-end autonomous mining operations: Internet of Things (IoT)-based parallel mining, consisting of the concept definition, the solution given, and the concrete realization. The proposed parallel mining is inspired by the artificial societies (A) for modeling, computational experiments (C) for analysis, and parallel execution (P) for control (ACP) approach. The basic framework of parallel mining is given and its advantages are expounded. Then, the solution of parallel mining is proposed, which is mainly composed of four parts: 1) the management and control center for autonomous mining; 2) the autonomous transportation platform of truck; 3) the semiautonomous mining/shovel platform; and 4) the remote takeover platform. Key technologies of IoT-based parallel mining are discussed in detail, namely, network communication, virtual parallel mining construction, mining environment perception over-the-horizon for the moving area and obstacle detection, collaborative decision making, planning, and control for unmanned mining equipment, and parallel taking-over and remote control. Finally, the performance of IoT-based parallel mining, including fusion perception, collaborative decision making, planning, and control, is evaluated. The realization of parallel mining can fundamentally improve the safety of personnel and equipment, reduce the cost of mining operation, and increase the production rate.

Parallel end-to-end driving aims to improve the performance of end-to-end driving models using both simulated- and real-world data. However, how to efficiently utilize the data from both the simulated world and the real world remains a difficult issue, since these data are usually not well aligned. In this article, we build a data set called the parallel end-to-end driving data set (PED) for parallel end-to-end driving research. PED consists of 13 000 images from the simulated world and 13 000 images from the real world that are used to train the model, as well as 2700 images from the real world that are used to test the model. The simulated-world data in PED are constructed according to the real world, and each simulated-world image corresponds to a real-world image. PED also contains the vehicle measurement data (GPS, speed, steering angle, and heading direction of the vehicle) related to both the simulated- and real-world images, which are not available in some other data sets. We conduct two types of experiments to illustrate the effectiveness and the superiority of PED and explore some ways to mix the simulated-world data with the real-world data to improve the performance of end-to-end driving models.

Recent geometric deep learning works define convolution operations in local regions and have enjoyed remarkable success on non-Euclidean data, including graph and point clouds. However, the high-level geometric correlations between the input and its neighboring coordinates or features are not fully exploited, resulting in suboptimal segmentation performance. In this article, we propose a novel graph convolution architecture, which we term as Taylor Gaussian mixture model (GMM) network (TGNet), to efficiently learn expressive and compositional local geometric features from point clouds. The TGNet is composed of basic geometric units, TGConv, that conduct local convolution on irregular point sets and are parametrized by a family of filters. Specifically, these filters are defined as the products of the local point features and the neighboring geometric features extracted from local coordinates. These geometric features are expressed by Gaussian weighted Taylor kernels. Then, a parametric pooling layer aggregates TGConv features to generate new feature vectors for each point. TGNet employs TGConv on multiscale neighborhoods to extract coarse-to-fine semantic deep features while improving its scale invariance. Additionally, a conditional random field (CRF) is adopted within the output layer to further improve the segmentation results. Using three point cloud data sets, qualitative and quantitative experimental results demonstrate that the proposed method achieves 62.2% average accuracy on ScanNet, 57.8% and 68.17% mean intersection over union (mIoU) on Stanford Large-Scale 3D Indoor Spaces (S3DIS) and Paris-Lille-3D data sets, respectively.

Surrounding vehicle detection is one of the most important modules for a vision-based driver assistance system (VB-DAS) or an autonomous vehicle. In this paper, we put forward a wireless panoramic camera system for real-time and seamless imaging of the 360-degree driving scene. Using an embedded FPGA design, the proposed panoramic camera system can perform fast image stitching and produce panoramic videos in real-time, which greatly relives the computation and storage burden of a traditional multi-camera-based panoramic system. For surrounding vehicle detection, we present a novel deep convolutional neural network - EZ-Net, which perceives the potential vehicles by using 13 convolutional layers and locates the vehicles by a local non-maximum suppression process. Experimental results demonstrate that, the proposed EZ-Net performs vehicle detection on the panoramic video at a speed of 140 fps while holding a competing accuracy with the state-of-the-art detectors.

Driver model adaptation (DMA) provides a way to model the target driver when sufficient data are not available. Traditional DMA methods running at the model level are restricted by the specific model structures and cannot make full use of the historical data. In this paper, a novel DMA framework based on transfer learning (TL) is proposed to deal with the adaptation of driver models in lane-changing scenarios at the data level. Under the proposed DMA framework, a new TL approach named DTW-LPA that combines dynamic time warping (DTW) and local Procrustes analysis (LPA) is developed. Using the DTW, the relationship between the datasets for different drivers can be found automatically. Based on this relationship, the LPA can transfer the data in the historical dataset to the dataset of a newly-involved driver (target driver). In this way, sufficient data can be obtained for the target driver. After the data transferring process, a proper modeling method, such as the Gaussian mixture regression (GMR), can be applied to train the model for the target driver. Data collected from a driving simulator and realistic driving scenes are used to validate the proposed method in various experiments. Compared with the GMR-only and GMR-MAP methods, the DTW-LPA shows better performance on the model accuracy with much lower predicting errors in most cases.

Although 3-D reconstruction of dynamic road environment by moving cameras has been broadly applied in recognition and navigation systems, this task is still considered challenging, especially under circumstances with moving objects, where the reconstruction precision is strongly harassed by the ghosting problem. To address this issue, in this paper, we propose a novel approach for reconstructing 3-D maps of complete static scenes, based on a combination of an elaborately designed moving-object filtering mechanism and a map repairing and blank refilling procedure, where both plausible color and depth information from stereo image pairs are utilized. In this approach, first, we employ the planarity knowledge into the initial depth map based on the simple linear iterative cluster (SLIC) superpixel segmentation. The dynamic area in the image is determined under the supervision of odometry calculation. After wiping off moving objects, by collaboratively repairing color and depth information, the final 3-D map containing only static scene is obtained. The experimental results on extensive challenging real-world scenarios demonstrate the effectiveness and robustness of our approach.

Stereo matching methods consist of matching cost computation and several post processing steps. Deep learning methods have greatly raised the accuracy of matching cost and achieved the lowest error rate on several public datasets. However, their generality capabilities are not the best due to potential overfitting, which is the common problem of supervised learning approaches. This paper proposes a convolutional neural network (CNN) based cost estimation method for computing the similarity of image patches. In consideration of accuracy and generalization capability, small size convolution kernels are chosen in the convolution layer and dropout in the decision layer is used for preventing overfitting. After obtaining stereo matching cost from the output of the CNN, several post-processing operations are adopted for disparity optimization, which includes semi-global matching in 1-D from different directions, a left-right consistency check, and the slanted plane smoothing method. The method is evaluated on KITTI 2012, KITTI 2015, and Middlebury stereo datasets and the experimental results on the KITTI benchmark demonstrate the competitive accuracy performance of the approach. Additionally, to test the generalization of the method, a series of extended crossover experiments are conducted in which the training samples and testing samples come from different datasets. The results indicate superior generalization capability of our method than other supervised learning methods.

Vehicle detection from short-range aerial image faces challenges including vehicle blocking, irrelevant object interference, motion blurring, color variation etc., leading to the difficulty to achieve high detection accuracy and real-time detection speed. In this paper, benefiting from the recent development in MobileNet family network engineering, we propose a compressed MobileNet which is not only internally resistant to the above listed challenges but also gains the best detection accuracy/speed tradeoff when comparing with the original MobileNet. In a nutshell, we reduce the bottleneck architecture number during the feature map downsampling stage but add more bottlenecks during the feature map plateau stage, neither extra FLOPs nor parameters are thus involved but reduced inference time and better accuracy are expected. We conduct experiment on our collected 5-k short-range aerial images, containing six vehicle categories: truck, car, bus, bicycle, motorcycle, crowded bicycles and crowded motorcycles. Our proposed compressed MobileNet achieves 110 FPS (GPU), 31 FPS (CPU) and 15 FPS (mobile phone), 1.2 times faster and 2% more accurate (mAP) than the original MobileNet.

Vehicular networks (VNs) have received great attention as one of the crucial supportive techniques for intelligent transportation systems (ITSs). However, the introduction of dynamic and complex human behaviors into VNs makes it a cyber-social-physical system. Thus, artificial systems, computational experiments, parallel executions-based parallel VNs (PVN) are proposed in this paper. The framework of PVN is then designed and presented, its characteristics and applications are demonstrated, and its related research challenges are discussed. PVN uses software-defined artificial VNs for modeling and representation, computational experiments for analysis and evaluation, and parallel execution for control and management. Thus, more reliable and efficient traffic status and ultrahigh data rate communications are obtained among vehicles and infrastructures, which is expected to achieve the descriptive intelligence, predictive intelligence, and prescription intelligence for VNs. The proposed PVN offers a competitive solution for achieving a smooth, safe, and efficient cooperation among connected vehicles in future ITSs.

A motion planning method for autonomous vehicles confronting emergency situations where collision is inevitable, generating a path to mitigate the crash as much as possible, is proposed in this paper. The Model predictive control (MPC) algorithm is adopted here for motion planning. If avoidance is impossible for the model predictive motion planning system, the potential crash severity, and artificial potential field are filled into the controller objective to achieve general obstacle avoidance and the lowest crash severity. Furthermore, the vehicle dynamic is also considered as an optimal control problem. Based on the analysis mentioned earlier, the model predictive controller can optimize the command following, obstacle avoidance, vehicle dynamics, road regulation, and mitigate the inevitable crash based on the predicted values. The proposed MPC algorithm has been proved by simulation to have the ability to avoid obstacles and mitigate the crash if collision is inevitable.

The rapid development of intelligent vehicles has paved the way for active chassis lateral stability, which is a novel issue and critical to vehicle stability and handling performance. To obtain active chassis lateral stability for intelligent vehicles, a nonlinear model predictive control (NMPC) method integrating active front steering and an additional yaw moment is proposed. It adopts the tire sideslip angle to express vehicle lateral stability, and addresses the actuator and security constraints and the nonlinear properties of the tire-road force effectively. Moreover, the hardware implementation, based on the field programmable gate array (FPGA), is presented to satisfy miniaturization and to discuss the computational efficiency of the proposed NMPC method. To verify the effectiveness of the presented NMPC method, offline simulations comparing the NMPC method with the direct yaw moment control (DYC) method under various running conditions and a real-time implementation experiment are carried out. The results indicate that the proposed NMPC method controls better than the DYC-based method. In addition, the presented NMPC method exhibits good robustness when the longitudinal velocity and tire-road friction coefficient vary within a suitable range. Moreover, the computational time of the proposed NMPC controller, implemented using the FPGA, is only 4.994 ms during one sampling period, which can satisfy the real-time requirement of active chassis lateral stability control.

This paper proposes an extensive overview of the tire-road interaction estimation issue as it relates to automated driving from the prospectives of sensor configuration, tire modeling, and estimation approaches. The tire-road interactions needed for estimation are first determined and classified. Then, the sensor configuration schemes of different types of tire-road interactions are presented and analyzed. The following introduces various types of tire models and provides the limitations and advantages of different estimation approaches based on categorizing and summarizing those techniques. Moreover, some interesting perspectives for future research are listed based on the extensive experience of the authors.

Reconstruction based on the stereo camera has received considerable attention recently, but two particular challenges still remain. The first concerns the need to present and compress data in an effective way, and the second is to maintain as much of the available information as possible while ensuring sufficient accuracy. To overcome these issues, we propose a new 3D representation method, namely, planecell, that extracts planarity from the depth-assisted image segmentation and then directly projects these depth planes into the 3D world. The proposed method demonstrates its advancement especially dealing with large-scale structural environment, such as autonomous driving scene. The reconstruction result of our method achieves equal accuracy compared to dense point clouds and compresses the output file 200 times. To further obtain global surfaces, an energy function formulated from Conditional Random Field that generalizes the planar relationships is maximized. We evaluate our method with reconstruction baselines on the KITTI outdoor scene dataset, and the results indicate the superiorities compared to other 3D space representation methods in accuracy, memory requirements and the scope of applications.

With increasing popularity of Internet of Vehicles (IoV), concerns for reliable and low complexity communication techniques are proposed due to the requirements of signal reliability and transmission delay for vehicles. Meanwhile, the explosive and pervasive use of social network applications further adds drivers' social relationships and behavioural characteristics into it, and makes it a cyber-physical-social system (CPSS). This paper proposes and analyzes an improved forward scheme for full deplex cooperative vehicular networks in terms of its CPSS features. The proposed CPSS-based forwarding (CPSS-F) strategy forwards a soft estimate of the received signal based on social historic data at the relay node (vehicle/infrastructure) to the destination node (vehicle/infrastructure), which achieves improved reliability than the two conventional strategies in cooperative networks, i.e., amplify-and-forward (AF) and decode-and-forward (DF). Furthermore, the proposed CPSS-F relay achieves performance gains and complexity reduction compared to the conventional AF and DF. Experimental results further confirm the advantages of the proposed CPSS-F for cooperative vehicular networks. The proposed CPSS-F approach is easily extended to other cooperative vehicular social networks, for example, multi-way, half-duplex, or large-antenna networks.

This paper proposes a learning-based predictive control technique for self-driving hybrid electric vehicle (HEV). This approach is a hierarchical framework. The higher-level is a human-like driver model, which is applied to predict accelerations in the car following situation to replicate a human driver's demonstrations. The lower-level is a reinforcement learning (RL)-based controller, which enforces the battery and fuel consumption constraints to improve energy efficiency of HEV. In addition, we present induced matrix norm (IMN) to handle cases that the training data cannot provide sufficient information on how to operate in current driving situation. Simulation results illustrate that the proposed method can reproduce human driver's driving style and promote fuel economy.

Autonomous driving presents an exciting new development in vehicle technology. It poses a new challenge in driver-automation collaboration particularly during handover transitions between human and machine. In order to deal with this problem, this paper proposes a novel control framework for the haptic take-over system. The high-level framework of the haptic take-over control system, which takes driver cognitive workload, neuromuscular dynamics and optimal trajectory planning into consideration, is developed. Under the proposed framework, the determination approach of the optimal input sequence is introduced. The model of the allowed driver take-over authority, which is associated with driver's cognitive workload, as well as muscle readiness during take- over, is investigated and developed. The haptic feedback torque controller is then designed so as to minimize the deviation between the allowed control authority and driver's current degree of participation. A handover process, along with the proposed take-over control method, is also simulated. The simulation results validate the feasibility and effectiveness of the proposed approach.

Deep learning has made significant breakthroughs in the tasks of image classification, detection, segmentation, etc. However, the application of deep learning in robotics is still scarce. SLAM is a fundamental problem in robotics and loop closure detection is an important part of SLAM. This paper attempts to use supervised learning methods to solve the loop closure detection problem in vision SLAM. We proposed Siamese-ResNet network, which combines Siamese network with ResNet to detect loop closure. To show the effectiveness of Siamese-ResNet, we evaluate Siamese-ResNet and FabMap2.0 on several open published datasets, like TUM SLAM dataset and FabMap SLAM dataset. Compared with FabMap2.0, Siamese-ResNet shows higher accuracy, better robustness and shorter time-consuming.

The ability of modeling and predicting vehicle motion activities is important for automated vehicles. In this paper, we propose an And-Or Graph based model to give a simple and clear description of motion activities. Compared to other models, this new model relaxes the Markov property requirement in transition between activities and is thus more flexible. The parameters of this model can be easily learned from data. Using the trained new model, we can predict the on-going motion activity label and its corresponding probability. Experiments show that a high prediction accuracy (97%) can be achieved by this new model.

Drivers' decision and their corresponding behaviors are important aspects that can affect the driving safety, and it is necessary to understand the driver behaviors in real-time. In this study, an end-to-end driving-related tasks recognition system is proposed. Specifically, seven common driving activities are identified, which are normal driving, right mirror checking, rear mirror checking, left mirror checking, using in-vehicle video device, texting, and answering mobile phone. Among these, the first four activities are regarded as normal driving tasks, while the rest three are divided into distraction group. The images are collected using a consumer range camera, namely, Kinect. In total, five drivers are involved in the naturalistic data collection. Before training the identification model, the raw images are first segmented using a Gaussian mixture model (GMM) to extract the driver region from the background. Then, a pre-trained deep convolutional neural network (CNN) model is trained to classify the behaviors, which directly takes the processed RGB images as the input and outputs the identified label. In this work, the AIexNet is selected as the pre-trained CNN model. Then, to reduce the training cost, the transfer learning mechanism is applied to the CNN model. An average of 79% detection accuracy is achieved for the seven driving tasks. The proposed integration model can be used as a low-cost driver distraction and dangerous tasks recognition modeL.

Hybrid energy storage system (HESS) has been recognized as one of the most promising solutions to overcome the drawbacks of the expensive and short life lithium-ion battery with low power density, by introducing a proper number of supercapcitors. However, the hybridization introduces complicated sizing and energy management problems. This papers aims to investigate the sizing and real-time energy management of a devised HESS for electric vehicles with an electric race car as a case study. In particular, a proposed multi-objective Bi-level optimal sizing and control framework is implemented to find the optimal parameters of the energy management algorithm, the optimal number of the lithiumion battery cells and the supercapacitor banks. The simulation results have validated the effectiveness of the investigated methodology in minimizing the total mass of the HESS and maximizing the cycle life of the lithium-ion battery.

With the universal application of camera in intelligent vehicles, visual place recognition has become a major problem in intelligent vehicle localization. The traditional solution is to make visual description of place images using hand-crafted feature for matching places, but this description method is not very good for extreme variability, especially for seasonal transformation. In this paper, we propose a new method based on convolutional neural network (CNN), by putting images into the pre-trained network model to get automatically learned image descriptors, and through some operations of pooling, fusion and binarization to optimize them, then the similarity result of place recognition is presented with the Hamming distance of the place sequence. In the experimental part, we compare our method with some state-of-the-art algorithms, FABMAP, ABLE-M and SeqSLAM, to illustrate its advantages. The experimental results show that our method based on CNN achieves better performance than other methods on the representative public datasets.

Radio resource management (RRM), which aims to satisfy the requirements of both mobile users and service providers, can be seen as one of the typical issues of cyber-physical-social system since the social factors, that is, the requirements and priorities of users are extremely important in heterogeneous networks. In this paper, we propose a novel resource allocation and access control mechanism based on parallel network architecture, which provides a high-bandwidth connectivity with guaranteed quality of service (QoS) for mobile users in a seamless manner. In this mechanism, multiple users are classified into several types according to their social property such as priorities and bandwidth requirements of different users. Compared with the general received signal strength (RSS)-based method, the proposed user priority (UP)-based method achieves three main advantages as follows: 1) it further balances the load of base stations (BSs) when the resource is sufficient; 2) it provides a mechanism called high priority users higher QoS when the network is heavily loaded compared to the RSS-based method; and 3) it hands over a few users from a heavily loaded BS to a lightly loaded one to allow more users to access this network. The simulation results confirm the advantages of the proposed UP-based mechanism and show that the simulation results of the Q-learning method are consistent with its theoretical analysis.

Welcome to the third issue of the IEEE Transactions on Computational Social Systems (TCSS) for 2018.

Driver decisions and behaviors regarding the surrounding traffic are critical to traffic safety. It is important for an intelligent vehicle to understand driver behavior and assist in driving tasks according to their status. In this paper, the consumer range camera Kinect is used to monitor drivers and identify driving tasks in a real vehicle. Specifically, seven common tasks performed by multiple drivers during driving are identified in this paper. The tasks include normal driving, left-, right-, and rear-mirror checking, mobile phone answering, texting using a mobile phone with one or both hands, and the setup of in-vehicle video devices. The first four tasks are considered safe driving tasks, while the other three tasks are regarded as dangerous and distracting tasks. The driver behavior signals collected from the Kinect consist of a color and depth image of the driver inside the vehicle cabin. In addition, 3-D head rotation angles and the upper body (hand and arm at both sides) joint positions are recorded. Then, the importance of these features for behavior recognition is evaluated using random forests and maximal information coefficient methods. Next, a feedforward neural network (FFNN) is used to identify the seven tasks. Finally, the model performance for task recognition is evaluated with different features (body only, head only, and combined). The final detection result for the seven driving tasks among five participants achieved an average of greater than 80% accuracy, and the FFNN tasks detector is proved to be an efficient model that can be implemented for real-time driver distraction and dangerous behavior recognition.

As a typical cyber-physical system (CPS), electrified vehicle becomes a hot research topic due to its high efficiency and low emissions. In order to develop advanced electric powertrains, accurate estimations of the unmeasurable hybrid states, including discrete backlash nonlinearity and continuous half-shaft torque, are of great importance. In this paper, a novel estimation algorithm for simultaneously identifying the backlash position and half-shaft torque of an electric powertrain is proposed using a hybrid system approach. System models, including the electric powertrain and vehicle dynamics models, are established considering the drivetrain backlash and flexibility, and also calibrated and validated using vehicle road testing data. Based on the developed system models, the powertrain behavior is represented using hybrid automata according to the piecewise affine property of the backlash dynamics. A hybrid-state observer, which is comprised of a discrete-state observer and a continuous-state observer, is designed for the simultaneous estimation of the backlash position and half-shaft torque. In order to guarantee the stability and reachability, the convergence property of the proposed observer is investigated. The proposed observer are validated under highly dynamical transitions of vehicle states. The validation results demonstrates the feasibility and effectiveness of the proposed hybrid-state observer.

As an important safety-critical cyber-physical system (CPS), the braking system is essential to the safe operation of the electric vehicle. Accurate estimation of the brake pressure is of great importance for automotive CPS design and control. In this paper, a novel probabilistic estimation method of brake pressure is developed for electrified vehicles based on multilayer artificial neural networks (ANNs) with Levenberg-Marquardt backpropagation (LMBP) training algorithm. First, the high-level architecture of the proposed multilayer ANN for brake pressure estimation is illustrated. Then, the standard backpropagation (BP) algorithm used for training of the feed-forward neural network (FFNN) is introduced. Based on the basic concept of BP, a more efficient training algorithm of LMBP method is proposed. Next, real vehicle testing is carried out on a chassis dynamometer under standard driving cycles. Experimental data of the vehicle and the powertrain systems are collected, and feature vectors for FFNN training collection are selected. Finally, the developed multilayer ANN is trained using the measured vehicle data, and the performance of the brake pressure estimation is evaluated and compared with other available learning methods. Experimental results validate the feasibility and accuracy of the proposed ANN-based method for braking pressure estimation under real deceleration scenarios.

The platooning of automated vehicles has the potential to significantly benefit road traffic. This paper presents a distributed$\text{H}_{\mathrm {\infty }}$control method for multi-vehicle systems with identical dynamic controllers and rigid formation geometry. After compensating for the powertrain nonlinearity, the node dynamics in a platoon is mathematically described by a multiplicative uncertainty model. The platoon control system is then decomposed into an uncertain part and a diagonal nominal system through linear transformation and eigenvalue decomposition of the information-exchange-topology matrix. Robust stability, string stability, and distance tracking performance of the designed platoons are analyzed theoretically under the decoupled$\text{H}_{\mathrm {\infty }}$framework. A comparative simulation with non-robust controllers is used to demonstrate the effectiveness of this method.

Driver driving style plays an important role in vehicle energy management as well as driving safety. Furthermore, it is key for advance driver assistance systems development, toward increasing levels of vehicle automation. This fact has motivated numerous research and development efforts on driving style identification and classification. This paper provides a survey on driving style characterization and recognition revising a variety of algorithms, with particular emphasis on machine learning approaches based on current and future trends. Applications of driving style recognition to intelligent vehicle controls are also briefly discussed, including experts' predictions of the future development.

Within the complex driving environment, progress in autonomous vehicles is supported by advances in sensing and data fusion. Safe and robust autonomous driving can only be guaranteed provided that vehicles and infrastructure are fully aware of the driving scenario. This paper proposes a methodology for feature uncertainty prediction for sensor fusion by generating neural network surrogate models directly from data. This technique is particularly applied to vehicle location through odometry measurements, vehicle speed and orientation, to estimate the location uncertainty at any point along the trajectory. Neural networks are shown to be a suitable modeling technique, presenting good generalization capability and robust results.

In order to develop an advanced haptic take-over system for highly automated vehicles, research into the driver's neuromuscular dynamics is needed. In this paper a dynamic model of drivers' neuromuscular interaction with a steering wheel is firstly established. The transfer function and the natural frequency of the systems are analysed. In order to identify the key parameters of the driver-steering-wheel coupled system and investigate the system properties under different situations, experiments with drive-in-the-loop are carried out. For each test subject, two steering tasks, namely the passive and active steering tasks, are instructed to be completed. Furthermore, during the experiments, subjects manipulated the steering wheel with two distinct postures and three different hand positions. Based on the test results, key parameters of the transfer function and system properties are identified and investigated. The data and characteristics of the driver neuromuscular system are discussed and compared with respect to different steering tasks, hand positions and driver postures. These test results identified system properties that provide a good foundation for the development of a haptic take-over control system for automated vehicles.

With the development of intelligent technologies, self-driving vehicles are considered as a promising solution against accident, traffic congestion and pollution problems. Intelligent vehicle techniques have been the research focus all over the world. However, full self-driving vehicles are still far away from its realization and extensive application due to safety requirements and cost considerations. As a novel breakthrough, PArallel VEhicles (PAVE) incorporate the ACP theory, which facilitates real-time interaction and optimization of the actual self-driving vehicles and the artificial ones. As a result, PAVE can maintain intelligent control of the actual self-driving vehicles and achieve the global optimization via software-defined self-driving vehicles, intelligent infrastructure construction, and parallel control center. Besides, PAVE can effectively reduce the cost of high-precision equipments on the actual self-driving vehicles via remote processing and intelligent road(side) infrastructure, and also achieve improved safety and reliability via remote control, guidance and planning.

Driving cycles and road slope are two important factors affecting fuel saving performance of plug-in hybrid electric buses (PHEBs) in Chinese cities. Moreover, onboard auxiliary equipment (e.g., Global Position System receiver and General Packet Radio Service (GPRS) wireless module) of PHEB may provide potential means to communicate with the control center of the bus company, allowing for driving cycle prediction through data communication between foregoing buses and the control center. With this general approach in mind, and by utilizing driving data clustering and driving cycle classifier, this paper presents a multimode switched logic control strategy, targeting fuel economy improvement of the PHEB team for a particular city bus route. First, the normal feature parameters are extracted from the sampled driving history cycles, and the composed feature parameters are given by a mapping of normal feature parameters in this approach. A novel improved hierarchical clustering algorithm is applied for driving cycles' data clustering into four groups. Then, on the basis of the clustering results, support vector machine method is used to predict the current driving cycle. Finally, a switched driving controller is presented according to current type of driving cycle and slope information. Simulation results are compared with those of traditional methods in the given real-world driving cycles of city bus, showing significant improvement, which may offer a theoretical solution with engineering application. Experimental results also demonstrate that the proposed control approach is feasible in the tested bus routes.

In addition to decentralized controllers, the information flow among vehicles can significantly affect the dynamics of a platoon. This paper studies the influence of information flow topology on the internal stability and scalability of homogeneous vehicular platoons moving in a rigid formation. A linearized vehicle longitudinal dynamic model is derived using the exact feedback linearization technique, which accommodates the inertial delay of powertrain dynamics. Directed graphs are adopted to describe different types of allowable information flow interconnecting vehicles, including both radar-based sensors and vehicle-to-vehicle (V2V) communications. Under linear feedback controllers, a unified internal stability theorem is proved by using the algebraic graph theory and Routh-Hurwitz stability criterion. The theorem explicitly establishes the stabilizing thresholds of linear controller gains for platoons, under a large class of different information flow topologies. Using matrix eigenvalue analysis, the scalability is investigated for platoons under two typical information flow topologies, i.e., 1) the stability margin of platoon decays to zero as 0(1/N2) for bidirectional topology; and 2) the stability margin is always bounded and independent of the platoon size for bidirectional-leader topology. Numerical simulations are used to illustrate the results.

Great advances in simulation-based vehicle system design and development of various driver assistance systems have enhanced the research on improved modeling of driver steering skills. However, little effort has been made on developing driver steering skill models while capturing the uncertainties or statistical properties of the vehicle-road system. In this paper, a stochastic model predictive control (SMPC) approach is proposed to model the driver steering skill, which effectively incorporates the random variations in the road friction and roughness, a multipoint preview approach, and a piecewise affine (PWA) model structure that are developed to mimic the driver's perception of the desired path and the nonlinear internal vehicle dynamics. The SMPC method is then used to generate a steering command by minimization of a cost function, including the lateral path error and ease of driver control. In the analyses, first, the experimental data of Hongqi HQ430 are used to validate the driver steering skill controller. Then, the parametric studies of control performance during a nonlinear steering maneuver are provided. Finally, further discussions about the driver's adaption and the indication on vehicle dynamics tuning are given. The proposed switching-based SMPC driver steering control framework offers a new approach for driver behavior modeling.

Simulation-based vehicle system design and development of various active chassis control systems necessitate an enhanced understanding of driver-vehicle systems, in particular improved modeling of driver driving control characteristics. A number of research efforts have been made in developing driver models in the past few decades. However, little effort has been attempted in modeling driver steering control behavior capturing vehicle-road system parameter uncertainties. In this paper, a novel driver steering control model based on stochastic model predictive control (SMPC) is proposed to effectively incorporate the variations in the vehicle-road system parameters. The proposed SMPC-based driver steering control framework consists of three modules, namely perception, decision and execution, where a multi-point driver preview approach is employed. An internal vehicle dynamics model with the parameter uncertainty in road friction coefficient is formulated to represent the driver's knowledge and adaptation about the variations in road conditions. The SMPC method is then used to minimize a cost function that is a weighted combination of lateral path error and ease of driver control. Simulation analysis about the variant parameters and comparison with an MPC-based driver model demonstrate the effectiveness and robustness of the proposed SMPC-based driver steering control model.

The definition, formulation, and importance of vehicle 'critical speed' have been well documented in the vehicle dynamics literature. One of the main current research directions in vehicle dynamics and control is to enhance the understanding of characteristics of driver-vehicle systems. However, no efforts have been attempted in the literature to derive a new mathematical formulation for the 'critical speed' in the context of driver-vehicle systems, in order to capture the contributions of human driver properties, apart from the vehicle properties. This study derives two alternative analytical formulations for the 'critical speed' in driver-vehicle systems using two different simplified driver-vehicle system models. These two new formulations for the 'critical speed' are a function of vehicle/tire parameters as well as human driver model parameters, such as driver preview time, delay time, and control gain. Simulation analyses are then conducted to evaluate and demonstrate the effectiveness of the derived 'critical speed' formulations, compared with the results from a relatively more comprehensive driver-vehicle system model. It is further demonstrated that one of the formulations derived could also be able to predict the system instability or 'critical speed' for neutral-or under-steered vehicles, in driver-vehicle systems, apart from that for over-steered vehicles.