The inflexible human-autonomy relationship within autonomous driving scenarios still has not realized synergetic intelligence, therefore unable to provide adaptive and context-sensitive decision-making and sometimes leading to violation of human pReferences or even hazards. In this paper, we utilize functional near-infrared spectroscopy (fNIRS) signals as real-time human risk-perception feedback to establish a brain-in-the-loop (BiTL) trained artificial intelligence algorithm for decision-making. The proposed algorithm uses the result of driving risk reasoning as one input of reinforcement learning combining fNIRS-based risk and driving safety field model-based risk, realizing integrating human brain activity into the reinforcement learning scheme, then overcoming the disadvantage of machine-oriented intelligence that could violate human intentions. To achieve policy learning within limited BiTL training periods, we add two modification features to the proposed algorithm based on TD3. The experiment involving twenty participants has been conducted, and the results show that in continuously high-risk driving scenarios, compared to traditional reinforcement learning algorithms without human participation, the proposed algorithm can maintain a cautious driving policy and avoid potential collisions, validated with both proximal surrogate indicators and success rates.

With the rapid advancement of autonomous driving, safety concerns have become the primary barrier to its commercialization. Compliance with traffic laws is crucial for ensuring road safety. However, the current laws, formulated for human drivers, present challenges for autonomous systems due to ambiguous language description, complicating accurate judgment and government monitoring. It is imperative to transform traffic laws into machine-interpretable logical frameworks while simul-taneously resolving ambiguities in legal terminology to ensure clarity and precision. This study focuses on urban intersections, characterized by high traffic complexity and diverse participants. We propose a formalization method for right-of-way laws and develop a threshold analysis framework based on processed data from SIND, which rigorously defines the prioritization of right-of-way. The optimal compliance threshold is determined through sensitivity analysis, evaluated using the proposed Weighted TPN score (WTPNs). Meanwhile, the threshold was implemented in online monitoring at intersections. The dataset is available online via: https://github.comlSOTIF-AVLab/SinD

Accurate trajectory prediction is vital for various applications, including autonomous vehicles. However, the complexity and limited transparency of many prediction algorithms often result in black-box models, making it challenging to understand their limitations and anticipate potential failures. This further raises potential risks for systems based on these prediction models. This study introduces the performance monitor for black-box trajectory prediction model (PMBP) to address this challenge. The PMBP estimates the performance of black-box trajectory prediction models online, enabling informed decision-making. The study explores various methods’ applicability to the PMBP, including anomaly detection, machine learning, deep learning, and ensemble, with specific monitors designed for each method to provide online output representing prediction performance. Comprehensive experiments validate the PMBP’s effectiveness, comparing different monitoring methods. Results show that the PMBP effectively achieves promising monitoring performance, particularly excelling in deep learning-based monitoring. It achieves improvement scores of 0.81 and 0.79 for average prediction error and final prediction error monitoring, respectively, outperforming previous white-box and gray-box methods. Furthermore, the PMBP’s applicability is validated on different datasets and prediction models, while ablation studies confirm the effectiveness of the proposed mechanism. Hybrid prediction and autonomous driving planning experiments further show the PMBP’s value from an application perspective. Project page: https://swb19.github.io/PMBP/.Note to Practitioners—This research presents PMBP, a valuable tool for practitioners in the automation industry. The PMBP enables online monitoring of black-box trajectory prediction models, enhancing system reliability and facilitating informed decision-making. The practical application of PMBP lies in improving safety and reliability in critical domains, especially in the context of autonomous vehicles. Black-box trajectory prediction models commonly used in these domains may exhibit unexpected deficiencies, potentially leading to risks. By monitoring the prediction performance online, systems can proactively identify potential insufficiencies and make informed decisions to ensure safer and more reliable operations. The PMBP offers practitioners different monitoring solutions based on various approaches, addressing their specific needs effectively. While the PMBP has shown promising outcomes, further exploration and testing are necessary to fully harness and apply its monitoring results in automated systems. Practitioners are encouraged to adopt the PMBP as an essential monitoring mechanism to enhance the reliability of their trajectory prediction models and achieve safer and more efficient automation in their domains.

Sudden traffic hazards trigger collision-avoidance behaviors in drivers that can significantly impact vehicle dynamics, potentially conflicting with the existing advanced driver assistance systems (ADAS), such as autonomous emergency braking and steering. This behavior can lead to unexpected vehicle movements, further complicating the situation and elevating the risk of accidents. Understanding and tailoring the inference of drivers’ perception-response time (PRT) is essential for optimizing ADAS activation in intelligent vehicles. This approach allows customization for individual drivers, improving safety and ensuring that interventions are personalized and minimally disruptive to normal driving patterns. To achieve this objective, this study performs high-fidelity simulation experiments to gather a comprehensive multidimensional dataset on drivers’ responses in safety-critical scenarios, primarily focusing on PRT and its influencing factors. Using the collision-avoidance behavior data, a driver evidence accumulation model is created to explain PRT distribution and facilitate real-time personalized inferences. We also analyze the relationship between model parameters and real-world physical significance, demonstrating that driver decisions rely on visual evidence accumulation influenced by dynamic interactions in different scenarios. Our proposed model, by offering a detailed understanding of drivers’ perceptual and decision-making processes, aids in developing personalized driver assistance system activation recommendations. This approach seeks to create personalized and adaptive systems within intelligent vehicles, thereby reducing human-machine conflicts and improving the overall safety of intelligent transportation systems.

Automated Vehicle (AV) safety is a critical issue and appeals to worldwide focus. To ensure AV safety, AV functions should be tested and evaluated in an enormous number of scenarios. Since such AV testing is time-consuming, scenario filters have been developed to identify safety-critical scenarios and omit ordinary ones. However, the scenarios identified by these filters do not uniquely match the AV function to be tested and are most likely not critical for the AV function. Therefore, an enhanced scenario filter is proposed in this paper. It bears the following features: 1) Automated-driving-function-specific scenario identification; 2) High coverage of critical scenarios; 3) Enhanced identification efficiency by avoiding adopting a surrogate model; 4) High reliability of critical scenario identification. To enable the above features, the proposed filter formulates the identification problem into an optimization problem and solves it with a model-free approach. Experiments have been conducted to evaluate and validate the proposed filter. The results confirm that the proposed filter is able to improve coverage of critical scenarios, efficiency of identification, and reliability of identification compared to the state-of-the-art filter. Specifically, the proposed filter improves coverage by up to 70 percent, efficiency by up to 97 percent, and reliability by up to 22 percent. The results also reveal that the proposed filter shows an increasing advantage for testing AV functions with higher complexity.

In recent years, researchers have focused on identifying tasks related to driver state, traffic environment, and others to enhance the safety of autonomous driving assistance systems. However, current research on these tasks is conducted independently, neglecting the interconnections between the driver, traffic environment, and vehicle. In this paper, we propose a Unified Multi-task Assistive Driving Network Based on Multimodal Fusion (UMD-Net), the first unified model capable of recognizing four tasks simultaneously by utilizing multimodal data: driver behavior recognition, driver emotion recognition, traffic context recognition, and vehicle behavior recognition. In order to better enhance the synergistic effects between multiple tasks, we designed the position-sensitive multi-directional attention feature extraction subnetwork and recursive dynamic feature fusion module. The former captures the key features of multi-view images by different directions of attention mechanism to improve the generalization of the model across multiple tasks. The latter dynamically adjusts the fusion weight according to the multimodal features to enhance the representation ability of important features in multi-task learning. Our model was evaluated on the public dataset AIDE, achieving the best performance across all four tasks and a high accuracy of 95.31% in the traffic context recognition task, demonstrating the superiority of our approach. The code is available on https://github.com/Wenzhuo-Liu/UMD-Net.

Roadside perception can significantly enhance the safety of autonomous vehicles by extending their perceptual capabilities beyond the visual range and addressing occluded regions. However, current state-of-the-art vision-based roadside detection methods exhibit high accuracy on labeled scenes but perform poorly on new scenes. This limitation arises because roadside cameras remain stationary after installation and can only gather data from a single scene, leading the algorithm to overfit these roadside backgrounds and camera positions. To tackle this issue, we propose an innovativeScenarioGeneralization Framework forVision-based Roadside3DObject Detection, calledSGV3D. Specifically, we utilize a Background-suppressed Module (BSM) to reduce background overfitting in vision-centric pipelines by diminishing background features during the 2D to bird’s-eye-view projection. Furthermore, by introducing the Semi-supervised Data Generation Pipeline (SSDG) that employs unlabeled images from new scenes, we generate diverse foreground instances with varying camera poses, mitigating the risk of overfitting to specific camera positions. Experiments conducted on two large-scale roadside benchmarks demonstrate that SGV3D, with only a minimal increase in latency, effectively improves the scenario generalization capabilities of vision-based roadside 3D object detectors. The code is available here (https://github.com/yanglei18/SGV3D).

In the rapidly evolving field of autonomous driving, reliable prediction is pivotal for vehicular safety. However, trajectory predictions often deviate from actual paths, particularly in complex and challenging environments, leading to significant errors. To address this issue, our study introduces a novel method for Dynamic Occupancy Set (DOS) prediction, it effectively combines advanced trajectory prediction networks with a DOS prediction module, overcoming the shortcomings of existing models. It provides a comprehensive and adaptable framework for predicting the potential occupancy sets of traffic participants. The innovative contributions of this study include the development of a novel DOS prediction model specifically tailored for navigating complex scenarios, the introduction of precise DOS mathematical representations, and the formulation of optimized loss functions that collectively advance the safety and efficiency of autonomous systems. Through rigorous validation, our method demonstrates marked improvements over traditional models, establishing a new benchmark for safety and operational efficiency in intelligent transportation systems.

Truck platooning is a promising technology in freight transport. To commercialize truck platooning as early as possible, its evaluation is in urgent need. For truck platooning evaluation, simulation platforms play a crucial role. However, there has not been a simulation platform to meet the evaluation needs of various stakeholders, including Original Equipment Manufacturers (OEMs), Freight Operators (FOs) and Transportation Management Administrations (TMAs). To fill the research gap, this paper proposes a next-generation simulation platform. It integrates a traffic simulator, platoon management system, and truck control module to satisfy all the evaluation needs. The proposed platform bears the following features: i) Compatibility with various platooning decision makers, planners, controllers, vehicle types and platoon management strategies; ii) Capability of evaluating platoon performance on the lateral dimension; iii) Prototype platoon management system provided for FOs; iv) Capability of evaluating truck platoon management performance in terms of sustainability and economy; v) Capability of evaluating the impact of interactive background traffic on platoon performance. vi) Capability of evaluating the impact of truck platoon management on traffic mobility. The proposed platform is validated by comparison against an actual field test. Its credibility is confirmed in terms of truck platoon performance and interactive traffic simulation. Additional tests are conducted to evaluate truck platoon performance and the impact of truck platoons on mixed traffic. The results reveal that existing platoon lane-change technologies should be upgraded to be compatible with high-traffic-demand scenarios. It is also revealed that a localized and up-to-date assessment is required before allowing truck platooning.

Autonomous driving confronts great challenges in complex traffic scenarios, where the SOTIF risk can be triggered by the dynamic operational environment and system insufficiencies. The SOTIF risk is reflected not only intuitively in the collision risk with objects outside the autonomous vehicles, but also inherently in the performance limitation risk of the implemented algorithms. How to minimize the SOTIF risk for autonomous driving is currently a critical, difficult, and unresolved issue. Therefore, this paper proposes the “Self-Surveillance and Self-Adaption System” as a systematic approach to online minimize the SOTIF risk, which aims to provide a systematic solution for monitoring, quantification, and mitigation of inherent and external risks. As a demonstration of the system, the risk monitoring of the perception algorithm is highlighted. Moreover, the inherent perception algorithm risk and external collision risk are jointly quantified via SOTIF entropy, which is then propagated downstream to the decision-making module and mitigated. Finally, Hardware-in-the-Loop experiments are conducted to verify the efficiency and effectiveness of the system. The results demonstrate that the system enables dependable online monitoring, quantification, and mitigation of SOTIF risk in real-time critical traffic environments.

Instance segmentation of traffic participants using vision-based techniques serves as a cornerstone for numerous intelligent transportation systems. Although existing deep learning-based methods have made significant advancements in this field, these algorithms still present considerable challenges with regard to generalizability for commercial deployment. Specifically, the mean average perception (mAP) of these algorithms degrades rapidly when dealing with intersections that are not included in the training set. To address this limitation, a novel instance segmentation approach named Si-GAIS is proposed, which incorporates a siamese structure for the first time with the proposed Generalizable-Attention Encoder (GA). Through the proposed Foreground-Background Fusion Unit (FBF) within GA, efficient feature-level fusion for the foreground and background images is achieved. Additionally, the Interpretable Attention Neck (IA) in GA enables the feature encoder to focus exclusively on the foreground traffic participants while ignoring various backgrounds. To utilize Si-GAIS, an unsupervised method named P-DBSCAN is proposed to obtain high-quality background image for each intersection with slow-moving traffic and camera jitters. Finally, the first multi-intersection multi-category instance segmentation datasets named RopeIns is proposed for validation. Si-GAIS achieves a 7.7% mAP (All APs used in this paper are abbreviations of AP$_{\textit {50}}$the same with PASCAL VOC.) accuracy improvement compared to the state-of-the-art (SOTA) methods while using fewer parameters, with only a 6.4% decline in AP for car segmentation in unseen intersections and weather conditions, whereas all other SOTA methods decline more than 10%. The proposed dataset and source code are publicly available athttps://github.com/441599828/SiGAISand we hope Si-GAIS will be a new baseline for IPS instance segmentation research.

Although the majority of recent autonomous driving systems concentrate on developing perception methods based on ego-vehicle sensors, there is an overlooked alternative approach that involves leveraging intelligent roadside cameras to help extend the ego-vehicle perception ability beyond the visual range. We discover that most existing monocular 3D object detectors rely on the ego-vehicle prior assumption that the optical axis of the camera is parallel to the ground. However, the roadside camera is installed on a pole with a pitched angle, which makes the existing methods not optimal for roadside scenes. In this paper, we introduce a novel framework for Roadside Monocular 3D object detection with ground-aware embeddings, named MonoGAE. Specifically, the ground plane is a stable and strong prior knowledge due to the fixed installation of cameras in roadside scenarios. In order to reduce the domain gap between the ground geometry information and high-dimensional image features, we employ a supervised training paradigm with a ground plane to predict high-dimensional ground-aware embeddings. These embeddings are subsequently integrated with image features through cross-attention mechanisms. Furthermore, to improve the detector’s robustness to the divergences in cameras’ installation poses, we replace the ground plane depth map with a novel pixel-level refined ground plane equation map. Our approach demonstrates a substantial performance advantage over all previous monocular 3D object detectors on widely recognized 3D detection benchmarks for roadside cameras. The code and pre-trained models will be released soon.

Motion prediction is essential for safe and efficient autonomous driving. However, the inexplicability and uncertainty of complex artificial intelligence models may lead to unpredictable failures of the motion prediction module, which may mislead the system to make unsafe decisions. Therefore, it is necessary to develop methods to guarantee reliable autonomous driving, where failure detection is a potential direction. Uncertainty estimates can be used to quantify the degree of confidence a model has in its predictions and may be valuable for failure detection. We propose a framework of failure detection for motion prediction from the uncertainty perspective, considering both motion uncertainty and model uncertainty, and formulate various uncertainty scores according to different prediction stages. The proposed approach is evaluated based on different motion prediction algorithms, uncertainty estimation methods, uncertainty scores, etc., and the results show that uncertainty is promising for failure detection for motion prediction but should be used with caution.

Perception algorithms in autonomous driving systems confront great challenges in long-tail traffic scenarios, where the problems of Safety of the Intended Functionality (SOTIF) could be triggered by the algorithm performance insufficiency and dynamic operational environment. However, such scenarios are not systematically included in current open-source datasets, and this paper fills the gap accordingly. Based on the analysis and enumeration of trigger conditions, a high-quality diverse dataset is released, including various long-tail traffic scenarios collected from multiple resources. Considering the development of probabilistic object detection (POD), this dataset marks trigger sources that may cause perception SOTIF problems in the scenarios as key objects. In addition, an evaluation protocol is suggested to verify the effectiveness of POD algorithms in identifying the key objects via uncertainty. The dataset never stops expanding, and the first batch of open-source data includes 1126 frames with an average of 2.27 key objects and 2.47 normal objects in each frame. To demonstrate how to use this dataset for SOTIF research, this paper further quantifies the perception SOTIF entropy to confirm whether a scenario is unknown and unsafe for a perception system. The experimental results show that the quantified entropy can effectively and efficiently reflect the failure of the perception algorithm.

Trajectory prediction is one of the key components of the autonomous driving software stack. Accurate prediction for the future movement of surrounding traffic participants is an important prerequisite for ensuring the driving efficiency and safety of intelligent vehicles. Trajectory prediction algorithms based on artificial intelligence have been widely studied and applied in recent years and have achieved remarkable results. However, complex artificial intelligence models are uncertain and difficult to explain, so they may face unintended failures when applied in the real world. In this paper, a self-aware trajectory prediction method is proposed. By introducing a self-awareness module and a two-stage training process, the original trajectory prediction module's performance is estimated online, to facilitate the system to deal with the possible scenario of insufficient prediction function in time, and create conditions for the realization of safe and reliable autonomous driving. Comprehensive experiments and analysis are performed, and the proposed method performed well in terms of self-awareness, memory footprint, and real-time performance, showing that it may serve as a promising paradigm for safe autonomous driving.

Facts proved that obeying traffic laws keeps the promise to promote the safety of self-driving vehicles. Current self-driving vehicles usually have fixed algorithms during autonomous driving, however the traffic laws may differ or change in different regions or times, e.g., tidal lanes. It raises a crucial requirement to make self-driving vehicles adapt to the newly received traffic laws. The challenges are that traffic laws are usually semantic and manually designed, but the original algorithms may not always contain the pre-designed interface to adapt to emerging laws. To this end, this work proposes a traffic law adaptive decision-making platform, which uses the linear temporal logic (LTL) formula to consistently describe the semantic traffic laws. Then, an LTL-based reinforcement learning framework is designed to estimate the probability of illegal behavior under different traffic laws. Finally, a law-specific backup policy is designed to maintain the performance threshold by monitoring the probability of illegal behavior. This work takes three typical scenarios where the traffic laws differ for instance to prove the effectiveness of the proposed approach, i.e., law amendment presented by the government, law difference between different regions, and temporary traffic control. The results show that the proposed method can help the original decision-making algorithms adapt to the traffic laws well without pre-defined interfaces. This method provides a way to administer on-road driving self-driving vehicles.

Accurate trajectory prediction is essential for safe and efficient autonomous driving in complex traffic environments. While artificial intelligence has shown great promise in improving prediction accuracy, its inherent uncertainty and lack of explainability may lead to unpredictable failures, creating challenges for safety-critical decision-making. This study aims to address these challenges by exploring the impact of traffic environment on prediction algorithms. The study proposes a trajectory prediction framework with epistemic uncertainty estimation ability to output high uncertainty when facing unforeseeable or unknown scenarios. The framework analyzes the environmental effect on the trajectory prediction by considering scenario features and shifts. Features are divided into kinematic features of a target agent, features of surrounding traffic participants, and other scenario features. Feature correlation and importance analyses are performed to study their influence on prediction error and epistemic uncertainty. The impact of unavoidable distributional shifts in the real world on trajectory predictions is investigated using multiple intersection datasets. The results indicate that deep ensemble-based methods have advantages in improving robustness while estimating epistemic uncertainty. Consistent conclusions were obtained from the correlation and importance analyses, indicating that kinematic features of the target agent have relatively strong effects on both prediction error and epistemic uncertainty. Finally, the study analyzes the accuracy deterioration caused by distributional shifts and the potential of the deep ensemble-based method. Through deep ensemble, the errors of the prediction methods based on GRIP++ and Trajectron++ have been improved by 6.4% and 10.8% in the same-dataset test, and 6.3% and 10.8% in the cross-dataset test.

3D Multi-object tracking (MOT) ensures consistency during continuous dynamic detection, conducive to subsequent motion planning and navigation tasks in autonomous driving. However, camera-based methods suffer in the case of occlusions and it can be challenging to track the irregular motion of objects for LiDAR-based methods accurately. Some fusion methods work well but do not consider the untrustworthy issue of appearance features under occlusion. At the same time, the false detection problem also significantly affects tracking. As such, we propose a novel camera-LiDAR fusion 3D MOT framework based on Combined Appearance-Motion Optimization (CAMO-MOT), which uses both camera and LiDAR data and significantly reduces tracking failures caused by occlusion and false detection. For occlusion problems, we are the first to propose an occlusion head to select the best object appearance features multiple times effectively, reducing the influence of occlusions. To decrease the impact of false detection in tracking, we design a motion cost matrix based on confidence scores which improve the positioning and object prediction accuracy in 3D space. As existing multi-object tracking methods always evaluate each category separately and do not consider the mismatch between objects of different categories, we also propose to build a multi-category cost to implement multi-object tracking in multi-category scenes. A series of validation experiments are conducted on the KITTI and nuScenes tracking benchmarks. Our proposed method achieves state-of-the-art performance with 79.99% HOTA and the lowest identity switches (IDS) value (23 for Car and 137 for Pedestrian) among all multi-modal MOT methods on the KITTI test dataset. And our method achieves state-of-the-art performance among all algorithms on the nuScenes test dataset with 75.3% AMOTA.

Motion prediction modules are crucial for autonomous vehicles to forecast the future behavior of surrounding road users. Failures in prediction modules can mislead a downstream planner to make unsafe decisions. Currently, deep learning technology has been widely used to design prediction models due to its impressive performance. However, such models may fail in long-tail driving scenarios where the training data are insufficient or unavailable, which represents the so-called epistemic uncertainty of prediction models. This paper proposes a risk-aware decision-making (RADM) framework to handle the epistemic uncertainty arising from training the prediction model on insufficient data. First, a multi-agent prediction network with epistemic uncertainty quantification is proposed. This network uses the historical states of nearby road users, map information, and traffic lights as inputs. Then, the RADM utilizes model predictive control technique to not only process the multi-agent prediction results but also to consider the epistemic uncertainty of the prediction model. In addition, the accuracy of the established prediction model is verified on real-world driving datasets. Furthermore, the proposed RADM is evaluated on the log-replay data obtained from real-world driving logs and using the SUMO simulator, considering multiple challenging cases where pedestrians and non-motorized vehicles cross the intersection illegally. The experimental results demonstrate that RADM can reduce the driving risk and improve driving safety and supplementary videos are provided athttps://github.com/SOTIF-AVLab/RADM.

Autonomous driving is considered one of the revolutionary technologies shaping humanity’s future mobility and quality of life. However, safety remains a critical hurdle in the way of commercialization and widespread deployment of autonomous vehicles on public roads. Safety concerns require the autonomous driving system to handle uncertainties from multiple sources that are either preexisting, e.g., the stochastic behavior of traffic participants or scenario occlusion, or introduced as a result of processing, e.g., the application of neural networks. Thus, it is crucial to analyze the sources of uncertainties and quantify the risks associated with them, including the propagated risks that accumulate in the decision-making system. In this context, this paper provides an overview of uncertainty challenges and state-of-the-art techniques for mitigating these challenges. We argue that the uncertainties mainly originate from two aspects: 1) the external traffic environment, and 2) the internal autonomous driving system. Specifically, this paper first analyzes the safety challenges caused by the uncertainties and summarizes their sources. In addition, the corresponding techniques that mitigate and quantify the risk of uncertainties are presented. Finally, research perspectives are highlighted to facilitate future studies for guaranteeing the safety of autonomous vehicles.

The safety of autonomous vehicles is hard to ensure in adverse weather since the sensors will degrade drastically. Setting a variable speed limit based on real-time weather condition is the most efficient method to make the vehicle safe. But most current speed limit methods are based on human visibility rather than the sensor, which is not suitable for autonomous vehicles. Thus, it is necessary to explore the performance of sensors in different weathers and propose a speed limit method based on sensor performance. Safety decisions will be made based on the calculated speed limit to ensure safety.This paper describes how to make safety decisions based on sensor performance and road conditions in real-time. The experiment explores the degradation of different sensors, and variable speed limit methods are proposed for rainy and foggy days. MPC controller is used to generate safety decisions.

Accurate trajectory prediction of surrounding vehicles enables lower risk path planning in advance for autonomous vehicles, thus promising the safety of automated driving. A low-risk and high-efficiency path planning approach is proposed for autonomous driving based on the high-performance and practical trajectory prediction method. A long short-term memory (LSTM) network is trained and tested using the highD dataset, and the validated LSTM is used to predict the trajectories of surrounding vehicles combining the information extracted from vehicle-to-vehicle (V2V) technology. A risk assessment and mitigation-based local path planning algorithm is proposed according to the information of predicted trajectories of surrounding vehicles. Two driving scenarios are extracted and reconstructed from the highD dataset for validation and evaluation, i.e., an active lane-change scenario and a longitudinal collision-avoidance scenario. The results illustrate that the risk is mitigated and the driving efficiency is improved with the proposed path planning algorithm comparing to the constant-velocity prediction and the prediction method of the nonlinear input–output (NIO) network, especially when the velocity and trajectory with sudden changes. Note to Practitioners—This article was motivated by the problem of promising the safety decision-making and path planning through accurate environment prediction. There are two main parts included in this article. First, this article proposed one pragmatic approach to predict the environment movement correctly based on the long short-term memory (LSTM) approach. The prediction performance of LSTM was compared with nonlinear input–output (NIO). The results showed that the LSTM approach has a significant advantage in motivation prediction of the surrounded vehicles during path planning. The second part of this article is to make the decision and realize local path planning based on the risk assessment. The potential field-based approach is implemented on the risk assessment based on these accurate predictions. Some primary results demonstrate that the decision-making algorithm performs better under the accurate prediction model. The results also show that the safety and driving efficiency of the ego vehicle were improved by tracking the trajectory, which was planned based on the risk assessment. The only concern for the real-time application is the computation time; in future, we will figure it out how to further reduce the computation time.

Most environment perception methods in autonomous vehicles rely on deep neural networks because of their impressive performance. However, neural networks have black-box characteristics in nature, which may lead to perception uncertainty and untrustworthy autonomous vehicles. Thus, this work proposes a decision-making method to adapt the potential perception uncertainty due to the sensor noises, fuzzy features, and unfamiliar inputs. The whole method is named as Perception Neural Networks Uncertainty Aware Decision-Making (PNNUAD) method. PNNUAD first uses the Monte Carlo dropout method to estimate the perception neural network uncertainty into a distribution around the original output. Then, the perception uncertainty will be considered in a designed reinforcement learning-based planner using a distributed value function. Finally, a backup policy will maintain the vehicle’s performance to avoid disastrous perception uncertainty. The evaluation section uses an augmented reality urban driving scenario; namely, the scenario builds in the CARLA simulator while the perception uncertainty comes from the real dataset. This case study focuses on the object class uncertainty of a widely used neural network, i.e., YOLO-V3. The results indicate that the proposed method can maintain AV safety even with poor perception performance. Meanwhile, the AV has not become too conservative by defending the perception uncertainty. This work is necessary for applying the statistics neural networks to safety-critical autonomous vehicles, and the source code will be open-source in this work.

The rollover dynamics has been few touched on during the study of path planning in current researches. This paper proposed an anti-rollover motion planning for heavy-duty vehicle. Taking the coupling of roll motion of sprung mass of the front axle with that of the drive axle into consideration, 7 degrees of freedom vehicle rollover model is established, and an evaluation index that can accurately describe the rollover motion state is derived. Then, the Model Predictive Control (MPC) is designed for motion planning which combining the rollover dynamics, artificial potential field for obstacle avoidance and trajectory tracking. Thus, the optimal path without collision risk, rollover trend and vehicle dynamics constraint is calculated. Finally, some typical scenarios are applied to validate the performance of the proposed motion planning. Results state that the anti-rollover motion planning can avoid collisions effectively and reduce the rollover risk simultaneously.

A motion planning method for autonomous vehicles confronting emergency situations where collision is inevitable, generating a path to mitigate the crash as much as possible, is proposed in this paper. The Model predictive control (MPC) algorithm is adopted here for motion planning. If avoidance is impossible for the model predictive motion planning system, the potential crash severity, and artificial potential field are filled into the controller objective to achieve general obstacle avoidance and the lowest crash severity. Furthermore, the vehicle dynamic is also considered as an optimal control problem. Based on the analysis mentioned earlier, the model predictive controller can optimize the command following, obstacle avoidance, vehicle dynamics, road regulation, and mitigate the inevitable crash based on the predicted values. The proposed MPC algorithm has been proved by simulation to have the ability to avoid obstacles and mitigate the crash if collision is inevitable.

This paper proposes a learning-based predictive control technique for self-driving hybrid electric vehicle (HEV). This approach is a hierarchical framework. The higher-level is a human-like driver model, which is applied to predict accelerations in the car following situation to replicate a human driver's demonstrations. The lower-level is a reinforcement learning (RL)-based controller, which enforces the battery and fuel consumption constraints to improve energy efficiency of HEV. In addition, we present induced matrix norm (IMN) to handle cases that the training data cannot provide sufficient information on how to operate in current driving situation. Simulation results illustrate that the proposed method can reproduce human driver's driving style and promote fuel economy.

A path planning approach to generate a path which mitigates the effects of an inevitable crash for autonomous vehicles is presented in this brief. The model predictive control algorithm is adopted here for path planning. The artificial potential field, which describes the obstacles and the potential crash severity, are added to the control objectives to avoid the obstacle, and also to mitigate the inevitable crash. The vehicle dynamic is also considered as an optimal control objective. Based on the analysis above, the model predictive controller can guarantee the command following, obstacle avoidance, vehicle dynamics, and mitigate the inevitable crash. Simulation results verified that the proposed MPC has the abilities of obstacles avoidance and mitigation of the inevitable crash.

Differential drive assistance steering (DDAS) is an emerging assisted steering mechanism in in-wheel-motor driven (IWMD) electric vehicles, yielded by the differential moment of the front tires in the steering system. DDAS can steer the front wheels when there is no steering power from the steering motor, and thus can be used as a redundant steering mechanism. To realize the yaw control when the active front steering entirely breaks down and guarantee the transient control performance therein, this paper proposes an integral sliding mode control (ISMC) approach for IWMD electric vehicles steered by DDAS. Two contributions are made in this paper: 1) An improved disturbance observer based ISMC strategy is designed to cope with the unknown mismatched disturbances, and the composite nonlinear feedback technique is employed to design the nominal part of the controller to restrain overshoots and remove steady-state errors considering the tire force saturations; 2) An adaptive super-twisting control approach is proposed to deal with the disturbances with unknown boundaries using a continuous controller while eliminating the chattering effect. The system stability and robustness are proved via Lyapunov approach. CarSim-Simulink simulation has verified the effectiveness of the proposed control approach in the case of the steering fault.

This paper introduces a novel rotation-based framework for arbitrary-oriented text detection in natural scene images. We present theRotation Region Proposal Networks, which are designed to generate inclined proposals with text orientation angle information. The angle information is then adapted for bounding box regression to make the proposals more accurately fit into the text region in terms of the orientation. TheRotation Region-of-Interestpooling layer is proposed to project arbitrary-oriented proposals to a feature map for a text region classifier. The whole framework is built upon a region-proposal-based architecture, which ensures the computational efficiency of the arbitrary-oriented text detection compared with previous text detection systems. We conduct experiments using the rotation-based framework on three real-world scene text detection datasets and demonstrate its superiority in terms of effectiveness and efficiency over previous approaches.

The rapid advances of transportation infrastructure have led to a dramatic increase in the demand for smart systems capable of monitoring traffic and street safety. Fundamental to these applications are a community-based evaluation platform and benchmark for object detection and multi-object tracking. To this end, we organize the AVSS2017 Challenge on Advanced Traffic Monitoring, in conjunction with the International Workshop on Traffic and Street Surveillance for Safety and Security (IWT4S), to evaluate the state-of-the-art object detection and multi-object tracking algorithms in the relevance of traffic surveillance. Submitted algorithms are evaluated using the large-scale UA-DETRAC benchmark and evaluation protocol. The benchmark, the evaluation toolkit and the algorithm performance are publicly available from the website http://detrac-db.rit.albany.edu.

We perform fast vehicle detection from traffic surveillance cameras. A novel deep learning framework, namely Evolving Boxes, is developed that proposes and refines the object boxes under different feature representations. Specifically, our framework is embedded with a light-weight proposal network to generate initial anchor boxes as well as to early discard unlikely regions; a fine-turning network produces detailed features for these candidate boxes. We show intriguingly that by applying different feature fusion techniques, the initial boxes can be refined for both localization and recognition. We evaluate our network on the recent DETRAC benchmark and obtain a significant improvement over the state-of-the-art Faster RCNN by 9.5% mAP. Further, our network achieves 9–13 FPS detection speed on a moderate commercial GPU.

In this paper, we introduce an active annotation and learning framework for the face recognition task. Starting with an initial label deficient face image training set, we iteratively train a deep neural network and use this model to choose the examples for further manual annotation. We follow the active learning strategy and derive the Value of Information criterion to actively select candidate annotation images. During these iterations, the deep neural network is incrementally updated. Experimental results conducted on LFW benchmark and MS-Celeb-1M challenge demonstrate the effectiveness of our proposed framework.