Yang Xing 0002

dblp:149/4493-2 · DBLP profile ↗
← Back
25ranked-venue papers
9as first author
16since 2021 · last 2026
—ORCID · conflict

Domains — the database's venue-derived domains; a paper can count in several

Applied, interdisciplinary, general and emerging computing · 15 · 4 first-author · 10 since 2021Human-computer interaction and ubiquitous computing · 7 · 3 first-author · 5 since 2021Artificial intelligence and machine learning · 5 · 4 first-author · 3 since 2021Computer networks · 2 · 1 since 2021Systems, architecture and hardware · 1
YearPublicationVenuePosition
2026 Augmenting Human Hazard Situational Awareness With Haptic Interface for Heterogeneous Autonomous Vehicles
abstract
As vehicle autonomy increases, human operators become more susceptible to distractions and a loss of situational awareness (SA) due to cognitive limitations. Rapidly enhancing human SA in hazardous situations is, therefore, critical for timely hazard perception and collision avoidance, particularly in human-vehicle teaming contexts that demand fast, accurate hazard reasoning. This study evaluates the efficiency of a low-cost vibrotactile interface for enhancing hazard SA of human operators when teaming with heterogeneous autonomous vehicles, including both ground and aerial autonomous vehicles. To do so, we evaluate the effectiveness of a vibrotactile interface in challenging time-critical scenarios considering adversarial attacks to better understand the cognitive constraints faced by human operators. Our quantitative analysis, based on the data collected from 39 participants, demonstrates that: first, haptic cues can significantly enhance human hazard SA across various metrics for the ground and aerial scenarios; second, perception of aerial attacks in a 3-D environment is more challenging than ground risk perception.
Yang Xing 0002, Xiangqi Kong, Weisi Guo, Antonios Tsourdos
IEEE Trans. Hum. Mach. Syst.1
2025 Explaining Autonomous Navigation to Human-in-the-Loop Operator in Multi-Task Rotorcraft Search & Rescue Operations
abstract
Aerial search and rescue (SAR) rotorcrafts currently need multiple specialist human operators, increasing cost and the risk of downtime due to crew unavailability and mental stress. Autonomy can aid fewer operators performing multiple tasks, but the human operator must maintain situation awareness (SA) of crucial autonomous decisions. A key challenge is the cognitive stress on a multi-tasking human-in-the-loop (HITL) due to the AI agent making decisions without human understanding. Explainable AI (XAI) has often been proposed as a way to explain autonomy decisions, but current XAI solutions doesn’t adapt to real-time human factors in high stress and high stakes situations. Here, we allow an AI agent to perform autonomous rotorcraft navigation, whilst the HITL operator has to perform two simultaneous tasks: (i) search for a target on the ground by toggling an onboard camera, and (ii) maintain SA of the autonomous navigation task through our novel XAI interface. Our novel XAI approach leverages on dimensionality reduction techniques to visualize the reinforcement learning (RL) navigation’s internal states, highlighting patterns in its decision-making process through intuitive interactive clustering on saliency maps. To ensure convergence on performance, we design a two-way interface that allows the human to interpret AI decisions and then give feedback via a Large Language Model to modify the autonomous navigation. Testing demonstrates increased task performance (+43%), while experiencing substantial human reductions in physical demand (-53%), time pressure (-30%), effort (-23%), and frustration (-26%), but at the cost of slightly increased mental demand (+12%).
Nathaniel Amadi, Samuel Cartwright, Noe Claudel, Jamal Mohammed, Kenechukwu Agbo, Paris Chatzithanos, Mariusz Wisniewski, Antonios Tsourdos, Yang Xing 0002, Weisi Guo
SMC9
2025 Generative Street-View using Satellite Images with Hallucination Reduction via Semantic Constraining
abstract
Autonomous navigation requires training data in diverse transport settings. Accurate street/ground level representation is important to train autonomous driving, tourism planning, environmental protection, and a wide range of sectors. Many parts of the inhabited and most of the uninhabited world lacks street view imagery. Current street image generation can transform satellite imagery into synthetic 3D images, but there is a high level of hallucination. Here, we develop a Neural Gazetteer that integrates semantic narrative data (e.g., review comments and place attributes) to reduce hallucination in generative street-view images. Our novel work flow involves using satellite imagery to extract a geometry projection of the area and then integrating semantic narrative data into a diffusion model to generate realistic street views. We perform a wide range of comparisons with ground truth for urban and rural areas to identify the performance of our approach at both the feature-scale as well as the human perception semantic-scale.
Oluwatoni Esan, Mariam Gugushvili, Jean Eudes Konain, Hanish Kasturilal Uppal, Thomas Prosser, Minqing Qiu, Mariusz Wisniewski, Yang Xing 0002, Weisi Guo
SMC8
2025 A Cross-Platform Study of Human Situational Awareness for Heterogeneous Low Altitude Autonomy
abstract
Human–autonomy teaming in the Low Altitude Economy (LAE) requires operators to manage both ground and aerial autonomous agents under time pressure, spatial uncertainty, and cognitive load. This study investigates how visual and haptic feedback affect operator situational awareness (SA) in simulated collision avoidance tasks involving cars and drones. A high-fidelity virtual environment was built using Unreal Engine 4 and AirSim, with haptic cues delivered through a wearable bHaptics vest. Twenty-two participants performed within-subject trials across visual-only and visual–haptic conditions. Results showed that haptic feedback significantly enhanced SA, particularly in dimensions related to information acquisition and spare mental capacity. Improvements were more consistent in car-based tasks, while drone scenarios exhibited greater inter-individual variability. These findings demonstrate the potential of multimodal interfaces to support cognitive performance and reduce platform-related disparities in operator SA. This work provides empirical evidence for designing adaptive, perception-aware interfaces in safety-critical human–autonomy teaming systems.
Yang Xing 0002, Argyrios C. Zolotas, Adolfo Perrusquía, Weisi Guo, Antonios Tsourdos
SMC2
2025 Interaction-Aware and Driving Style-Aware Trajectory Prediction for Heterogeneous Vehicles in Mixed Traffic Environment
abstract
Trajectory prediction (TP) of surrounding vehicles (SVs) is crucial for autonomous vehicles (AVs) to understand traffic situations and achieve safe-efficient decision-making and motion planning. However, different drivers’ personalized driving preferences will bring uncertainties for long-term TP in the mixed traffic environment. To this end, this paper proposes a TP model with interaction awareness and driving style awareness for long-term TP of heterogeneous SVs. Firstly, the driving conditions in the highD dataset are distinguished, and three different driving styles of the vehicle in the car-following condition are obtained based on an unsupervised clustering algorithm. Then, an encoder-decoder architecture based on novel lane attention and multi-head attention mechanisms is proposed, where the encoder analyzes historical trajectory patterns and the decoder generates future trajectory sequences. The lane attention mechanism enhances the spatial perception capability of vehicles towards the target lane, and the multi-head attention mechanism extracts high-dimensional global interaction information about the heterogeneous vehicle group (HVG) surrounding the target vehicle (TV). Experimental results show that the proposed model outperforms state-of-the-art models in root-mean-square-error (RMSE) for long-term TP and exhibits excellent adaptability to diverse driving tasks. Moreover, this paper verifies that the driving style topology within the HVG has multiple impacts on the TP accuracy of the TV.
Yang Xing 0002, Jinxiang Wang 0002, Zhenwu Fang, Guodong Yin
IEEE Trans. Intell. Transp. Syst.2
2024 RGANFormer: Relativistic Generative Adversarial Transformer for Time-Series Signal Forecasting on Intelligent Vehicles
abstract
Time-series modelling (TSM) is a critical task for intelligent vehicles (IVs), covering areas like fault detection, health monitoring, and inference of road user intentions. In this study, we present a novel TSM approach for enhancing the accuracy of multi-variate signal forecasting in intelligent vehicles. Our method leverages advanced Transformer networks within a relativistic generative adversarial network (RGAN) training framework. The RGAN training framework efficiently improves the accuracy of vehicle states forecasting for IV, demonstrating effective learning of long-time dependencies for more accurate predictions over extended sequences. Additionally, we introduce a high-dimensional extension (HDE) built-in block for the time-series Transformer to explore the impact of higher-dimensional features on representing long-term sequences. The experimental data is collected from a real-world electric vehicle testing bed. We evaluate the proposed RGANFormer framework and the HDE block on two popular time-series models, namely, Autoformer and FiLM. The results demonstrate that the RGANFormer, along with the built-in HDE block, significantly enhances long-term sequential forecasting accuracy for both multivariate and univariate tasks.
Yang Xing 0002, Xiangqi Kong, Antonios Tsourdos
IV1
2024 Multiscale Human Activity Recognition and Anticipation Network
abstract
Deep convolutional neural networks have been leveraged to achieve huge improvements in video understanding and human activity recognition performance in the past decade. However, most existing methods focus on activities that have similar time scales, leaving the task of action recognition on multiscale human behaviors less explored. In this study, a two-stream multiscale human activity recognition and anticipation (MS-HARA) network is proposed, which is jointly optimized using a multitask learning method. The MS-HARA network fuses the two streams of the network using an efficient temporal-channel attention (TCA)-based fusion approach to improve the model's representational ability for both temporal and spatial features. We investigate the multiscale human activities from two basic categories, namely, midterm activities and long-term activities. The network is designed to function as part of a real-time processing framework to support interaction and mutual understanding between humans and intelligent machines. It achieves state-of-the-art results on several datasets for different tasks and different application domains. The midterm and long-term action recognition and anticipation performance, as well as the network fusion, are extensively tested to show the efficiency of the proposed network. The results show that the MS-HARA network can easily be extended to different application domains.
Yang Xing 0002, Stuart Golodetz, Aluna Everitt, Andrew Markham, Agathoniki Trigoni
IEEE Trans. Neural Networks Learn. Syst.1
2023 An Adaptive Energy Efficient MAC Protocol for RF Energy Harvesting WBANs
abstract
Continuous and remote health monitoring medical applications with heterogeneous requirements can be realized through wireless body area networks (WBANs). Energy harvesting is adopted to enable low-power health applications and long-term monitoring without battery replacement, which have drawn significant interest recently. Because energy harvesting WBANs are obviously different from battery-powered ones, network protocols should be designed accordingly to improve network performance. In this article, an efficient cross-layer media access control protocol is proposed for radio frequency powered energy harvesting WBANs. We redesigned the superframe structure, which can be rescheduled by the coordinator dynamically. A time switching (TS) strategy is used when sensors harvest energy from radio frequency signals broadcast by the coordinator, and a transmission power adjustment scheme is proposed for sensors based on the energy harvesting efficiency and the network environment. Energy efficiency can be effectively improved that more packets can be uploaded using limited energy. The length of the energy harvesting period is determined by the coordinator to balance the channel resources and energy requirements of sensors and further improve the network performance. Numerical simulation results show that our protocol can provide superior system performance for long-term periodic health monitoring applications.
Juncheng Hu 0002, Gaochao Xu, Liang Hu 0001, Yang Xing 0002
IEEE Trans. Commun.5
2023 Milestones in Autonomous Driving and Intelligent Vehicles - Part I: Control, Computing System Design, Communication, HD Map, Testing, and Human Behaviors
abstract
Interest in autonomous driving (AD) and intelligent vehicles (IVs) is growing at a rapid pace due to the convenience, safety, and economic benefits. Although a number of surveys have reviewed research achievements in this field, they are still limited in specific tasks and lack systematic summaries and research directions in the future. Our work is divided into three independent articles and the first part is a survey of surveys (SoS) for total technologies of AD and IVs that involves the history, summarizes the milestones, and provides the perspectives, ethics, and future research directions. This is the second part (Part I for this technical survey) to review the development of control, computing system design, communication, high-definition map (HD map), testing, and human behaviors in IVs. In addition, the third part (Part II for this technical survey) is to review the perception and planning sections. The objective of this article is to involve all the sections of AD, summarize the latest technical milestones, and guide abecedarians to quickly understand the development of AD and IVs. Combining the SoS and Part II, we anticipate that this work will bring novel and diverse insights to researchers and abecedarians, and serve as a bridge between past and future.
Long Chen 0005, Yuchen Li 0004, Chao Huang 0006, Yang Xing 0002, Daxin Tian, Li Li 0013, Zhongxu Hu, Siyu Teng, Chen Lv 0001, Jinjun Wang, Dongpu Cao, Nanning Zheng 0001, Fei-Yue Wang 0001
IEEE Trans. Syst. Man Cybern. Syst.4
2022 Risk Assessment and Mitigation in Local Path Planning for Autonomous Vehicles With LSTM Based Predictive Model
abstract
Accurate trajectory prediction of surrounding vehicles enables lower risk path planning in advance for autonomous vehicles, thus promising the safety of automated driving. A low-risk and high-efficiency path planning approach is proposed for autonomous driving based on the high-performance and practical trajectory prediction method. A long short-term memory (LSTM) network is trained and tested using the highD dataset, and the validated LSTM is used to predict the trajectories of surrounding vehicles combining the information extracted from vehicle-to-vehicle (V2V) technology. A risk assessment and mitigation-based local path planning algorithm is proposed according to the information of predicted trajectories of surrounding vehicles. Two driving scenarios are extracted and reconstructed from the highD dataset for validation and evaluation, i.e., an active lane-change scenario and a longitudinal collision-avoidance scenario. The results illustrate that the risk is mitigated and the driving efficiency is improved with the proposed path planning algorithm comparing to the constant-velocity prediction and the prediction method of the nonlinear input–output (NIO) network, especially when the velocity and trajectory with sudden changes. Note to Practitioners—This article was motivated by the problem of promising the safety decision-making and path planning through accurate environment prediction. There are two main parts included in this article. First, this article proposed one pragmatic approach to predict the environment movement correctly based on the long short-term memory (LSTM) approach. The prediction performance of LSTM was compared with nonlinear input–output (NIO). The results showed that the LSTM approach has a significant advantage in motivation prediction of the surrounded vehicles during path planning. The second part of this article is to make the decision and realize local path planning based on the risk assessment. The potential field-based approach is implemented on the risk assessment based on these accurate predictions. Some primary results demonstrate that the decision-making algorithm performs better under the accurate prediction model. The results also show that the safety and driving efficiency of the ego vehicle were improved by tracking the trajectory, which was planned based on the risk assessment. The only concern for the real-time application is the computation time; in future, we will figure it out how to further reduce the computation time.
Hong Wang 0014, Bing Lu 0005, Jun Li 0082, Yang Xing 0002, Chen Lv 0001, Dongpu Cao, Ehsan Hashemi
IEEE Trans Autom. Sci. Eng.5
2022 CogEmoNet: A Cognitive-Feature-Augmented Driver Emotion Recognition Model for Smart Cockpit
abstract
Driver’s emotion recognition is vital to improving driving safety, comfort, and acceptance of intelligent vehicles. This article presents a cognitive-feature-augmented driver emotion detection method that is based on emotional cognitive process theory and deep networks. Different from the traditional methods, both the driver’s facial expression and cognitive process characteristics (age, gender, and driving age) were used as the inputs of the proposed model. Convolutional techniques were adopted to construct the model for driver’s emotion detection simultaneously considering the driver’s facial expression and cognitive process characteristics. A driver’s emotion data collection was carried out to validate the performance of the proposed method. The collected dataset consists of 40 drivers’ frontal facial videos, their cognitive process characteristics, and self-reported assessments of driver emotions. Another two deep networks were also used to compare recognition performance. The results prove that the proposed method can achieve well detection results for different databases on the discrete emotion model and dimensional emotion model, respectively.
Wenbo Li 0003, Guanzhong Zeng, Juncheng Zhang, Yang Xing 0002, Dongpu Cao, Fei-Yue Wang 0001
IEEE Trans. Comput. Soc. Syst.5
2022 Cooperative Decision Making of Connected Automated Vehicles at Multi-Lane Merging Zone: A Coalitional Game Approach
abstract
To address the safety and efficiency issues of vehicles at multi-lane merging zones, a cooperative decision-making framework is designed for connected automated vehicles (CAVs) using a coalitional game approach. Firstly, a motion prediction module is established based on the simplified single-track vehicle model for enhancing the accuracy and reliability of the decision-making algorithm. Then, the cost function and constraints of the decision making are designed considering multiple performance indexes, i.e. the safety, comfort and efficiency. Besides, in order to realize human-like and personalized smart mobility, different driving characteristics are considered and embedded in the modeling process. Furthermore, four typical coalition models are defined for CAVS at the scenario of a multi-lane merging zone. Then, the coalitional game approach is formulated with model predictive control (MPC) to deal with decision making of CAVs at the defined scenario. Finally, testings are carried out in two cases considering different driving characteristics to evaluate the performance of the developed approach. The testing results show that the proposed coalitional game based method is able to make reasonable decisions and adapt to different driving characteristics for CAVs at the multi-lane merging zone. It guarantees the safety and efficiency of CAVs at the complex dynamic traffic condition, and simultaneously accommodates the objectives of individual vehicles, demonstrating the feasibility and effectiveness of the proposed approach.
Peng Hang, Chen Lv 0001, Chao Huang 0006, Yang Xing 0002, Zhongxu Hu
IEEE Trans. Intell. Transp. Syst.4
2022 Multi-Agent Trajectory Prediction With Heterogeneous Edge-Enhanced Graph Attention Network
abstract
Simultaneous trajectory prediction for multiple heterogeneous traffic participants is essential for safe and efficient operation of connected automated vehicles under complex driving situations. Two main challenges for this task are to handle the varying number of heterogeneous target agents and jointly consider multiple factors that would affect their future motions. This is because different kinds of agents have different motion patterns, and their behaviors are jointly affected by their individual dynamics, their interactions with surrounding agents, as well as the traffic infrastructures. A trajectory prediction method handling these challenges will benefit the downstream decision-making and planning modules of autonomous vehicles. To meet these challenges, we propose a three-channel framework together with a novel Heterogeneous Edge-enhanced graph ATtention network (HEAT). Our framework is able to deal with the heterogeneity of the target agents and traffic participants involved. Specifically, agents’ dynamics are extracted from their historical states using type-specific encoders. The inter-agent interactions are represented with a directed edge-featured heterogeneous graph and processed by the designed HEAT network to extract interaction features. Besides, the map features are shared across all agents by introducing a selective gate-mechanism. And finally, the trajectories of multiple agents are predicted simultaneously. Validations using both urban and highway driving datasets show that the proposed model can realize simultaneous trajectory predictions for multiple agents under complex traffic situations, and achieve state-of-the-art performance with respect to prediction accuracy. The achieved final displacement error (FDE@3sec) is 0.66 meter under urban driving, demonstrating the feasibility and effectiveness of the proposed approach.
Xiaoyu Mo, Zhiyu Huang, Yang Xing 0002, Chen Lv 0001
IEEE Trans. Intell. Transp. Syst.3
2021 Deep convolutional neural network-based Bernoulli heatmap for head pose estimation
Zhongxu Hu, Yang Xing 0002, Chen Lv 0001, Peng Hang, Jie Liu 0017
Neurocomputing2
2021 Human-Like Decision Making for Autonomous Driving: A Noncooperative Game Theoretic Approach
abstract
Considering that human-driven vehicles and autonomous vehicles (AVs) will coexist on roads in the future for a long time, how to merge AVs into human drivers' traffic ecology and minimize the effect of AVs and their misfit with human drivers, are issues worthy of consideration. Moreover, different passengers have different needs for AVs, thus, how to provide personalized choices for different passengers is another issue for AVs. Therefore, a human-like decision making framework is designed for AVs in this paper. Different driving styles and social interaction characteristics are formulated for AVs regarding driving safety, ride comfort and travel efficiency, which are considered in the modeling process of decision making. Then, Nash equilibrium and Stackelberg game theory are applied to the noncooperative decision making. In addition, potential field method and model predictive control (MPC) are combined to deal with the motion prediction and planning for AVs, which provides predicted motion information for the decision-making module. Finally, two typical testing scenarios of lane change, i.e., merging and overtaking, are carried out to evaluate the feasibility and effectiveness of the proposed decision-making framework considering different human-like behaviors. Testing results indicate that both the two game theoretic approaches can provide reasonable human-like decision making for AVs. Compared with the Nash equilibrium approach, under the normal driving style, the cost value of decision making using the Stackelberg game theoretic approach is reduced by over 20%.
Peng Hang, Chen Lv 0001, Yang Xing 0002, Chao Huang 0006, Zhongxu Hu
IEEE Trans. Intell. Transp. Syst.3
2021 Toward Safe and Smart Mobility: Energy-Aware Deep Learning for Driving Behavior Analysis and Prediction of Connected Vehicles
abstract
Connected automated driving technologies have shown tremendous improvement in recent years. However, it is still not clear how driving behaviors and energy consumption correlate with each other and to what extent these factors related to connected vehicles can influence the motion prediction performance. The precise recognition of driving behaviors and prediction of the vehicle motion is critical to the driving safety for connected automated vehicles (CAVs). Hence, in this study, an energy-aware driving pattern analysis and motion prediction system are proposed for CAVs using a deep learning-based time-series modeling approach. First, energy-aware longitudinal acceleration and deceleration behaviors and lateral lane-change behaviors are statistically analyzed. Then, a sliding standard deviation (SSD) test is applied to evaluate the smoothness of the trajectory and velocity signals considering different energy consumption levels. An energy-aware personalized joint time-series modeling (PJTSM) approach based on a deep recurrent neural network (RNN) and long short-term memory (LSTM) cell are proposed for accurate motion (trajectory and velocity) prediction of the leading vehicle. Finally, the differences in the prediction performance regarding different energy consumption levels are compared and discussed. It is shown that due to the higher randomness of the driving behaviors, the prediction accuracy for heavy energy users is the lowest among the three categories, which means it is harder to anticipate the driving behaviors of cars exhibiting heavy energy consumption. The personalized estimation of driving behaviors of CAVs will contribute to safer automated driving and transportation systems.
Yang Xing 0002, Chen Lv 0001, Xiaoyu Mo, Zhongxu Hu, Chao Huang 0006, Peng Hang
IEEE Trans. Intell. Transp. Syst.1
2020 Interaction-Aware Trajectory Prediction of Connected Vehicles using CNN-LSTM Networks
abstract
Predicting the future trajectory of a surrounding vehicle in congested traffic is one of the necessary abilities of an autonomous vehicle. In congestion, a vehicle's future movement is the result of its interaction with surrounding vehicles. A vehicle in congestion may have many neighbors in a relatively short distance, while only a small part of neighbors affect its future trajectory mostly. In this work, An interaction-aware method that predicts the future trajectory of an ego vehicle considering its interaction with eight surrounding vehicles is proposed. The dynamics of vehicles are encoded by LSTMs with shared weights, and the interaction is extracted with a simple CNN. The proposed model is trained and tested on trajectories extracted from the publicly accessible NGSIM US-101 dataset. Quantitative experimental results show that the proposed model outperforms previous models in root-mean-square error (RMSE). Results visualization shows that the model is able to predict future trajectory induced by lane change before the vehicle operates noticeable lateral movement to initiate lane changing.
Xiaoyu Mo, Yang Xing 0002, Chen Lv 0001
IECON2
2020 Driver-Automation Collaboration for Automated Vehicles: A Review of Human-Centered Shared Control
abstract
The automated driving vehicles are experiencing a rapid development in worldwide recently. It is commonly believed that before the achievement of fully autonomous driving, the driver will always need to remain within the vehicle control loop. Hence, intelligent interaction and collaboration between the human driver and the automation will be an efficient solution for the improvement of road safety, traffic efficiency, and social acceptance to the automated vehicles. As a popular collaboration method, shared control has been widely studied in the past two decades. While it is still a challenging task to involve rich human factors into the shared control system to increase the driving experience and acceptance of the automation. In this study, a literature review on human-centered shared control is proposed towards solid research on driver-vehicle collaboration. First, the basic background and literature surveys on the human-machine collaboration (HMC) is proposed, and the important factors for efficient multi-agent collaboration and teaming are discussed. Then, different driver behavior and state modeling methods are reviewed. Based on the HMC schemes and driver behavior recognition techniques, literature surveys on human-centered shared control are proposed. Finally, challenges and future works on human-centered shared control are analyzed.
Yang Xing 0002, Chao Huang 0006, Chen Lv 0001
IV1
2020 Multi-Scale Driver Behaviors Reasoning System for Intelligent Vehicles Based on a Joint Deep Learning Framework
abstract
The mutual understanding between driver and vehicle is critically important to the design of intelligent vehicles and customized interaction interface. In this study, a deep learning-based joint driver behavior reasoning system toward multi-scale and multi-tasks behavior recognition is proposed. Specifically, a multi-scale driver behavior recognition system is designed to recognize both the driver's physical and mental states based on a deep encoder-decoder framework. The system jointly recognizes three driver behaviors, namely, mirror-checking, lane change intention, and emotions based on the shared encoder network. The encoder network is designed based on a deep convolutional neural network (CNN), and several decoders for different driver states estimation are proposed with fully connected (FC), and long short-term memory (LSTM) based recurrent neural networks (RNN), respectively. The proposed framework can be used as a solution to exploit the relationship between different driver states for intelligent vehicles towards an efficient driver-side understanding. The testing results on the Brain4Car dataset show accurate performance and outperform existing methods on driver postures, intention, and emotion recognition.
Yang Xing 0002, Zhongxu Hu, Zhiyu Huang, Chen Lv 0001, Dongpu Cao, Efstathios Velenis
SMC1
2020 Continuous Driver Steering Intention Prediction Considering Neuromuscular Dynamics and Driving Postures
abstract
Predicting driver steering intention enables intelligent vehicles to optimize its assistance and collaborative strategies with the human driver in advance, which contribute to an intelligent mutual-understanding system for driver-vehicle collaboration. In this study, a deep time-series learning-enabled driver steering intention prediction system is developed based on the Electromyography (EMG) signal processing. Specifically, the connection between the upper limb EMG signals from different muscles and the steering torque is established using a deep bi-directional long short-term memory (BiLSTM) recurrent neural network (RNN). The deep time-series model is trained to predict the future steering torque with historical EMG signals, and the prediction horizon is selected as 200 ms in this study. Moreover, three different steering postures with different hand positions on the steering wheel are studied. A joint BiLSTM network with shared temporal pattern extraction layers is developed to investigate the impact of the hand positions on the steering intention prediction. It is found that based on the joint BiLSTM network, the most accurate steering intention can be achieved with both hands on 3-clock positions. The experiments are conducted on a driving simulator environment with 21 participants. The proposed system can be used for precise driver steering intention prediction system towards a better mutual-understanding module on the intelligent and automated driving vehicles.
Yang Xing 0002, Chen Lv 0001, Yifan Zhao 0001, Dongpu Cao
SMC1
2020 Special Issue on Internet of Things for Connected Automated Driving
abstract
Internet of Things (IoT) is becoming increasingly prevalent in transportation systems. The traffic system depends on safer, faster, and more intelligent vehicles. Vehicular networks [vehicle-to-vehicle (V2V) and vehicle-to- Infrastructure (V2I)] and automated driving technique are two of the cornerstone technologies enabling the construction of the future-generation highly functional and intelligent transportation system. The IoT-based transportation system can provide enormous connections of devices and sensors for the networked automated vehicles. The capacity of connected autonomous vehicles (CAVs) is expected to be dramatically enhanced by employing IoT techniques.
Dongpu Cao, Li Li 0013, Clara Marina Martinez, Long Chen 0005, Yang Xing 0002, Weihua Zhuang
IEEE Internet Things J.5
2020 Transfer Learning for Driver Model Adaptation in Lane-Changing Scenarios Using Manifold Alignment
abstract
Driver model adaptation (DMA) provides a way to model the target driver when sufficient data are not available. Traditional DMA methods running at the model level are restricted by the specific model structures and cannot make full use of the historical data. In this paper, a novel DMA framework based on transfer learning (TL) is proposed to deal with the adaptation of driver models in lane-changing scenarios at the data level. Under the proposed DMA framework, a new TL approach named DTW-LPA that combines dynamic time warping (DTW) and local Procrustes analysis (LPA) is developed. Using the DTW, the relationship between the datasets for different drivers can be found automatically. Based on this relationship, the LPA can transfer the data in the historical dataset to the dataset of a newly-involved driver (target driver). In this way, sufficient data can be obtained for the target driver. After the data transferring process, a proper modeling method, such as the Gaussian mixture regression (GMR), can be applied to train the model for the target driver. Data collected from a driving simulator and realistic driving scenes are used to validate the proposed method in various experiments. Compared with the GMR-only and GMR-MAP methods, the DTW-LPA shows better performance on the model accuracy with much lower predicting errors in most cases.
Chao Lu 0006, Fengqing Hu, Dongpu Cao, Jianwei Gong, Yang Xing 0002
IEEE Trans. Intell. Transp. Syst.5
2018 End-to-End Driving Activities and Secondary Tasks Recognition Using Deep Convolutional Neural Network and Transfer Learning
abstract
Drivers' decision and their corresponding behaviors are important aspects that can affect the driving safety, and it is necessary to understand the driver behaviors in real-time. In this study, an end-to-end driving-related tasks recognition system is proposed. Specifically, seven common driving activities are identified, which are normal driving, right mirror checking, rear mirror checking, left mirror checking, using in-vehicle video device, texting, and answering mobile phone. Among these, the first four activities are regarded as normal driving tasks, while the rest three are divided into distraction group. The images are collected using a consumer range camera, namely, Kinect. In total, five drivers are involved in the naturalistic data collection. Before training the identification model, the raw images are first segmented using a Gaussian mixture model (GMM) to extract the driver region from the background. Then, a pre-trained deep convolutional neural network (CNN) model is trained to classify the behaviors, which directly takes the processed RGB images as the input and outputs the identified label. In this work, the AIexNet is selected as the pre-trained CNN model. Then, to reduce the training cost, the transfer learning mechanism is applied to the CNN model. An average of 79% detection accuracy is achieved for the seven driving tasks. The proposed integration model can be used as a low-cost driver distraction and dangerous tasks recognition modeL.
Yang Xing 0002, Jianlin Tang, Chen Lv 0001, Dongpu Cao, Efstathios Velenis, Fei-Yue Wang 0001
Intelligent Vehicles Symposium1
2018 Identification and Analysis of Driver Postures for In-Vehicle Driving Activities and Secondary Tasks Recognition
abstract
Driver decisions and behaviors regarding the surrounding traffic are critical to traffic safety. It is important for an intelligent vehicle to understand driver behavior and assist in driving tasks according to their status. In this paper, the consumer range camera Kinect is used to monitor drivers and identify driving tasks in a real vehicle. Specifically, seven common tasks performed by multiple drivers during driving are identified in this paper. The tasks include normal driving, left-, right-, and rear-mirror checking, mobile phone answering, texting using a mobile phone with one or both hands, and the setup of in-vehicle video devices. The first four tasks are considered safe driving tasks, while the other three tasks are regarded as dangerous and distracting tasks. The driver behavior signals collected from the Kinect consist of a color and depth image of the driver inside the vehicle cabin. In addition, 3-D head rotation angles and the upper body (hand and arm at both sides) joint positions are recorded. Then, the importance of these features for behavior recognition is evaluated using random forests and maximal information coefficient methods. Next, a feedforward neural network (FFNN) is used to identify the seven tasks. Finally, the model performance for task recognition is evaluated with different features (body only, head only, and combined). The final detection result for the seven driving tasks among five participants achieved an average of greater than 80% accuracy, and the FFNN tasks detector is proved to be an efficient model that can be implemented for real-time driver distraction and dangerous behavior recognition.
Yang Xing 0002, Chen Lv 0001, Zhaozhong Zhang, Huaji Wang, Xiaoxiang Na, Dongpu Cao, Efstathios Velenis, Fei-Yue Wang 0001
IEEE Trans. Comput. Soc. Syst.1
2018 Levenberg-Marquardt Backpropagation Training of Multilayer Neural Networks for State Estimation of a Safety-Critical Cyber-Physical System
abstract
As an important safety-critical cyber-physical system (CPS), the braking system is essential to the safe operation of the electric vehicle. Accurate estimation of the brake pressure is of great importance for automotive CPS design and control. In this paper, a novel probabilistic estimation method of brake pressure is developed for electrified vehicles based on multilayer artificial neural networks (ANNs) with Levenberg-Marquardt backpropagation (LMBP) training algorithm. First, the high-level architecture of the proposed multilayer ANN for brake pressure estimation is illustrated. Then, the standard backpropagation (BP) algorithm used for training of the feed-forward neural network (FFNN) is introduced. Based on the basic concept of BP, a more efficient training algorithm of LMBP method is proposed. Next, real vehicle testing is carried out on a chassis dynamometer under standard driving cycles. Experimental data of the vehicle and the powertrain systems are collected, and feature vectors for FFNN training collection are selected. Finally, the developed multilayer ANN is trained using the measured vehicle data, and the performance of the brake pressure estimation is evaluated and compared with other available learning methods. Experimental results validate the feasibility and accuracy of the proposed ANN-based method for braking pressure estimation under real deceleration scenarios.
Chen Lv 0001, Yang Xing 0002, Junzhi Zhang, Xiaoxiang Na, Yutong Li 0002, Dongpu Cao, Fei-Yue Wang 0001
IEEE Trans. Ind. Informatics2