Industry 5.0 heralds a new wave of the industrial revolution, placing a spotlight on human-centric intelligent manufacturing. At the core of Industry 5.0 lies the human-cyber-physical system (HCPS), a composite intelligent system where interactions among humans, cyberspace, and physical assets are orchestrated across diverse manufacturing levels and phases. Understanding the pivotal roles played by humans in these advanced systems is of paramount importance. Nonetheless, the exploration of HCPS within the context of Industry 5.0 remains in its infancy. This paper presents a holistic literature review of industrial HCPS from a human-centric perspective. A united architecture is employed to encompass the aspects of cognitive-to-technology integration and human-to-human interaction in HCPS, highlighting human-in-the-loop, human-on-the-loop, and human-in-the-society paradigms. The mechanisms of these paradigms and their effects on design, production, and service are investigated to expand the research landscape of intelligent manufacturing in Industry 5.0. Key enabling technologies that facilitate harmonious tri-space integration are introduced, and the future challenges of industrial HCPS are discussed. This work is expected to attract more open discussions and in-depth research on HCPS in the new industrial revolution era.Note to Practitioners—This paper is motivated by the emergence of Industry 5.0 that integrates humans into cyber-physical systems to offset drawbacks on both sides. It presents an overview of HCPS-related works to identify the state-of-the-art and open problems in the Industry 5.0 era. The review of HCPS applications in the design, production, and service phases can benefit engineers in the intelligent manufacturing area. Key enabling technologies on human ability augmentation, human-robot interaction, digital twin, human-cyber-physical data fusion, crowdsourcing, and system modeling, are analyzed to attract researchers in broader research fields to join in the development of industrial HCPS.

Autonomous driving (AD) will rely, either fully or partially, on data-driven approaches. As such, being aware of the algorithm limitation is crucial when implementing learning-based methods in such safety-critical contexts. A comprehensive AD monitor allows control authority to be transferred promptly to a contingency backup solution when the vehicle is recognized in impasses. To address this challenge, we propose MonitorGAN, a Bayesian discriminator trained within an adversarial framework, designed to recognize unknown traffic scenarios and monitor planning quality in open-world autonomous driving. Additionally, it is designed to be aware of its own limitations using a Bayesian approach. Unlike previous epistemic uncertainty estimation algorithms for self-driving, MonitorGAN is independent and planner-agnostic, capable of monitoring various types of planners without requiring real outlier exposure. MonitorGAN is trained exclusively on Argoverse 2 and tested through extensive cross-dataset experiments, including NGISM, HighD, RounD, and NuScenes, across three common planning schemes: learning-based, polynomial-based, and optimization-based, all of which use the same training dataset for interaction-aware planning. Both quantitative results and qualitative comparisons with other epistemic uncertainty estimation algorithms indicate that our approach can estimate the feasibility of the AD’s planning in a planner-agnostic manner and ensure safety.

Advancements in autonomous driving technologies continue to revolutionize transportation, yet the full realization of self-driving vehicles remains hampered by several critical challenges. Automated vehicles continue to encounter significant challenges in perception, prediction, and decision-making, while their low-level modules are relatively mature and robust. Conversely, humans surpass machines in terms of high-level intelligence but may suffer from control performance degradation. To coalesce the strength of the human and machine, a novel collaboration scheme is proposed to compensate for the prediction-decision uncertainty via human guidance while providing low-level control feedback to the driver. This approach generates multiple decision candidates and corresponding predictions for other road users using a transformer-based socially compliant generative adversarial network (SCGAN). The driver can assist in choosing the appropriate candidate using the context-understanding capability, while concurrently, the control projection of this chosen decision guides the driver to achieve the desired objective via haptic steering feedback. The haptic feedback can reflect the decision uncertainties enabled by the decision-control projection of the intention estimation of the ego vehicle. A Type-II fuzzy controller is utilized to determine the control authority to account for the complexity of the future movement. We verify the effectiveness of the proposed algorithm through a real-time human-in-the-loop experiment, including an ablation study and comparisons with other human-machine collaboration schemes. The results demonstrate that the proposed scheme can minimize human-machine conflicts while increasing system safety.

Bayesian learning provides a unified skeleton to solve the electrophysiological source imaging task. From this perspective, existing source imaging algorithms utilize the Gaussian assumption for the observation noise to build the likelihood function for Bayesian inference. However, the electromagnetic measurements of brain activity are usually affected by miscellaneous artifacts, leading to a potentially non-Gaussian distribution for the observation noise. Hence the conventional Gaussian likelihood model is a suboptimal choice for the real-world source imaging task. In this study, we aim to solve this problem by proposing a new likelihood model which is robust with respect to non-Gaussian noises. Motivated by the robust maximum correntropy criterion, we propose a new improper distribution model concerning the noise assumption. This new noise distribution is leveraged to structure a robust likelihood function and integrated with hierarchical prior distributions to estimate source activities by variational inference. In particular, the score matching is adopted to determine the hyperparameters for the improper likelihood model. A comprehensive performance evaluation is performed to compare the proposed noise assumption to the conventional Gaussian model. Simulation results show that, the proposed method can realize more precise source reconstruction by designing known ground-truth. The real-world dataset also demonstrates the superiority of our new method with the visual perception task. This study provides a new backbone for Bayesian source imaging, which would facilitate its application using real-world noisy brain signal.

Ensuring safety and achieving human-level driving performance remain challenges for autonomous vehicles, especially in safety-critical situations. As a key component of artificial intelligence, reinforcement learning is promising and has shown great potential in many complex tasks; however, its lack of safety guarantees limits its real-world applicability. Hence, further advancing reinforcement learning, especially from the safety perspective, is of great importance for autonomous driving. As revealed by cognitive neuroscientists, the amygdala of the brain can elicit defensive responses against threats or hazards, which is crucial for survival in and adaptation to risky environments. Drawing inspiration from this scientific discovery, we present a fear-neuro-inspired reinforcement learning framework to realize safe autonomous driving through modeling the amygdala functionality. This new technique facilitates an agent to learn defensive behaviors and achieve safe decision making with fewer safety violations. Through experimental tests, we show that the proposed approach enables the autonomous driving agent to attain state-of-the-art performance compared to the baseline agents and perform comparably to 30 certified human drivers, across various safety-critical scenarios. The results demonstrate the feasibility and effectiveness of our framework while also shedding light on the crucial role of simulating the amygdala function in the application of reinforcement learning to safety-critical autonomous driving domains.

Accurate vision-based driver attention estimation is a challenging task due to the limitations of the visual sensor, and it is a critical and fundamental function of building a human-centered intelligent driving system. Unlike previous investigations which consider it a classification task, this study newly introduces scenario contextual information to improve the accuracy and obtain a fine-grained estimation. Therefore, a data-driven hybrid architecture for context-aware driver attention estimation is proposed to jointly model the scene and state of the driver during driving. A visual saliency map is typically assumed to highlight a distinct area that can capture human attention. To leverage this characteristic, a multi-hierarchy fusion network is proposed to extract effectively saliency features of a scene image. A gaze-tracking network is employed to estimate the potential focus zone of the driver, and this coarse estimation is optimized subsequently using the extracted saliency information to obtain a fine-grained estimation. Three related and commonly used task-agnostic and task-driven datasets are adopted to evaluate the proposed saliency estimation model, and experimental results show that it can achieve state-of-the-art performance. To verify the joint modeling methodology, two new driving attention datasets supplemented with driver information are collected based on the existing ones. The results of comparative experiments indicate that the consideration of saliency features can significantly improve the estimation performance of gaze fixation, demonstrating the feasibility and efficiency of the proposed method.

Accurate recognition of driver distraction is significant for the design of human-machine cooperation driving systems. Existing studies mainly focus on classifying varied distracted driving behaviors, which depend heavily on the scale and quality of datasets and only detect the discrete distraction categories. Therefore, most data-driven approaches have limited capability of recognizing unseen driving activities and cannot provide a reasonable solution for downstream applications. To address these challenges, this paper develops a vision Transformer-enabled weakly supervised contrastive (W-SupCon) learning framework, in which distracted behaviors are quantified by calculating their distances from the normal driving representation set. The Gaussian mixed model (GMM) is employed for the representation clustering, which centralizes the distribution of the normal driving representation set to better identify distracted behaviors. A novel driver behavior dataset and the other three ones are employed for the evaluation, experimental results demonstrate that our proposed approach has more accurate and robust performance than existing methods in the recognition of unknown driver activities. Furthermore, the rationality of distraction levels for different driving behaviors is evaluated through driver skeleton poses. The constructed dataset and demo videos are available athttps://yanghh.io/Driver-Distraction-Quantification.

Driver drowsiness detection is of great significance in improving driving safety and has been widely studied in recent years. However, some existing methods have not fully utilized the drowsiness-related information, and some methods are susceptible to interference from the redundant information of input data. To address these issues, a video-based driver drowsiness detection method according to the key facial features including facial landmarks and local facial areas (VBFLLFA) is proposed in this paper. In order to fully utilize the key facial features related to drowsiness and exclude the interference of redundant information, the head movement information is obtained through facial landmark analysis and the movement information of eyes and mouth is acquired from the local facial areas. And the spatial filtering based on the common spatial pattern (CSP) algorithm is introduced to improve the discrimination of different classes of samples. To adequately extract the temporal and spatial features, a two-branch multi-head attention (TB-MHA) module is designed in this paper. Furthermore, the center loss with center vector distance penalty is introduced to further improve the discrimination of different classes of samples in the feature space. In addition to two public datasets, we specifically create a novel video-based driver drowsiness detection (VBDDD) dataset to evaluate the effectiveness of our method. The experimental results verify that our method can achieve very excellent performance in driver drowsiness detection tasks.

Interest in autonomous driving (AD) and intelligent vehicles (IVs) is growing at a rapid pace due to the convenience, safety, and economic benefits. Although a number of surveys have reviewed research achievements in this field, they are still limited in specific tasks and lack systematic summaries and research directions in the future. Our work is divided into three independent articles and the first part is a survey of surveys (SoS) for total technologies of AD and IVs that involves the history, summarizes the milestones, and provides the perspectives, ethics, and future research directions. This is the second part (Part I for this technical survey) to review the development of control, computing system design, communication, high-definition map (HD map), testing, and human behaviors in IVs. In addition, the third part (Part II for this technical survey) is to review the perception and planning sections. The objective of this article is to involve all the sections of AD, summarize the latest technical milestones, and guide abecedarians to quickly understand the development of AD and IVs. Combining the SoS and Part II, we anticipate that this work will bring novel and diverse insights to researchers and abecedarians, and serve as a bridge between past and future.

To address the coordination issue of connected automated vehicles (CAVs) at urban scenarios, a game-theoretic decision-making framework is proposed that can advance social benefits, including the traffic system efficiency and safety, as well as the benefits of individual users. Under the proposed decision-making framework, in this work, a representative urban driving scenario, i.e. the unsignalized intersection, is investigated. Once the vehicle enters the focused zone, it will interact with other CAVs and make collaborative decisions. To evaluate the safety risk of surrounding vehicles and reduce the complexity of the decision-making algorithm, the driving risk assessment algorithm is designed with a Gaussian potential field approach. The decision-making cost function is constructed by considering the driving safety and passing efficiency of CAVs. Additionally, decision-making constraints are designed and include safety, comfort, efficiency, control and stability. Based on the cost function and constraints, the fuzzy coalitional game approach is applied to the decision-making issue of CAVs at unsignalized intersections. Two types of fuzzy coalitions are constructed that reflect both individual and social benefits. The benefit allocation in the two types of fuzzy coalitions is associated with the driving aggressiveness of CAVs. Finally, the effectiveness and feasibility of the proposed decision-making framework are verified with three test cases.

To address the safety and efficiency issues of vehicles at multi-lane merging zones, a cooperative decision-making framework is designed for connected automated vehicles (CAVs) using a coalitional game approach. Firstly, a motion prediction module is established based on the simplified single-track vehicle model for enhancing the accuracy and reliability of the decision-making algorithm. Then, the cost function and constraints of the decision making are designed considering multiple performance indexes, i.e. the safety, comfort and efficiency. Besides, in order to realize human-like and personalized smart mobility, different driving characteristics are considered and embedded in the modeling process. Furthermore, four typical coalition models are defined for CAVS at the scenario of a multi-lane merging zone. Then, the coalitional game approach is formulated with model predictive control (MPC) to deal with decision making of CAVs at the defined scenario. Finally, testings are carried out in two cases considering different driving characteristics to evaluate the performance of the developed approach. The testing results show that the proposed coalitional game based method is able to make reasonable decisions and adapt to different driving characteristics for CAVs at the multi-lane merging zone. It guarantees the safety and efficiency of CAVs at the complex dynamic traffic condition, and simultaneously accommodates the objectives of individual vehicles, demonstrating the feasibility and effectiveness of the proposed approach.

Driver attention modeling is a crucial technique in building human-centric intelligent driving systems. Considering the human visual mechanism, this study leverages multi-level visual content, including low-level texture features, middle-level optical flows, and high-level semantic information, as the model input. Subsequently, a heterogeneous model is proposed to handle the multi-level input, which integrates the graph and convolutional neural networks. Distinguished from the existing studies that use semantic segmentation, our study directly leverages the objection detection information in an interpretable manner. To deal with the detected objects, in this work, a graph attention network is used to explicitly construct the semantic information, rather than handle the features extracted by convolutional modules for building the latent space features, which are used in existing studies. Further, a semantic attention module is proposed to integrate the non-Euclidean output of the graph network with the Euclidean feature maps of the convolutional neural networks. Finally, these integrated features are decoded to generate a driver attention map. Three typical datasets are used to validate the proposed method. A comprehensive comparison and analysis have proven the feasibility and validity of our proposed method, as well as its ability to achieve state-of-the-art performance.

This paper investigates a human-machine cooperative trajectory planning and tracking control approach for automated vehicles. The proposed method is developed based on a novel algorithm of cooperative human-machine rapidly-exploring random (HM-RRT) for path planning, together with the risk assessment of driver behavior. First, the driver’s behaviour is assessed according to the information of the predicted vehicle trajectory, the identified safe driving area and the driving risks evaluated in both lateral and longitudinal directions. Based on the driver’s expected driving task, when driving risks are identified by real-time assessment, then the human-machine cooperation is activated during trajectory planning. By HM-RRT, the newly developed safety assurance mechanism for path planning, the cooperative trajectory is then generated, which incorporates the driver’s desire and actions and automation’s corrective actions, to ensure the safety, stability and smoothness of the human-vehicle system. The simulation and experimental results show that the proposed HM-RRT algorithm can effectively improve the convergence rate and reduce the computation load, comparing to the conventional method. Beyond this, the proposed human-machine cooperation approach is able to simultaneously ensure the safety, stability and smoothness of the vehicle and largely reduce human-machine conflicts in real-time applications, demonstrating its feasibility and effectiveness.

Considering that human-driven vehicles and autonomous vehicles (AVs) will coexist on roads in the future for a long time, how to merge AVs into human drivers' traffic ecology and minimize the effect of AVs and their misfit with human drivers, are issues worthy of consideration. Moreover, different passengers have different needs for AVs, thus, how to provide personalized choices for different passengers is another issue for AVs. Therefore, a human-like decision making framework is designed for AVs in this paper. Different driving styles and social interaction characteristics are formulated for AVs regarding driving safety, ride comfort and travel efficiency, which are considered in the modeling process of decision making. Then, Nash equilibrium and Stackelberg game theory are applied to the noncooperative decision making. In addition, potential field method and model predictive control (MPC) are combined to deal with the motion prediction and planning for AVs, which provides predicted motion information for the decision-making module. Finally, two typical testing scenarios of lane change, i.e., merging and overtaking, are carried out to evaluate the feasibility and effectiveness of the proposed decision-making framework considering different human-like behaviors. Testing results indicate that both the two game theoretic approaches can provide reasonable human-like decision making for AVs. Compared with the Nash equilibrium approach, under the normal driving style, the cost value of decision making using the Stackelberg game theoretic approach is reduced by over 20%.

Connected automated driving technologies have shown tremendous improvement in recent years. However, it is still not clear how driving behaviors and energy consumption correlate with each other and to what extent these factors related to connected vehicles can influence the motion prediction performance. The precise recognition of driving behaviors and prediction of the vehicle motion is critical to the driving safety for connected automated vehicles (CAVs). Hence, in this study, an energy-aware driving pattern analysis and motion prediction system are proposed for CAVs using a deep learning-based time-series modeling approach. First, energy-aware longitudinal acceleration and deceleration behaviors and lateral lane-change behaviors are statistically analyzed. Then, a sliding standard deviation (SSD) test is applied to evaluate the smoothness of the trajectory and velocity signals considering different energy consumption levels. An energy-aware personalized joint time-series modeling (PJTSM) approach based on a deep recurrent neural network (RNN) and long short-term memory (LSTM) cell are proposed for accurate motion (trajectory and velocity) prediction of the leading vehicle. Finally, the differences in the prediction performance regarding different energy consumption levels are compared and discussed. It is shown that due to the higher randomness of the driving behaviors, the prediction accuracy for heavy energy users is the lowest among the three categories, which means it is harder to anticipate the driving behaviors of cars exhibiting heavy energy consumption. The personalized estimation of driving behaviors of CAVs will contribute to safer automated driving and transportation systems.

The mutual understanding between driver and vehicle is critically important to the design of intelligent vehicles and customized interaction interface. In this study, a deep learning-based joint driver behavior reasoning system toward multi-scale and multi-tasks behavior recognition is proposed. Specifically, a multi-scale driver behavior recognition system is designed to recognize both the driver's physical and mental states based on a deep encoder-decoder framework. The system jointly recognizes three driver behaviors, namely, mirror-checking, lane change intention, and emotions based on the shared encoder network. The encoder network is designed based on a deep convolutional neural network (CNN), and several decoders for different driver states estimation are proposed with fully connected (FC), and long short-term memory (LSTM) based recurrent neural networks (RNN), respectively. The proposed framework can be used as a solution to exploit the relationship between different driver states for intelligent vehicles towards an efficient driver-side understanding. The testing results on the Brain4Car dataset show accurate performance and outperform existing methods on driver postures, intention, and emotion recognition.