Cross-dataset driver drowsiness recognition with EEG is important for the advancement of a calibration-free driver drowsiness recognition system. Nevertheless, this task is challenging due to the impact of distribution drift on recognition accuracy. In this paper, we propose a novel model named entropy optimization network (EON) for the task. The model takes a novel two-step strategy to separate the unlabeled data from the target domain. It firstly uses a novel modified entropy loss to encourage unlabeled samples well aligned with the source domain to form clear clusters. Next, it gradually separates samples from the target domain with a self-training framework by taking adequate advantage of underlying patterns inherent in it. The proposed method is tested on the domain adaptation task with two public datasets and achieves 2-class recognition accuracies of and , which beats other baseline methods. Our work illuminates a promising direction in achieving the ultimate objective of developing a driver drowsiness recognition system without calibration.

Developing an effective and efficient electroencephalography (EEG)-based drowsiness monitoring system is crucial for enhancing road safety and reducing the risk of accidents. For general usage, cross-subject evaluation is indispensable. Despite progress in unsupervised domain adaptation (UDA) and source-free domain adaptation (SFDA) methods, these often rely on the availability of labeled source data or white-box source models, posing potential privacy risks. This study explores a more challenging setting of UDA for EEG-based drowsiness detection, termed black-box domain adaptation (BBDA). In BBDA, adaptation in the target domain relies solely on a black-box source model, without access to the source data or parameters of the source model. Specifically, we propose a framework called Self-distillation and Pseudo-labelling for Ensemble Deep Random Vector Functional Link (edRVFL)-based Black-box Knowledge Adaptation (SPARK). SPARK employs entropy-based selection of high-confidence samples, which are then pseudo-labeled to train a student edRVFL network. Subsequently, ensemble self-distillation is performed to extract knowledge by training the edRVFL using refined labels introduced by ensemble learning. This process further improves the robustness of the student edRVFL network. The features of the edRVFL are beneficial for improving the computational efficiency of the framework, making it more suitable for tasks involving small datasets. The proposed SPARK framework is evaluated on two publicly available driver drowsiness datasets. Experimental results demonstrate its superior performance over strong baselines, while significantly reducing training time. These findings underscore the potential for practical integration of the proposed framework into drowsiness monitoring systems, thereby contributing substantially to the privacy preservation of source subjects.

Randomized neural networks (RNNs) have shown outstanding performance in many different fields. The superiority of having fewer training parameters and closed-form solutions makes them popular in small datasets analysis. However, automatically decoding raw electroencephalogram (EEG) data using RNNs is still challenging in EEG-based passive brain–computer interface (pBCI) classification tasks. Models with the high-dimension input of EEG may suffer from overfitting and the intrinsic characteristics of non-stationary, high-level noises and subject variability could limit the generation of distinctive features in the hidden layers. To address these problems in EEG-based pBCI tasks, this work proposes a spectral-ensemble deep random vector functional link (SedRVFL) network that focuses on feature learning in the frequency domain. Specifically, an unsupervised feature-refining (FR) block is proposed to improve the low feature learning capability in RNNs. Moreover, a dynamic direct link (DDL) is performed to further complement the frequency information. The proposed model has been evaluated on a self-collected dataset as well as a public driving dataset. The cross-subject classification results obtained demonstrated its effectiveness. This work offers a new solution for EEG decoding, i.e., using optimized RNNs for decoding complex raw EEG data and boosting the classification performance of EEG-based pBCI tasks.

In the context of electroencephalogram (EEG)-based driver drowsiness recognition, it is still challenging to design a calibration-free system, since EEG signals vary significantly among different subjects and recording sessions. Many efforts have been made to use deep learning methods for mental state recognition from EEG signals. However, existing work mostly treats deep learning models as black-box classifiers, while what have been learned by the models and to which extent they are affected by the noise in EEG data are still underexplored. In this article, we develop a novel convolutional neural network combined with an interpretation technique that allows sample-wise analysis of important features for classification. The network has a compact structure and takes advantage of separable convolutions to process the EEG signals in a spatial-temporal sequence. Results show that the model achieves an average accuracy of 78.35% on 11 subjects for leave-one-out cross-subject drowsiness recognition, which is higher than the conventional baseline methods of 53.40%-72.68% and state-of-the-art deep learning methods of 71.75%-75.19%. Interpretation results indicate the model has learned to recognize biologically meaningful features from EEG signals, e.g., alpha spindles, as strong indicators of drowsiness across different subjects. In addition, we also explore reasons behind some wrongly classified samples with the interpretation technique and discuss potential ways to improve the recognition accuracy. Our work illustrates a promising direction on using interpretable deep learning models to discover meaningful patterns related to different mental states from complex EEG signals.

Since situation awareness (SA) plays an important role in many fields, the measure of SA is one of the most concerning problems. Using physiological signals to evaluate SA is becoming a popular research topic because of their advantages of non-intrusiveness and objectivity. However, previous studies mainly exploited the use of single physiological signals such as electroencephalogram (EEG) or eye tracking. The multi-modal SA recognition is still a research gap. Therefore, this work conducts a pilot study to investigate SA recognition by using two modalities: EEG and eye tracking data. Specifically, an optimized Stroop test that is more compatible with the definition of SA was used to induce different states of SA and collect physiological data. Furthermore, a random vector functional link-based stacking (RVFL-S) model was proposed to perform the multi-modal SA recognition. Experiment results showed that using the combination of EEG and eye tracking data can boost the performance of SA recognition. Moreover, the proposed RVFL-S model can effectively integrate the classification information from two modalities. It showed better performance than baseline methods, achieving 77.62% leave-one-subject-out (LOSO) average accuracy. This was around 5% improvement compared with the baseline classification models with input of only one modality. This pilot study demonstrated that the use of multi-modality is a potential strategy for SA recognition.

For EEG-based drowsiness recognition, it is desirable to use subject-independent recognition since conducting calibration on each subject is time-consuming. In this paper, we propose a novel Convolutional Neural Network (CNN)-Long Short-Term Memory (LSTM) model for subject-independent drowsiness recognition from single-channel EEG signals. Different from existing deep learning models that are mostly treated as black-box classifiers, the proposed model can “explain” its decisions for each input sample by revealing which parts of the sample contain important features identified by the model for classification. This is achieved by a visualization technique by taking advantage of the hidden states output by the LSTM layer. Results show that the model achieves an average accuracy of 72.97% on 11 subjects for leave-one-out subject-independent drowsiness recognition on a public dataset, which is higher than the conventional baseline methods of 55.42%-69.27%, and state-of-the-art deep learning methods. Visualization results show that the model has discovered meaningful patterns of EEG signals related to different mental states across different subjects.

The maritime industry is switching to new types of fuel such as Liquefied Natural Gas (LNG). On one hand, these kinds of fuel are more sustainable to the environment, on the other hand, training on handling such fuel safely and dealing with emergency situation is necessary. Videos and lecture-based learning is commonly used to deliver such knowledge to the maritime trainees. In recent years, the advances in Virtual Reality (VR) have brought new opportunities to such training. It provides an immersive while safe environment for training on certain operations that are extraordinary or dangerous in real life. It also allows the learners to practice the tasks repeatedly. The VR-based training is mostly used for improving technical skills, however, to guarantee a more efficient and better assessment of trainee's performance, nontechnical skills such as decision making, situation awareness, vigilance are needed to be assessed and trained as well. In this paper, a VR-based LNG evacuation training system is presented. The system provides two training scenarios for learning the evacuation procedure. A novel human factors evaluation based on the behavioral data captured by VR was proposed and integrated with the training, which includes both technical and non-technical skills assessment. An experiment with 14 subjects was conducted to validate the human factors evaluation and to get feedback towards the VR-based training.

Nowadays, increasing attention has been drawn to hybrid 2D-3D visualization tools, while evaluating them with a convenient and objective tool has only been carried out in a small number of areas. In this study, a revised radar chart-based usability evaluation approach was proposed. The approach was adopted to evaluate the hybrid 2D-3D radar display in air traffic management. The holding stack in air traffic management is analyzed and simulated, two generic tasks are designed accordingly. The hybrid 2D-3D radar display settings are evaluated based on six indicators from eye-tracking and brain dynamics data, namely, the frequency of fixation, fixation mean duration, fixation time on an area of interest, emotion, workload, and stress. The results reveal that the hybrid 2D-3D radar display induces spatial memory loss and high workload, while requires a shorter fixation duration.

Abstract Shunting of trains is a task that requires meticulous adherence to all steps to guarantee safety for everyone involved during and after the procedure. These steps are currently taught using classical teaching materials, such as printouts, videos and training by experienced supervisors. However, due to limited availability of locomotives, hours for training and manpower, training of shunting operation becomes challenging in real life. In this paper, we implemented a lifelike, collaborative virtual environment for shunting training including a novel human factors evaluation system for fatigue and stress monitoring. An experiment with 12 subjects and 3 trainers has been designed and carried out to validate the usage of VR-based shunting training. Positive feedback toward the VR-based training was obtained from the subjects and trainers.

Extracting intra- and inter-subject parameters from Electroencephalogram (EEG) representing different Situation Awareness (SA) status is a critical challenge for objective SA recognition. Most of the existing work focuses on the subject-dependent classification that applies power spectrum density (PSD) features. In this paper, we propose a novel spectral-spatial (S-S) model for cross-subject fatigue-related SA recognition. The S-S model not only considers the biological topology across different brain regions to capture both local and global relations among different EEG channels, but also extracts spectral features for each EEG channel. Specifically, we firstly model the topological structure of EEG channels via an adjacency matrix which is built based on the Euclidean distance between EEG channels. Then, the graph convolution operation is employed to perform the neighbourhood aggregation for extracting spatial features. We test our model on a public dataset collected during driver’s task performance. The subject-independent performance of the model is explored. Results demonstrate (1) the superior performance of our model compared with the state-of-the-art models on SA recognition from EEG signals. Specifically, our S-S model achieves 70.6% accuracy which is higher than traditional machine learning methods by 2.7%-6.8% and deep learning methods by 10.3%-11.6%; (2) EEG signal at the occipital region can better reflect the change of SA.

Virtual Reality (VR) has been used for training aircraft pilots, maritime seafarers, operators, etc as it provides an immersive environment with realistic lifelike quality. We developed and implemented a VR-based Liquefied Natural Gas (LNG) firefighting simulation system with head-mounted displays (HMD) and novel human factors evaluation that could train and assess both technical and non-technical skills in the firefighting scenarios. The proposed human factors evaluation is based on a competence model and the non-technical skills such as situation awareness, vigilance, and decision making of seafarers could be assessed. An experiment was carried out with 6 trainees and 2 trainers using the implemented LNG firefighting simulation system. The results show that that the maritime trainees felt the VR scene was realistic to them, evoked similar emotions (such as fear, stress) during the demanding events as in the real world and made them attentive during the experience.

Methods from supervised machine learning allow the classification of new data automatically and are tremendously helpful for data analysis. The quality of supervised maching learning depends not only on the type of algorithm used, but also on the quality of the labelled dataset used to train the classifier. Labelling instances in a training dataset is often done manually relying on selections and annotations by expert analysts, and is often a tedious and time-consuming process. Active learning algorithms can automatically determine a subset of data instances for which labels would provide useful input to the learning process. Interactive visual labelling techniques are a promising alternative, providing effective visual overviews from which an analyst can simultaneously explore data records and select items to a label. By putting the analyst in the loop, higher accuracy can be achieved in the resulting classifier. While initial results of interactive visual labelling techniques are promising in the sense that user labelling can improve supervised learning, many aspects of these techniques are still largely unexplored. This paper presents a study conducted using the mVis tool to compare three interactive visualisations, similarity map, scatterplot matrix (SPLOM), and parallel coordinates, with each other and with active learning for the purpose of labelling a multivariate dataset. The results show that all three interactive visual labelling techniques surpass active learning algorithms in terms of classifier accuracy, and that users subjectively prefer the similarity map over SPLOM and parallel coordinates for labelling. Users also employ different labelling strategies depending on the visualisation used.

Mental fatigue is common at work places, and it can lead to decreased attention, vigilance and cognitive performance, which is dangerous in the situations such as driving, vessel maneuvering, etc. By directly measuring the neurophysiological activities happening in the brain, electroencephalography (EEG) signal can be used as a good indicator of mental fatigue. A classic EEG-based brain state recognition system requires labeled data from the user to calibrate the classifier each time before the use. For fatigue recognition, we argue that it is not practical to do so since the induction of fatigue state is usually long and weary. It is desired that the system can be calibrated using readily available fatigue data, and be applied to a new user with adequate recognition accuracy. In this paper, we explore performance of cross-subject fatigue recognition algorithms using the recently published EEG dataset labeled with two levels of fatigue. We evaluate three categories of classification method: classic classifier such as logistic regression, transfer learning-enabled classifier using transfer component analysis, and deep-learning based classifier such as EEGNet. Our results show that transfer learning-enabled classifier can outperform the other two for cross-subject fatigue recognition on a consistent basis. Specifically, transfer component analysis (TCA) improves the cross-subject recognition accuracy to 72.70 % that is higher than using just logistic regression (LR) by 9.08 % and EEGNet by 8.72 - 12.86 %.

Currently, many modern humanoid robots have little appeal due to their simple designs and bland appearances. To provide recommendations for designers and improve the designs of humanoid robots, a study of human's perception on humanoid robot designs is conducted using Electroencephalogram (EEG), eye tracking information and questionnaires. We proposed and carried out an experiment with 20 subjects to collect the EEG and eye tracking data to study their reaction to different robot designs and the corresponding preference towards these designs. This study can possibly give us some insights on how people react to the aesthetic designs of different humanoid robot models and the important traits in a humanoid robot design, such as the perceived smartness and friendliness of the robots. Another point of interest is to investigate the most prominent feature of the robot, such as the head, facial features and the chest. The result shows that the head and facial features are the focus. It is also discovered that more attention is paid to the robots that appear to be more appealing. Lastly, it is affirmed that the first impressions of the robots generally do not change over time, which may imply that a good humanoid robot design impress the observers at first sight.

Naturalness of the mid-air interaction interface for virtual assembly and shape modeling is important. In order to design an interface perceived as "natural" by most people, common behaviors and mental patterns for mid-air interaction of people have to be recognized, which is an area merely explored yet. This paper serves this purpose of understanding the users' mental interaction models, in order to provide standards and recommendation for devising a natural virtual interaction interface. We tested three kinds of tasks --- manipulating tasks, deforming tasks and tool-based operating tasks on 16 participants. We have found that: 1) different features of mental models were observed for different types of tasks. Interaction techniques should be designed to match these features; 2) virtual hand self-avatar helps estimate size of virtual objects, as well as helps plan and visualize the complex process and procedures of a task, which is especially helpful for tool-based tasks; 3) bimanual interaction is witnessed as a dominant interaction mode preferred by the majority; 4) natural gestures for deforming tasks always reflect forces exerted. These suggestions are useful for designing a midair interaction interface matching users' mental models.

In this paper, we propose a web-enabled shape modeling system with natural free-hand interaction, which can be easily learned by users while imposing least mental load on them. The deformation interface allows for performing various deformations, including stretching, compressing, squeezing, enlarging, twisting and tapering, on shapes interactively mimicking how they are done in real life. The manipulation interface allows an object to be directly grabbed and manipulated with either one or two hands, while also smoothly switching between them. Constrained methods are also provided for precise manipulation. An intuitive metaphor is designed to help the users to discover the interaction techniques by themselves without any manuals or instructions. A rendering pipeline, based on function-based extension of VRML/X3D, is designed with hidden complexity to support the proposed functionalities of the system. Hands motions are captured by Leap Motion controller. The user study proves the naturalness of the modeling system, and its easiness to be learned and remembered.

Common shape modelling is usually done with virtual tools controlled by interactive or hap tic devices, which have one interface point for simulating collision between the object and the modelling tool. Shape modeling with some kind of virtual hand where several fingers deform the object was not very common since the respective devices are either too expensive or not very precise. Introduction of affordable Leap Motion device opens new prospects for free hand shape modeling. This paper is a feasibility study on using Leap Motion for shape modeling in VRML/X3D environments where virtual objects are defined by mathematical functions and hence can easily be exchanged across the internet due to their small size.