Fatigue is one of the main factors that contribute to operator performance and maritime safety, making it important to develop fatigue recognition algorithms that can predict operator fatigue. In this paper. we propose a subject independent algorithm for fatigue recognition from heart rate using machine learning techniques for human factors evaluation. The final model with the Random Forest Classifier produced a mean classification accuracy of $67.2 \%$ for recognizing 2-levels of stress for unseen data. With a 1-minute data window for fatigue recognition updated every second, the proposed method could be applied for human factors evaluation including vessel traffic operators’ fatigue monitoring.

Being in a state of stress can affect operators’ performance and might lead to a decrease in attention and cognition, which could affect vessel operational safety within the Vessel Traffic Management System. By directly measuring operator’s biosignals, it is possible to predict the level of his/her stress for corrective action to be taken, such as getting support from colleagues if needed. One of the convenient ways to monitor operators’ stress is using mobile heart rate devices. In this paper, we propose a cross-subject stress recognition algorithm utilizing Heart Rate Variability (HRV) features and the Adaboost classifier. The algorithm is subject independent and can be calibrated using previously collected stress data to recognize the 2-level stress states of an unseen subject with 82.4% accuracy.

This work investigated the use of an ensemble deep random vector functional link (edRVFL) network for electroencephalogram (EEG)-based driver fatigue recognition. Against the low feature learning capability of the edRVFL network from raw EEG signals, two strategies were exploited in this work. Specifically, the first one was to exploit the advantages of the feature extractor module in CNNs, i.e., use CNN features as the input of the edRVFL network. The second one was to improve the feature learning capability of the edRVFL network. An enhanced edRFVL network named FGloWD-edRVFL was proposed, in which four enhancements were implemented, including random forest-based Feature selection, Global output layer, Weighting and entropy-based Dynamic ensemble. The proposed FGloWD-edRVFL network was evaluated on the challenging cross-subject driver fatigue recognition tasks. The results indicated that the proposed model could boost the recognition performance, significantly outperforming all strong baselines. The step-wise analysis further demonstrated the effectiveness of the proposed enhancements in the edRVFL network.

Randomized neural networks (RNNs) have shown outstanding performance in many different fields. The superiority of having fewer training parameters and closed-form solutions makes them popular in small datasets analysis. However, automatically decoding raw electroencephalogram (EEG) data using RNNs is still challenging in EEG-based passive brain–computer interface (pBCI) classification tasks. Models with the high-dimension input of EEG may suffer from overfitting and the intrinsic characteristics of non-stationary, high-level noises and subject variability could limit the generation of distinctive features in the hidden layers. To address these problems in EEG-based pBCI tasks, this work proposes a spectral-ensemble deep random vector functional link (SedRVFL) network that focuses on feature learning in the frequency domain. Specifically, an unsupervised feature-refining (FR) block is proposed to improve the low feature learning capability in RNNs. Moreover, a dynamic direct link (DDL) is performed to further complement the frequency information. The proposed model has been evaluated on a self-collected dataset as well as a public driving dataset. The cross-subject classification results obtained demonstrated its effectiveness. This work offers a new solution for EEG decoding, i.e., using optimized RNNs for decoding complex raw EEG data and boosting the classification performance of EEG-based pBCI tasks.

In the context of electroencephalogram (EEG)-based driver drowsiness recognition, it is still challenging to design a calibration-free system, since EEG signals vary significantly among different subjects and recording sessions. Many efforts have been made to use deep learning methods for mental state recognition from EEG signals. However, existing work mostly treats deep learning models as black-box classifiers, while what have been learned by the models and to which extent they are affected by the noise in EEG data are still underexplored. In this article, we develop a novel convolutional neural network combined with an interpretation technique that allows sample-wise analysis of important features for classification. The network has a compact structure and takes advantage of separable convolutions to process the EEG signals in a spatial-temporal sequence. Results show that the model achieves an average accuracy of 78.35% on 11 subjects for leave-one-out cross-subject drowsiness recognition, which is higher than the conventional baseline methods of 53.40%-72.68% and state-of-the-art deep learning methods of 71.75%-75.19%. Interpretation results indicate the model has learned to recognize biologically meaningful features from EEG signals, e.g., alpha spindles, as strong indicators of drowsiness across different subjects. In addition, we also explore reasons behind some wrongly classified samples with the interpretation technique and discuss potential ways to improve the recognition accuracy. Our work illustrates a promising direction on using interpretable deep learning models to discover meaningful patterns related to different mental states from complex EEG signals.

Since situation awareness (SA) plays an important role in many fields, the measure of SA is one of the most concerning problems. Using physiological signals to evaluate SA is becoming a popular research topic because of their advantages of non-intrusiveness and objectivity. However, previous studies mainly exploited the use of single physiological signals such as electroencephalogram (EEG) or eye tracking. The multi-modal SA recognition is still a research gap. Therefore, this work conducts a pilot study to investigate SA recognition by using two modalities: EEG and eye tracking data. Specifically, an optimized Stroop test that is more compatible with the definition of SA was used to induce different states of SA and collect physiological data. Furthermore, a random vector functional link-based stacking (RVFL-S) model was proposed to perform the multi-modal SA recognition. Experiment results showed that using the combination of EEG and eye tracking data can boost the performance of SA recognition. Moreover, the proposed RVFL-S model can effectively integrate the classification information from two modalities. It showed better performance than baseline methods, achieving 77.62% leave-one-subject-out (LOSO) average accuracy. This was around 5% improvement compared with the baseline classification models with input of only one modality. This pilot study demonstrated that the use of multi-modality is a potential strategy for SA recognition.

Deep learning for electroencephalogram-based classification is confronted with data scarcity, due to the time-consuming and expensive data collection procedure. Data augmentation has been shown as an effective way to improve data efficiency. In addition, contrastive learning has recently been shown to hold great promise in learning effective representations without human supervision, which has the potential to improve the electroencephalogram-based recognition performance with limited labeled data. However, heavy data augmentation is a key ingredient of contrastive learning. In view of the limited number of sample-based data augmentation in electroencephalogram processing, three methods, performance-measure-based time warp, frequency noise addition and frequency masking, are proposed based on the characteristics of electroencephalogram signal. These methods are parameter learning free, easy to implement, and can be applied to individual samples. In the experiment, the proposed data augmentation methods are evaluated on three electroencephalogram-based classification tasks, including situation awareness recognition, motor imagery classification and brain-computer interface steady-state visually evoked potentials speller system. Results demonstrated that the convolutional models trained with the proposed data augmentation methods yielded significantly improved performance over baselines. In overall, this work provides more potential methods to cope with the problem of limited data and boost the classification performance in electroencephalogram processing.

For EEG-based drowsiness recognition, it is desirable to use subject-independent recognition since conducting calibration on each subject is time-consuming. In this paper, we propose a novel Convolutional Neural Network (CNN)-Long Short-Term Memory (LSTM) model for subject-independent drowsiness recognition from single-channel EEG signals. Different from existing deep learning models that are mostly treated as black-box classifiers, the proposed model can “explain” its decisions for each input sample by revealing which parts of the sample contain important features identified by the model for classification. This is achieved by a visualization technique by taking advantage of the hidden states output by the LSTM layer. Results show that the model achieves an average accuracy of 72.97% on 11 subjects for leave-one-out subject-independent drowsiness recognition on a public dataset, which is higher than the conventional baseline methods of 55.42%-69.27%, and state-of-the-art deep learning methods. Visualization results show that the model has discovered meaningful patterns of EEG signals related to different mental states across different subjects.

The maritime industry is switching to new types of fuel such as Liquefied Natural Gas (LNG). On one hand, these kinds of fuel are more sustainable to the environment, on the other hand, training on handling such fuel safely and dealing with emergency situation is necessary. Videos and lecture-based learning is commonly used to deliver such knowledge to the maritime trainees. In recent years, the advances in Virtual Reality (VR) have brought new opportunities to such training. It provides an immersive while safe environment for training on certain operations that are extraordinary or dangerous in real life. It also allows the learners to practice the tasks repeatedly. The VR-based training is mostly used for improving technical skills, however, to guarantee a more efficient and better assessment of trainee's performance, nontechnical skills such as decision making, situation awareness, vigilance are needed to be assessed and trained as well. In this paper, a VR-based LNG evacuation training system is presented. The system provides two training scenarios for learning the evacuation procedure. A novel human factors evaluation based on the behavioral data captured by VR was proposed and integrated with the training, which includes both technical and non-technical skills assessment. An experiment with 14 subjects was conducted to validate the human factors evaluation and to get feedback towards the VR-based training.

Nowadays, increasing attention has been drawn to hybrid 2D-3D visualization tools, while evaluating them with a convenient and objective tool has only been carried out in a small number of areas. In this study, a revised radar chart-based usability evaluation approach was proposed. The approach was adopted to evaluate the hybrid 2D-3D radar display in air traffic management. The holding stack in air traffic management is analyzed and simulated, two generic tasks are designed accordingly. The hybrid 2D-3D radar display settings are evaluated based on six indicators from eye-tracking and brain dynamics data, namely, the frequency of fixation, fixation mean duration, fixation time on an area of interest, emotion, workload, and stress. The results reveal that the hybrid 2D-3D radar display induces spatial memory loss and high workload, while requires a shorter fixation duration.

Abstract Shunting of trains is a task that requires meticulous adherence to all steps to guarantee safety for everyone involved during and after the procedure. These steps are currently taught using classical teaching materials, such as printouts, videos and training by experienced supervisors. However, due to limited availability of locomotives, hours for training and manpower, training of shunting operation becomes challenging in real life. In this paper, we implemented a lifelike, collaborative virtual environment for shunting training including a novel human factors evaluation system for fatigue and stress monitoring. An experiment with 12 subjects and 3 trainers has been designed and carried out to validate the usage of VR-based shunting training. Positive feedback toward the VR-based training was obtained from the subjects and trainers.

Extracting intra- and inter-subject parameters from Electroencephalogram (EEG) representing different Situation Awareness (SA) status is a critical challenge for objective SA recognition. Most of the existing work focuses on the subject-dependent classification that applies power spectrum density (PSD) features. In this paper, we propose a novel spectral-spatial (S-S) model for cross-subject fatigue-related SA recognition. The S-S model not only considers the biological topology across different brain regions to capture both local and global relations among different EEG channels, but also extracts spectral features for each EEG channel. Specifically, we firstly model the topological structure of EEG channels via an adjacency matrix which is built based on the Euclidean distance between EEG channels. Then, the graph convolution operation is employed to perform the neighbourhood aggregation for extracting spatial features. We test our model on a public dataset collected during driver’s task performance. The subject-independent performance of the model is explored. Results demonstrate (1) the superior performance of our model compared with the state-of-the-art models on SA recognition from EEG signals. Specifically, our S-S model achieves 70.6% accuracy which is higher than traditional machine learning methods by 2.7%-6.8% and deep learning methods by 10.3%-11.6%; (2) EEG signal at the occipital region can better reflect the change of SA.

Virtual Reality (VR) has been used for training aircraft pilots, maritime seafarers, operators, etc as it provides an immersive environment with realistic lifelike quality. We developed and implemented a VR-based Liquefied Natural Gas (LNG) firefighting simulation system with head-mounted displays (HMD) and novel human factors evaluation that could train and assess both technical and non-technical skills in the firefighting scenarios. The proposed human factors evaluation is based on a competence model and the non-technical skills such as situation awareness, vigilance, and decision making of seafarers could be assessed. An experiment was carried out with 6 trainees and 2 trainers using the implemented LNG firefighting simulation system. The results show that that the maritime trainees felt the VR scene was realistic to them, evoked similar emotions (such as fear, stress) during the demanding events as in the real world and made them attentive during the experience.

Mental fatigue is common at work places, and it can lead to decreased attention, vigilance and cognitive performance, which is dangerous in the situations such as driving, vessel maneuvering, etc. By directly measuring the neurophysiological activities happening in the brain, electroencephalography (EEG) signal can be used as a good indicator of mental fatigue. A classic EEG-based brain state recognition system requires labeled data from the user to calibrate the classifier each time before the use. For fatigue recognition, we argue that it is not practical to do so since the induction of fatigue state is usually long and weary. It is desired that the system can be calibrated using readily available fatigue data, and be applied to a new user with adequate recognition accuracy. In this paper, we explore performance of cross-subject fatigue recognition algorithms using the recently published EEG dataset labeled with two levels of fatigue. We evaluate three categories of classification method: classic classifier such as logistic regression, transfer learning-enabled classifier using transfer component analysis, and deep-learning based classifier such as EEGNet. Our results show that transfer learning-enabled classifier can outperform the other two for cross-subject fatigue recognition on a consistent basis. Specifically, transfer component analysis (TCA) improves the cross-subject recognition accuracy to 72.70 % that is higher than using just logistic regression (LR) by 9.08 % and EEGNet by 8.72 - 12.86 %.

Currently, many modern humanoid robots have little appeal due to their simple designs and bland appearances. To provide recommendations for designers and improve the designs of humanoid robots, a study of human's perception on humanoid robot designs is conducted using Electroencephalogram (EEG), eye tracking information and questionnaires. We proposed and carried out an experiment with 20 subjects to collect the EEG and eye tracking data to study their reaction to different robot designs and the corresponding preference towards these designs. This study can possibly give us some insights on how people react to the aesthetic designs of different humanoid robot models and the important traits in a humanoid robot design, such as the perceived smartness and friendliness of the robots. Another point of interest is to investigate the most prominent feature of the robot, such as the head, facial features and the chest. The result shows that the head and facial features are the focus. It is also discovered that more attention is paid to the robots that appear to be more appealing. Lastly, it is affirmed that the first impressions of the robots generally do not change over time, which may imply that a good humanoid robot design impress the observers at first sight.

To deal with the increasing demands in Air Traffic Control (ATC), new working place designs are proposed and developed that need novel human factors evaluation tools. In this paper, we propose a novel application of Electroencephalogram (EEG)-based emotion, workload, and stress recognition algorithms to investigate the optimal length of training for Air Traffic Control Officers (ATCOs) to learn working with three-dimensional (3D) display as a supplementary to the existing 2D display. We tested and applied the state-of-the-art EEG-based subject-dependent algorithms. The following experiment was carried out. Twelve ATCOs were recruited to take part in the experiment. The participants were in charge of the Terminal Control Area, providing navigation assistance to aircraft departing and approaching the airport using 2D and 3D displays. EEG data were recorded, and traditional human factors questionnaires were given to the participants after 15-minute, 60-minute, and 120-minute training. Different from the questionnaires, the EEG-based evaluation tools allow the recognition of emotions, workload, and stress with different temporal resolutions during the task performance by subjects. The results showed that 50-minute training could be enough for the ATCOs to learn the new display setting as they had relatively low stress and workload. The study demonstrated that there is a potential of applying the EEG-based human factors evaluation tools to assess novel system designs in addition to traditional questionnaire and feedback, which can be beneficial for future improvements and developments of the systems and interfaces.

Affective brain-computer interface (aBCI) introduces personal affective factors into human-computer interactions, which could potentially enrich the user's experience during the interaction with a computer. However, affective neural patterns are volatile even within the same subject. To maintain satisfactory emotion recognition accuracy, the state-of-the-art aBCIs mainly tailor the classifier to the subject-of-interest and require frequent re-calibrations for the classifier. In this paper, we demonstrate that the recognition accuracy of aBCIs deteriorates when re-calibration is ruled out during the long-term usage for the same subject. Then, we propose a stable feature selection method to choose the most stable affective features, for mitigating the accuracy deterioration to a lesser extent and maximizing the aBCI performance in the long run. We validate our method on a dataset comprising six subjects' EEG data collected during two sessions per day for each subject for eight consecutive days.

For practical applications, it is desirable for a trained classification system to be independent of task and/or subject. In this study, we show one-way transfer between two independent EEG workload datasets: from a large multitasking dataset with 48 subjects to a second Stroop test dataset with 18 subjects. This was achieved with a classification system trained using sparse encoded representations of the decomposed wavelets in the alpha, beta and theta power bands, which learnt a feature representation that outperformed benchmark power spectral density features by 3.5%. We also explore the possibility of enhancing performance with the utilization of domain adaptation techniques using transfer component analysis (TCA), obtaining 30.0% classification accuracy for a 4-class cross dataset problem.

When people are exhausted both physically and mentally from overexertion, they experience fatigue. Fatigue can lead to a decrease in motivation and vigilance which may result in certain accidents or injuries. It is crucial to monitor fatigue in workplace for safety reasons and well-being of the workers. In this paper, Electroencephalogram (EEG)-based evaluation of mental fatigue is investigated using the state-of-the-art machine learning algorithms. An experiment lasted around 2 hours and 30 minutes was designed and carried out to induce four levels of fatigue and collect EEG data from seven subjects. The results show that for subject-dependent 4-level fatigue recognition, the best average accuracy of 93.45% was achieved by using 6 statistical features with a linear SVM classifier. With subject-independent approach, the best average accuracy of 39.80% for 4 levels was achieved by using fractal dimension, 6 statistical features and a linear discriminant analysis classifier. The EEG-based fatigue recognition has the potential to be used in workplace such as cranes to monitor the fatigue of operators who are often subjected to long working hours with heavy workloads.

Deep investment in the maritime industries has led to many cutting edge technological advances in shipping navigation and operational safety to ensure safe and efficient logistical transportations. However, even with the best technology equipped onboard, maritime accidents are still occurring with at least three quarters of them attributed to human errors. Due to the rising need to address the human factors in shipping operations, various human factors studies are conducted in maritime domain. In this paper, an Electroencephalogram (EEG)-based cadets training and performance assessment system is proposed and implemented that could be used in the maritime virtual simulator. The system includes an EEG processing and analyses part and an evaluation part. It could recognize the brain states such as mental workload, emotions, and stress from raw EEG signal recorded during the exercises in the simulator and then give an indicative recommendation on "pass", "retrain", or "fail" of the cadet based on the EEG recognition results and input of the level of the task difficulty performed.

Spectral band power features are one of the most widely used features in the studies of electroencephalogram (EEG)-based emotion recognition. The power spectral density of EEG signals is partitioned into different bands such as delta, theta, alpha and beta band etc. Though based on neuroscientific findings, the partition of frequency bands is somewhat on an ad-hoc basis, and the definition of frequency ranges of the bands of interest can vary between studies. On the other hand, it is also arguable that one definition of power bands could perform equally well on all subjects. In this paper, we propose to use autoencoder to automatically learn from each subject the salient frequency components from power spectral density estimated as periodogram by Fast Fourier Transform (FFT). We propose a network architecture especially for EEG feature extraction, one that adopts hidden unit clustering with added pooling neuron per cluster. The classification accuracy with features extracted by our proposed method is benchmarked against that with standard power features. Experimental results show that our proposed feature extraction method achieves accuracy ranging from 44% to 59% for three-emotion classification. We also see a 4-20% accuracy improvement over standard band power features.

Many studies have shown that the majority of maritime accidents/incidents are attributed to human errors as the initiating cause. Efforts have been made to study human factors that can result in a safer maritime transportation. Among all techniques, Electroencephalogram (EEG) has the advantages such as high time resolution, possibility to continuously monitor brain states with high accuracy, recognition of human mental workload, emotion, stress, vigilance, etc. In this paper, we designed and carried out an experiment to collect the EEG signals to study stress and sharing of the mental workload among crew members during collaboration tasks performance on the ship's bridge virtual simulator. Four maritime trainees were monitored in the experiment. Each of them had a role such as an officer on watch, captain, pilot, or steersman. The results show that the captain had the highest stress and workload. However, the other three trainees experienced low workload and stress due to shared work and responsibility. The EEG is a promising evaluation tool to be used in the human factors study in the maritime domain.

Neurofeedback training is one type of the biofeedback training that allows the subject do self-regulation during the training according to his/her real-time brain activities recognized from Electroencephalogram (EEG) and given to him/her through visual, audio or haptic feedback. The Neurofeedback training has been proved to be helpful in improvement of cognitive abilities not only for patients with mental illnesses but also for healthy subjects including athletes. In this paper, we proposed an experiment in which we recruited elite shooters and conducted a novel individual beta-1/theta based neurofeedback training to confirm the usage of neurofeedback training in boosting the performance of rifle shooters. The efficiency of the neurofeedback training was examined by comparing the shooting scores and results of DAUF test assessing the ability of sustained attention of the shooters before and after neurofeedback training.

With the growing volume and complexity of air traffic, air traffic controllers (ATCOs) encounter heavier burden nowadays. Therefore, human factors study in air traffic control (ATC) is increasingly essential, paving the way to a safer air transportation system. In this paper, we conducted an ATC experiment, where Electroencephalogram (EEG) data were collected throughout the experiment. Compared to traditional questionnaires and psychological tests used in human factors study, the proposed novel EEG approach provides monitoring of situation awareness (SA) in a non-invasive and non-interruptive fashion. SA was represented as the response latency in situation-present assessment method (SPAM), which was predicted from EEG signals using three machine learning algorithms. Support vector regression obtained the lowest prediction error of 1.5 seconds, which is lower than 10% of the range of actual response latency. The results show that EEG is a promising approach forward in measuring situation awareness of ATCOs in both real-time and accurate manner.

Neuroscience-based or neuroscience-informed design is a new application area of Brain-Computer Interaction (BCI). It takes its roots in study of human well-being in architecture, human factors study in engineering and manufacturing including neuroergonomics. In traditional human factors studies and/or well-being study, mental workload, stress, and emotion are obtained through questionnaires that are administered upon completion of some task and/or the whole experiment. Recent advances in BCI research allow for using Electroencephalogram (EEG) based brain state recognition algorithms to assess the interaction between brain and human performance. We propose and develop an EEG-based system CogniMeter to monitor and analyze human factors measurements of newly designed software/hardware systems and/or working places. Machine learning techniques are applied to the EEG data to recognize levels of mental workload, stress and emotions during each task. The EEG is used as a tool to monitor and record the brain states of subjects during human factors study experiments. We describe two applications of CogniMeter system: human performance assessment in maritime simulator and EEG-based human factors evaluation in Air Traffic Control (ATC) workplace. By utilizing the proposed EEG-based system, true understanding of subjects working patterns can be obtained. Based on the analyses of the objective real time EEG-based data together with the subjective feedback from the subjects, we are able to reliably evaluate current systems/hardware and/or working place design and refine new concepts and design of future systems.

Emotion recognition is an integral part of affective computing. An affective brain-computer-interface (BCI) can benefit the user in a number of applications. In most existing studies, EEG (electroencephalograph)-based emotion recognition is explored in a classificatory manner. In this manner, human emotions are discretized by a set of emotion labels. However, human emotions are more of a continuous phenomenon than discrete. A regressive approach is more suited for continuous emotion recognition. Few studies have looked into a regressive approach. In this study, we investigate a portfolio of EEG features including fractal dimension, statistics and band power. Support vector regression (SVR) is employed in this study to estimate subject's valence level by means of different features under two evaluation schemes. In the first scheme, a SVR is constructed with full training resources, whereas in the second scheme, a SVR only receives minimal training resources. MAE (mean absolute error) averages of 0.74 and 1.45 can be achieved under the first and the second scheme, respectively, by fractal feature. The advantages of a regressive approach over classificatory approach lie in continuous emotion recognition and the possibility to reduce training resources to minimal level.

The individual alpha peak frequency (IAPF) is an important biological indicator in Electroencephalogram (EEG) studies, with many research publications linking it to various cognitive functions. In this paper, we propose novel Power Spectral Density (PSD) alpha features based on IAPF to classify 2 and 4 levels of EEG multitasking workload data. When optimized IAPF was considered, a 1.55% and 1.56% increase in average accuracy for 48 subjects' data, with 35 and 33 subjects showing improvement was observed for 2 and 4 class cases respectively. This trend suggests that individual specific features are able to improve classification performance compared to generalized features for subject dependent cases. The proposed features, which incorporates the biological meaning of the IAPF and provides subject specific information, can be considered as a viable alternative to the general alpha power feature when designing novel subject dependent feature sets for BCI workload recognition applications.

Haptic interaction is a form of a user-computer interaction where physical forces are delivered to the user via vibrations, displacements and rotations of special haptic devices. When quality of the experience of the haptic interaction is assessed, mostly subjective tests using various questionnaires are performed. We proposed novel neurocognitive tools for assessing both overall experience of the haptic interaction, as well as particular time-stamped activities. Our assessment tools are based on recognition of emotions and stress obtained from Electroencephalograms (EEG). We used them in a feasibility study on adding haptic interaction to Skype video conversation.

Polychronous neuronal group (PNG), a type of cell assembly, is one of the putative mechanisms for neural information representation. According to the reader-centric definition, some readout neurons can become selective to the information represented by polychronous neuronal groups under ongoing activity. Here, in computational models, we show that the frequently activated polychronous neuronal groups can be learned by readout neurons with joint weight-delay spike-timing-dependent plasticity. The identity of neurons in the group and their expected spike timing at millisecond scale can be recovered from the incoming weights and delays of the readout neurons. The detection performance can be further improved by two layers of readout neurons. In this way, the detection of polychronous neuronal groups becomes an intrinsic part of the network, and the readout neurons become differentiated members in the group to indicate whether subsets of the group have been activated according to their spike timing. The readout spikes representing this information can be used to analyze how PNGs interact with each other or propagate to downstream networks for higher-level processing.

Real-time EEG (Electroencephalogram)-based user's emotion, mental workload and stress monitoring is a new direction in research and development of human-machine interfaces. It has attracted recently more attention from the research community and industry as wireless portable EEG devices became easily available on the market. EEG-based technology has been applied in anesthesiology, psychology, serious games or even in marketing. In this work, we describe available real-time algorithms of emotion recognition, mental workload, and stress recognition from EEG and propose a novel interface Cogni Meter for the user's mental state visual monitoring. The system can be used in real time to assess human current emotions, levels of mental workload and stress. Currently, it is applied to monitor the user's emotional state, mental workload and stress in simulation scenarios or used as a tool to assess the subject's mental state in human factor study experiments.

Multitasking is a prevalent phenomenon in our daily lives. Certain occupations, especially in the aviation industry, consider proficient multitasking as a key skill set in their hiring process for pilot or air traffic controller candidates. There is a growing interest in the testing and training of the multitasking ability, with in house software or commercial psychological products, usually implemented in a static task battery format. In this paper, we propose a 3D game, Multitask In Neurofeedback Driving (MIND) for training and testing of the multitasking ability. The game is developed using the Unreal 3 game engine and incorporates neurofeedback, a technique used in the training of human cognitive abilities, to further enhance the potential benefits of the training procedure. The tasks used in the multitasking condition are inspired by various psychological tests and implemented in a manner that attempts to simulate the general cognitive processes required for multitasking while driving a vehicle or piloting an aircraft. The game comes in three variants, single task condition, multitasking condition and multitasking with neurofeedback condition, for the purpose of validating the training outcomes in future studies.

The difference in cognitive abilities of humans could be assessed by indexes extracted from EEG. In this paper, we propose and implement an experiment with 60 subjects to study how cognitive abilities can be identified through EEG. We analyzed parameters of the individual frequency band that can be used for prediction of cognitive abilities of subjects. In the experiment, the subjects performed cognitive tests with EEG recording done prior to the tests. Different patterns of alpha band activity are proven to be indicators of cognitive abilities and performances. Our hypothesis is that cognitive abilities can be predicted based on EEG measurements. The results of analysis of the experiment show significant correlation between subjects' cognitive abilities assessed by the tests and the EEG measurements.

Due to the precise spike timing in neural coding, spiking neural network (SNN) possesses richer spatiotemporal dynamics compared to neural networks with firing rate coding. One of the distinct features of SNN, polychronous neuronal group (PNG), receives much attention from both computational neuroscience and machine learning communities. However, all existing algorithms detect PNGs from the spike recording collected after simulation in an offline manner. There is currently no algorithm that detects PNGs actually being activated in runtime (online manner), which could be potentially used as inputs to higher level neural processing. We propose a runtime detection algorithm particularly for activated PNGs, using PNG readout neurons, to fill this gap. The proposed algorithm can reveal the spatiotemporal PNG patterns embedded in spike trains, which is higher level neuronal dynamics. We demonstrate through an example that for composed input patterns, new PNGs except the constituent PNGs can be easily found using the proposed algorithm. As an important interpretation, we give further insights on how to use PNG readout neurons to construct layered network structure.

In this paper, we proposed a driver drowsiness detection method for which only eyelid movement information was required. The proposed method consists of two major parts. 1) In order to obtain accurate eye openness estimation, a vision based eye openness recognition method was proposed to obtain an regression model that directly gave degree of eye openness from a low-resolution eye image without complex geometry modeling, which is efficient and robust to degraded image quality. 2) A novel feature extraction method based on unsupervised learning was also proposed to reveal hidden pattern from eyelid movements as well as reduce the feature dimension. The proposed method was evaluated and shown good performance.

Everyone experiences stress in life. Moderate stress can be beneficial to human, however, excessive stress is harmful to the health. To monitor stress, different methods can be used. In this work, an algorithm for stress level recognition from Electroencephalogram (EEG) is proposed. To validate the algorithm, an experiment is designed and carried out with 9 subjects. A Stroop colour-word test is used as a stressor to induce 4 levels of stress, and the EEG data are recorded during the experiment. Different feature combinations and classifiers are proposed and analyzed. By combining fractal dimension and statistical features and using Support Vector Machine (SVM) as the classifier, four levels of stress can be recognized with an average accuracy of 67.06%, three levels of stress can be recognized with an accuracy of 75.22%, and two levels of stress can be recognized with an accuracy of 85.71%. The algorithm is integrated into the system CogniMeter for stress state monitoring. Stress level of the user is visualized on the meter in real time. The system can be applied for stress monitoring of air traffic controllers, operators, etc.

Recently, new interactive devices such as hap tic devices became available for game development. 6DOF hap tic devices give the user an opportunity "to feel" the simulated virtual environment in the way similar to the real world. Hap tic-based interaction can add a new dimension to "serious games" development. The user can "feel" objects surfaces and complex objects interaction forces in a 3D virtual environment. In this paper, we propose two hap tic-based serious games. In the first "T Puzzle" game, the user can "feel" the weight of objects, rotate and move the 3D puzzle pieces in the virtual world. This game can be used to improve the user's spatial abilities. In the second "Mol Docking" game, the player can feel interaction forces between molecular systems and learn the process of molecular docking in collaborative virtual environment.

Stability of algorithms is very important for electroencephalogram (EEG) based applications. Stable features should exhibit consistency among repeated measurements of the same subject. Previously, power features were reported to be one of the most stable EEG features in medical application. In this paper, stability of features in emotion recognition algorithms is studied. Our hypothesis is that the most stable features give the best intra-subject accuracy across different days in real-time emotion recognition algorithm. An experiment to induce 4 emotions such as pleasant, happy, frightened, and angry is designed and carried out in 8 consecutive days (two sessions per day) for 4 subjects to record EEG data. A novel real-time subject dependent algorithm with the most stable features is proposed and implemented. The algorithm needs just one training for each subject. The training results can be used in real-time emotion recognition applications without re-training with the adequate accuracy. The proposed algorithm is integrated with a real-time application "Emotional Avatar".

Neurofeeback training can be used to enhance cognitive abilities related to multi-tasking such as working memory, attention, etc. We propose and implement a neurofeedback system which includes a number of neurofeedback training algorithms and a Shooting game. To make neurofeedback training more effective, an Individual Alpha Peak (IAP) frequency and individual alpha bandwidth are calculated and applied in the algorithms. We do preliminary study on the effectiveness of the proposed neurofeedback system with three subjects taking 6 sessions each. The neurofeedback protocols based on the power of individual upper alpha or beta-1/theta ratio training are used. Our hypothesis is that after the neurofeedback training by playing the Shooting game, the individual alpha peak frequency increases. The results show that all subjects had a higher individual alpha peak frequency after the training that indicated an enhancement of the subjects' cognitive abilities related to multi-tasking.

In this paper, a real-time Electroencephalogram (EEG)-based emotion recognition algorithm using Higuchi Fractal Dimension (FD) Spectrum is proposed. As EEG is a nonlinear and multi-fractal signal, its FD spectrum can give a better understanding of the nonlinear property of EEG. Three values are selected from the whole spectrum and are combined with the other features such as statistical and Higher Order Crossings ones. The Support Vector Machine is used as the classifier. The proposed algorithm is validated on both benchmark database DEAP with video stimuli and our own dataset which used visual stimuli to evoke emotions. Up to 8 emotions can be recognized with only 4 channels. The experiment analysis results show that using FD spectrum features it is possible to improve classification accuracy.

Nowadays, advertising is a part of our daily life driving consumer behavior, social behavior, personal preferences, etc. As volumes of advertisements increase people become more immune to different types of advertisements. To make the advertisements more efficient is a challenging problem. It is confirmed in the experiments that the emotions felt by the participants during the viewing of an advertisement influence on the effectiveness of the advertisement. In this paper, we propose an emotion-enabled algorithm that can be used to personalize an advertising movie according to the user's current emotions to make the advertisement more efficient. Electroencephalogram (EEG) signals are used to recognize emotions of the user in real time. The proposed emotion-enabled algorithm can adjust the scene of the movie based on the real-time emotion feedback. An advertising movie that applies the emotion-enabled algorithm is designed and implemented.

Using Electroencephalogram (EEG) signals for affective interaction can make interfaces more intuitive. This project includes development of different affective applications based on an EEG-based real-time emotion recognition algorithm. The algorithm is subject-dependent one and consists from two parts: feature extraction and classification. The algorithm can recognize up to eight emotions with good accuracy. The demo includes affective games and emotional avatar applications. After a short session to train the classifier, an application is able to monitor the user's emotions, and the recognized emotions are used as the input to the applications.

Smooth haptic force feedback is an important task for multirate 6-DOF haptic rendering for both rigid and deformable objects. The update rate of the haptic force may be too low and changed during the simulation as the high computation time is required for complex physical simulation. Therefore, to implement a stable and smooth haptic rendering, we need a method to update the force in a higher regular rate during the physical simulation. We propose a prediction method using interpolation to calculate smooth haptic interaction force in a high update rate. The auto-regressive model is used to predict the force value based on the previous haptic force calculation. In addition, we introduce a spline function to interpolate force values for the haptic force output. We demonstrate that the proposed method can provide smooth and accurate haptic force feedback in a high update rate during a low frequency physical simulation of complex and deformable models. In the experiments, we show the feasibility of the proposed method and compare its accuracy and stability with the linear force prediction algorithm.

Emotion recognition from Electroencephalogram (EEG) rapidly gains interest from research community. Two affective EEG databases are presented in this paper. Two experiments are conducted to set up the databases. Audio and visual stimuli are used to evoke emotions during the experiments. The stimuli are selected from IADS and IAPS databases.14 subjects participated in each experiment. Emotiv EEG device is used for the data recording. The EEG data are rated by the participants with arousal, valence, and dominance levels. The correlation between powers of different EEG bands and the affective ratings is studied. The results agree with the literature findings and analyses of benchmark DEAP database that proves the reliability of the two databases. Similar brain patterns of emotions are obtained between the established databases and the benchmark database. A SVM-based emotion recognition algorithm is proposed and applied to both databases and the benchmark database. Use of a Fractal Dimension feature in combination with statistical and Higher Order Crossings (HOC) features gives us results with the best accuracy. Up to 8 emotions can be recognized. The accuracy is consistent between the established databases and the benchmark database.

In this paper, we propose and develop a novel adaptive haptic- based serious game for post stroke rehabilitation. Real-time patients emotions monitoring based on the Electroencephalogram (EEG) is used as an additional game control. A subject-dependent algorithm recognizing negative and positive emotions from EEG is integrated. Force feedback is proposed and implemented in the game. The proposed EEG-enabled haptic-based serious game could help to promote rehabilitation of the patients with motor deficits after stroke. Such games could be used by the patients for post stroke rehabilitation even at home convenience without a nurse presence.

In this paper, a new stable dynamic algorithm based on virtual coupling was proposed for 6-Degrees-of-Freedom (DOF) haptic rendering. It allows stable haptic manipulation of virtual objects when a virtual tool has physical property such as mass. In the haptic rendering process, we consider the dynamic property such as rotation inertia in each haptic frame. The main contribution of the stable dynamic algorithm is that it could overcome the "buzzing" problem appeared in the haptic rendering process. A nonlinear force/torque algorithm is proposed to calculate the haptic interaction when the collision happens between the virtual tool and virtual objects. The force/torque magnitude could saturate to the maximum force/torque value of the haptic device. The implemented algorithm was tested with peg-in-hole and Stanford bunny benchmarks. The experimental results showed that our algorithm was capable to provide stable 6-DOF haptic rendering for dynamic rigid virtual objects with physical property such as mass.

Emotions are important in human-computer interaction. Emotions could be classified based on 3-dimensional Valence-Arousal-Dominance model which allows defining any number of emotions even without discrete emotion labels. In this paper, we proposed a real-time EEG-based subject-dependent valence level recognition algorithm, where the thresholds were used to identify different levels of the valence dimension of the human emotion. The algorithm was tested by using the EEG data labeled with valence levels. The algorithm could identify valence levels continuously. The algorithm was tested with the experiment data and with the benchmark affective EEG database DEAP where up to 9 levels of valence dimension with high/low dominance were recognized. Then, the algorithm was applied to recognize 16 emotions defined by high/low arousal, high/low dominance and 4 levels of valence. At least 14 electrodes should be used to get the better accuracy. The proposed algorithm could be implemented in different real-time applications such as emotional avatar and E-learning systems.

Emotions recognized from Electroencephalogram (EEG) could reflect the real "inner" feelings of the human. Recently, research on real-time emotion recognition received more attention since it could be applied in games, e-learning systems or even in marketing. EEG signal can be divided into the delta, theta, alpha, beta, and gamma waves based on their frequency bands. Based on the Valence-Arousal-Dominance emotion model, we proposed a subject-dependent algorithm using the beta/alpha ratio to recognize high and low dominance levels of emotions from EEG. Three experiments were designed and carried out to collect the EEG data labeled with emotions. Sound clips from International Affective Digitized Sounds (IADS) database and music pieces were used to evoke emotions in the experiments. Our approach would allow real-time recognition of the emotions defined with different dominance levels in Valence-Arousal-Dominance model.

Haptic devices enable the user to manipulate the molecules and feel interactions during the docking process in virtual environment on the computer. Implementation of torque feedback allows the user to have more realistic experience during force simulation and find the optimum docking positions faster. In this paper, we propose a haptic rendering algorithm for biomolecular docking with torque force. It enables the user to experience six degree-of-freedom (DOF) haptic manipulation in docking process. The linear smoothing method was proposed to improve stability of the haptic rendering during molecular docking.

Emotions accompany everyone in the daily life, playing a key role in non-verbal communication, and they are essential to the understanding of human behavior. Emotion recognition could be done from the text, speech, facial expression or gesture. In this paper, we concentrate on recognition of “inner” emotions from electroencephalogram (EEG) signals as humans could control their facial expressions or vocal intonation. The need and importance of the automatic emotion recognition from EEG signals has grown with increasing role of brain computer interface applications and development of new forms of human-centric and human-driven interaction with digital media. We propose fractal dimension based algorithm of quantification of basic emotions and describe its implementation as a feedback in 3D virtual environments. The user emotions are recognized and visualized in real time on his/her avatar adding one more so-called “emotion dimension” to human computer interfaces.

Recently, EEG-based technology has become more popular in “serious” games designs and developments since new wireless headsets that meet consumer demand for wear ability, price, portability and ease-of-use are coming to the market. Originally, EEG-based technologies were used in neurofeedback games and brain-computer interfaces. Now, such technologies could be used in entertainment, e-learning and new medical applications. In this paper, we review on neurofeedback game designs and algorithms, and propose design, algorithm, and implementation of new EEG-based 2D and 3D concentration games. Possible future medical applications of the games are discussed.

Cyberworlds could be a platform for both research and e-learning particularly in research intensive disciplines such as biology, physical chemistry, molecular medicine, biophysics, structural biology, bioinformatics, etc. The computer simulation of assembling molecules has been studied intensively in the field of computer-aided rational drug design. The assembly of molecules in a three-dimensional space or molecular docking is used for rational drug design where a ligand docks onto a receptor. The computer-aided design systems allow real-time interactive visualization and manipulation of molecules in virtual environment. These techniques help the user to understand molecular interactions, and to evaluate the design of pharmaceutical drugs. Besides the visualization techniques, there has been increasing interest in using haptic interfaces to facilitate the exploration and analysis of molecular docking. Haptic device could enable the users to manipulate the molecules and feel its interaction during the docking process in virtual experiment on computer. In this paper, we propose visual haptic-based biomolecular docking system that could be used for biomolecular docking to study helix-helix interactions and in e-learning.

We seek to further expand the shared collaborative potential of cyberworlds by using haptic forcefeedback in shared virtual scenes. We propose how to define density of the objects, together with their geometry and appearance, by using mathematical functions. We illustrate this concept by developing software which allows us to touch and feel surfaces of VRML and X3D objects, convert them to solid objects as well as create any other solid objects using the function-based extension of VRML and X3D. We define geometry, appearance and density of the solid objects by implicit, explicit and parametric functions straight in the VRML/X3D code or in dynamic-link libraries. Since the function-based models are small in size, it is possible to perform their collaborative interactive modifications with concurrent synchronous visualization at each client computer with any required level of detail. We illustrate the proposed with several application examples.

Visual clustering should be one of the basic tools for time-dependent data analysis in cyberworlds. This paper describes a novel approach to spatial clustering and boundary detection based on geometric modeling and visualization. Datasets and boundaries of clusters are visualized as 3D points and surfaces of reconstructed solids changing over time. Our approach applies the concepts of geometric solid modeling and uses density as clustering criteria that comes from traditional density-based clustering techniques. Visual clustering allows the user to analyze results of clustering the data changing over time and to interactively choose appropriate parameters.

Visual interfaces are very important for human interactions in cyberworlds. Visual querying should be one of the basic tools for data mining and retrieval in cyberworlds. In this paper, function-based modeling of spatio-temporal range queries is described. It allows for creating an intuitive visual interface using 2D projection of 3D query shapes. The proposed approach combines visualization of time-dependent data with visualization of the range query formulation employing very compact function-based query model. It is also shown that the proposed geometric model makes constructing the queries on spatio-temporal attributes more intuitive, and allows for posing temporal range queries of any shape

Virtual enterprise enables companies to collaborate through sharing of resources, and therefore allows them to enjoy the benefits of virtual integration. The essence of virtual enterprise implementation on the Web lies in data integration. This paper describes key research issues in implementation of data integration in virtual enterprise using semantic Web service. The architectural framework of the proposed data integration infrastructure adopts a mediated ontology approach to data integration in which each data source is described by its own ontology and translations between different ontologies are by means of mediation. We also propose to use the concept of an active data warehousing system for virtual enterprise data aggregation. The paper briefly describes the preliminary study for the project proposal

This paper explores the use of ARIS for semantic Web service development. It proposes an ARIS-based transformation methodology for the automatic specification and development of semantic Web services. It also identifies research issues which have yet to be resolved for the development of a supporting software tool for the methodology.

In the recent years, the usage of 3D cyberworlds for educational purposes has increased. The metaphors behind the design of virtual places are quite diverse, from replication of real universities to art museums and scientific labs. Based on the results of a case study we have performed, this paper provides an initial set of requirements for a cyberworld representing an existing university. In this connection, we analyze place metaphors and associated design features of the virtual campus of Nanyang Technological University in Singapore in the context of related work. Finally, we discuss the correspondence between the identified metaphors and associated educational goals, providing directions for further development of the virtual campus

Temporal databases and data warehouses are essential components of intelligent information systems in cyberworlds. We describe a geometric model for querying time-dependent data in databases and warehouses. An implementation and application of the model for querying of results of numerical simulation in molecular dynamics is discussed. Data are interpreted geometrically as multidimensional points with time dimension. A geometric query is a query solid of any shape specified by its parameters, location and time. These queries are formulated with geometric objects and operations over them to form the query solid. The geometric objects and operations are described with implicit functions. With the uniform geometric model for querying time-dependent data, 3D visualization tools can be naturally incorporated into the molecular dynamics visualization system to pose the queries.

Databases and warehouses play significant role in real world providing the human with the necessary information for solving problems in all areas of life from marketing to bioinformatics. Therefore, databases and warehouses are essential components of all intelligent information systems in cyberworlds. This paper describes a geometric approach to clustering and querying data in databases and warehouses. Data are interpreted geometrically as multidimensional points. A geometric definition of the cluster as a multidimensional solid defined with implicit functions is introduced. A query window is a query solid of any shape specified by its location. The queries are formulated with geometric objects and operations over them. The geometric objects and operations are described with implicit functions. With the uniform geometric model of the clustering and querying, 3D visualization tools can be naturally incorporated in one system that allows us to visualize and query clusters in 3D space. The user clusters the data and poses the queries through a graphics interface accessing dynamically multidimensional points and solids.