: Reactive brain–computer interfaces (rBCIs) can deliver fast and reliable performance, yet the decision step — determining when to act based on neural evidence — remains an often overlooked component. In prior work, we proposed a decision-making framework based on partially observable Markov decision processes (POMDP). The present study moves this approach from offline validation to a real-time setting, integrating it into a code-modulated visual evoked potential (c-VEP) rBCI. Twelve healthy participants performed a five-class c-VEP control task across two sessions, each comprising a cued and a self-paced (Pinpad) task with a semi-dry electroencephalography (EEG) system. One session used a conventional accumulation-based decision strategy; the other employed the POMDP-based approach. In the POMDP condition, calibration data were collected during an engaging cued task, allowing the policy to be computed without interrupting user interaction. Across all tasks and conditions, participants achieved mean accuracies above 97%. In the self-paced task, the POMDP significantly reduced mean decoding time (1.55 s) compared to the accumulation baseline (1.97 s), while maintaining equivalent accuracy. This study provides the first online demonstration of a POMDP-based decision framework for rBCI, balancing speed and accuracy under the constrains of real-time operation . By removing the need for individual thresholds and integrating calibration into active use, the approach offers a flexible, user-centric pathway toward more user-centered and practical BCIs.

This paper proposes to investigate the design of user-friendly reactive Brain-Computer Interfaces (rBCIs) based on Code-Modulated Visual Evoked Potentials (c-VEP). The BCI was implemented using the StAR-Burst paradigm, which features small, randomly-oriented texture patches designed to optimize foveal neural responses while enhancing user visual comfort. We extended this paradigm to support an 11-class selection scenario using dry EEG electrodes. In addition, the study explored the impact of predictive visual feedback on user experience. Two feedback types—Halo and Depth—were compared against a control condition with no feedback, aiming to enhance users’ sense of control during interaction. Results demonstrated that the BCI achieved high classification accuracy with a dry EEG system across all three conditions (mean = 93,3%), using only 33 seconds of calibration data. Contrary to our expectations, predictive feedback did not lead to significant improvements in classification accuracy or decoding time. However, the Halo feedback significantly increased users’ anticipation of success, though it also caused greater peripheral distraction. Interestingly, decoding time improved significantly with practice, underscoring the role of user adaptation in enhancing performance. Overall, the findings highlight the need for explicit training to help users effectively interpret and utilize predictive feedback.

Code modulated Visually Evoked Potentials (cVEP) is an emerging paradigm for Brain-Computer Interfaces (BCIs) that offers reduced calibration times.However, cVEP-based BCIs still encounter challenges related to cross-session/subject variabilities.As Riemannian approaches have demonstrated good robustness to these variabilities, we propose the first study of deep Riemannian neural architectures, namely SPDNets, on cVEP-based BCIs.To evaluate their performance with respect to subject variabilities, we conduct classification tasks in a domain adaptation framework using a burst cVEP open dataset.This study demonstrates that SPDNet yields the best accuracy with single-subject calibration and promising results in domain adaptation.

Brain-Computer Interfaces (BCI) provide a unique communication channel between the brain and computer systems. After extensive research and implementation on ample fields of application, numerous challenges to assure reliable and quick data processing have resulted in the hybrid BCI (hBCI) paradigm, consisting on the combination of two BCI systems. However, not all challenges have been properly addressed (e.g. re-calibration, idle-state modelling, adaptive thresholds, etc) to allow hBCI implementation outside of the lab. In this paper, we review electroencephalography based hBCI studies and state potential limitations. We propose a sequential decision-making framework based on Partially Observable Markov Decision Process (POMDP) to design and to control hBCI systems. The POMDP framework is an excellent candidate to deal with the limitations raised above. To illustrate our opinion, an example of architecture using a POMDP-based hBCI control system is provided, and future directions are discussed. We believe this framework will encourage research efforts to provide relevant means to combine information from BCI systems and push BCI out of the laboratory.

This study aims at investigating the neural and physiological correlates of human-human and human-AI interactions under ecological settings. We designed a scenario in which a ground controller had to guide his/her pilot to reach a location. We also implemented a Controller-Bot and a Pilot-Bot using AI techniques to behave like real human operators. The cooperation between controllers and pilots were either genuine (‘Coop scenarios’ – four missions), explicitly notified as pilot-Bot and controller-Bot interactions (‘No coop scenarios’ – two missions), or with no notification that they were actually collaborating with their AI counterparts (‘fake coop scenarios’ – two missions). Sixteen participants (8 dyads) equipped with EEG and ECG took part in this experiment. Our findings disclosed that Human-Human dyads exhibited similar performance to Human-Bots dyads whether the human participants were aware that they were playing with a bot or not. Our participants declared that they did not realize they were playing with an AI in the fake cooperation condition. These findings indicate that 1) humans can be fooled by AI, and that 2) humans can behave in a natural way with AI. Interestingly enough, our analyses revealed that the cardiac activity of controllers and pilots was more synchronized when they were collaborating together than when they were playing with AI (being aware or not). Similarly, EEG analyses disclosed a higher cerebral efficiency and connectivity between the two brains when teammates were interacting together than when cooperating with AI.

Steady-States Visually Evoked Potentials (SSVEP) is currently one of the most widely used paradigms in Brain-Computer Interfaces (BCI). Although the SSVEP-BCI are characterized by their high and robust classification performance, the repetitive presentation of flickering stimuli is uncomfortable from a user experience perspective point of view. Indeed, the low-level visual features of SSVEP stimuli make them straining to the eyes over time and could be disruptive to the execution of tasks requiring sustained attention. They could even induce epileptic seizures. This study explores the reduction of the stimulation amplitude depth (a magnitude diminution of 90%) for the design of SSVEP stimuli as a solution to improve user comfort. The classification accuracy obtained by different pipelines was systematically compared between low amplitude and standard full amplitude SSVEP stimuli. The results reveal high classification accuracy for both high (99.8%) and low magnitude (80.2%) stimuli using the Task-Related Component Analysis (TRCA) classification method. The present findings demonstrate the validity of reducing SSVEP stimuli amplitude to increase users’ comfort paving the way for transparent BCI operation.

Attentional tunneling, that is the inability to detect unexpected changes in the environment, has been shown to have critical consequences in air traffic control. The motivation of this study was to assess the design of a cognitive countermeasure dedicated to mitigate such failure of attention. The Red Alert cognitive countermeasure relies on a brief orange-red flash (300 ms) that masks the entire screen with a 15% opacity. Twenty-two air traffic controllers faced two demanding scenarios, with or without the cognitive countermeasure. The volunteers were not told about the Red Alert so as to assess the intuitiveness of the design without prior knowledge. Behavioral results indicated that the cognitive countermeasure reduced reaction time and improved the detection of the notification when compared to the classical operational design. Further analyses showed this effect was even stronger for half of our participants (91.7% detection rate) who intuitively understood the purpose of this design.

Abstract The aim of this work is to predict human operator’s (HO) decisions in a specific operational context, such as a cooperative human-robot mission, by approximating his/her utility function based on prospect theory (PT). To this aim, a within-subject experiment was designed in which the HO has to decide with limited time and incomplete information. This experiment also involved a framing effect paradigm, a typical cognitive bias causing people to react differently depending on the context. Such an experiment allowed to acquire data concerning the HO’s decisions in two different mission scenarios: search and rescue and Mars rock sampling. The framing was manipulated (e.g. positive vs. negative) and the probability of the outcomes causing people to react differently depending on the context. Statistical results observed for this experiment supported the hypothesis that the way the problem was presented (positively or negatively framed) and the emotional commitment affected the HO’s decisions. Thus, based on the collected data, the present work is willed to propose: (i) a formal approximation of the HO’s utility function founded on the prospect theory and (ii) a model used to predict the HO’s decisions based on the economics approach of multi-dimensional consumption bundle and PT. The obtained results, in terms of utility function fit and prediction accuracy, are promising and show that similar modeling and prediction method should be taken into account when an intelligent cybernetic system drives human–robot interaction. The advantage of predicting the HO’s decision, in this operational context, is to anticipate his/her decision, given the way a question is framed to the HO. Such a predictor lays the foundation for the development of a decision-making system capable of choosing how to present the information to the operator while expecting to align his/her decision with the given operational guideline.

Synchrony at the physiological level is an objective measure that can be used to investigate cooperation between human agents. This physiological synchrony has been experimentally observed in different dyadic contexts through measures of the autonomous system such as cardiac measures. Various metrics are used to characterize synchrony between participants such as crosscorrelation, weighted coherence, or cross recurrence quantification analysis and with a wide variety of paradigms. We propose the delayed coincidence count as a new method for assessing cardiac synchrony. Delayed coincidence count has already been used to characterize synchrony in firing neurons populations. While being straightforward and computationally light, this method has already been formally proven to be statistically robust. A complex dynamic microworld is designed with two difficulty levels and two cooperation conditions. A total of 40 participants, i.e., 20 teams, voluntarily has conducted the experiment. The delayed coincidence count method (with a coincidence threshold δ of 20 ms) reveals a significant synchrony (p <; .01) during the cooperative and high difficulty condition only, while the other methods did not. The results are interpreted in terms of interaction intensity in accordance with recent literature.

Accident analyses have revealed that pilots can fail to process auditory stimuli such as alarms, a phenomenon known as inattentional deafness. The motivation of this research is to develop a passive brain computer interface that can predict the occurence of this critical phenomenon during real flight conditions. Ten volunteers, equipped with a dry-EEG system, had to fly a challenging flight scenario while responding to auditory alarms by button press. The behavioral results disclosed that the pilots missed 36% of the auditory alarms. ERP analyses confirm that this phenomenon affects auditory processing at an early (N100) and late (P300) stages as the consequence of a potential attentional bottleneck mechanism. Intersubject classification was carried out over frequency features extracted three second epochs before the alarms’ onset using sparse representation for classification (SRC), sparse and dense representation (SDR) and more conventional approach such as linear discriminant analysis (LDA), shrinkage LDA and nearest neighbor (1NN). In the best case, SRC and SDR gave respectively a performance of 66.9% and 65.4% of correct mean classification rate to predict the occurrence of inattentional deafness, outperforming LDA (60.6%), sLDA (60%) and 1 NN (59.6%). These results open promising perspectives for the implementation of neuroadaptive automation with as ultimate goal to enhance alarm stimulation delivery so that it is perceived and acted upon.

There is a growing momentum to design online tools to measure mental workload for neuroergonomic purposes. Most of the research focuses on the monitoring of a single human operator. However, in real-life situations, human operators work in cooperation to optimize safety and performance. This is particularly the case in aviation whereby crews are composed of a pilot flying and a pilot monitoring. The motivation of this study is to evaluate the possibility to apply an hyperscanning approach to estimate the mental workload of crews composed of two operators. We designed an experimental protocol in which ten crews (i.e. 20 subjects) had to perform a modified version of the NASA MATBII during 8 five-minute blocks (i.e. 4 mental workload level configurations * 2 cooperation v. non cooperation conditions). Mental workload and cooperation level were classified using a traditional passive brain-computer interface pipeline that includes a spatial filtering step on frequency features. Our results disclosed that all mental states' estimations were significantly above chance level. Intra-subject classification accuracy for mental workload (2 classes) was 63% for the pilot flying and 58% for the pilot monitoring. As for cooperation level, the binary classification reached 57% for the pilot flying and 60% for the pilot monitoring. Regarding the team, intra-team classification accuracy of the workload configuration of the team (4-class) reached 35%. As for the team cooperation level, the binary classifier reached 60% of accuracy. The results are discussed in terms of hyperscanning applications.

There is growing interest for implementing tools to monitor cognitive performance in naturalistic environments. Recent technological progress has allowed the development of new generations of brain imaging systems such as dry electrode electroencephalography (EEG) and functional near infrared spectroscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. These highly portable brain imaging devices offer interesting prospects to implement passive brain computer interfaces (pBCI) and neuroadaptive technology. We developed a fNIRS-EEG based pBCI to monitor cognitive fatigue using engagement related features (EEG engagement ratio and wavelet coherence fNIRS based metrics). This mental state is known to impair cognitive performance and can jeopardize flight safety. In this preliminary study, four participants were asked to perform four traffic patterns along with a secondary auditory task in a flight simulator and in an actual light aircraft. The two first traffic patterns were considered as the low cognitive fatigue class, whereas the two last traffic patterns were considered as the high cognitive fatigue class. As expected, the pilots missed more auditory targets in the second part than in the first part of the experiment. Classification accuracy reached 87.2% in the flight simulator condition and 87.6% in the actual flight conditions when combining the two modalities. This study demonstrates that fNIRS and EEG-based pBCIs can monitor mental states in operational and noisy environments.

Long monitoring tasks without regular actions, are becoming increasingly common from aircraft pilots to train conductors as these systems grow more automated. These task contexts are challenging for the human operator because they require inputs at irregular and highly interspaced moments even though these actions are often critical. It has been shown that such conditions lead to divided and distracted attentional states which in turn reduce the processing of external stimuli (e.g. alarms) and may lead to miss critical events. In this study we explored to which extent it is possible to predict an operator's behavioural performance in a Unmanned Aerial Vehicle (UAV) monitoring task using electroencephalographic (EEG) activity. More specifically we investigated the relevance of large-scale EEG connectivity for performance prediction by correlating relative coherence with reaction times (RT). We show that long-range EEG relative coherence, i.e. between occipital and frontal electrodes, is significantly correlated with RT and that different frequency bands exhibit opposite effects. More specifically we observed that coherence between occipital and frontal electrodes was: negatively correlated with RT at 6Hz (θ band), more coherence leading to better performance, and positively correlated with RT at 8Hz (lower α band), more coherence leading to worse performance. Our results suggest that EEG connectivity measures could be useful in predicting an operator's attentional state and her/his performances in ecological settings. Hence these features could potentially be used in a neuro-adaptive interface to improve operator-system interaction and safety in critical systems.

The objective of this study is to design a human-robots system that takes into account the non-deterministic nature of the human operator's behavior. Such a system is implemented in a proof of concept scenario relying on a (MO)MDP decision framework that takes advantage of an eye-tracker device to estimate the cognitive availability of the human operator, and, some human operator's inputs to deduce where he is focusing his attention. An experiment was conducted with ten participants interacting with a team of autonomous vehicles in a Search & Rescue scenario. Our results demonstrate the advantages of considering the cognitive availability of a human operator in such a complex context and also the interest of using such a decisional framework that can formally integrate the non-deterministic outcomes which model the human behavior.

Decision making is a critical issue for humans operating unmanned vehicles. However, it is well admitted that many cognitive biases affect human judgments, leading to suboptimal or irrational decisions. The framing effect is a typical cognitive bias causing people to react differently depending on the context, the probability of the outcomes and how the problem is presented (loss vs. gain). There is a need to better understand the effects of these biases in operational contexts to optimize human-robot interactions. We therefore conducted an experiment involving a framing paradigm in a search and rescue mission (earthquake) and in a Mars rock sampling mission. We manipulated the framing (positive vs. negative) and the probability of the outcomes. Our findings revealed that the way the problem was presented (positively or negatively framed) and the emotional commitment (saving lives vs. collecting the good rock) statistically affected the choices made by the human operators.

Conflicts between the pilot and the automation, when pilots detect but do not understand them, cause "automation surprise" situations and jeopardize flight safety. We conducted an experiment in a 3-axis motion flight simulator with 16 pilots equipped with an eye-tracker to analyze their behavior and eye movements during the occurrence of such a situation. The results revealed that this conflict engages participant's attentional abilities resulting in excessive and inefficient visual search patterns. This experiment confirmed the crucial need to design solutions for detecting the occurrence of conflictual situations and to assist the pilots. We therefore proposed an approach to formally identify the occurrence of "automation surprise" conflicts based on the analysis of "silent mode changes" of the autopilot. A demonstrator was implemented and allowed for the automatic trigger of messages in the cockpit that explains the autopilot behavior. We implemented a real-time demonstrator that was tested as a proof-of-concept with 7 subjects facing 3 different conflicts with automation. The results shown the efficacy of this approach which could be implemented in existing cockpits.

This study discusses the application of sequential decision making under uncertainty and mixed observability in a mixed-initiative robotic target search application. In such a robotic mission, two agents, a ground robot and a human operator, must collaborate to reach a common goal using, each in turn, their recognized skills. The originality of the work relies in considering that the human operator is not a providential agent when the robot fails. Using the data from previous experiments, a Mixed Observability Markov Decision Process (MOMDP) model was designed, which allows to consider aleatory failure events and the partial observable human operator's state while planning for a long-term horizon. Results show that the collaborative system was in general able to successfully complete or terminate the mission, even when many simultaneous sensors, devices and operator failures happened. So, the mixed-initiative framework highlighted in this study shows the relevancy of taking into account the cognitive state of the operator, which permits to compute a policy for the sequential decision problem which prevents to re-planning when unexpected (but known) events occurs.

The allocation of visual attention is a key factor for the humans when operating complex systems under time pressure with multiple information sources. In some situations, attentional tunneling is likely to appear and leads to excessive focus and poor decision making. In this study, we propose a formal approach to detect the occurrence of such an attentional impairment that is based on machine learning techniques. An experiment was conducted to provoke attentional tunneling during which psycho-physiological and oculomotor data from 23 participants were collected. Data from 18 participants were used to train an adaptive neuro-fuzzy inference system (ANFIS). From a machine learning point of view, the classification performance of the trained ANFIS proved the validity of this approach. Furthermore, the resulting classification rules were consistent with the attentional tunneling literature. Finally, the classifier was robust to detect attentional tunneling when performing over test data from four participants.

Design of future systems for flight-deck automation will reflect a trend of changing the paradigm of human-computer interaction from the master (human)- slave (machine) mode to more equilibrated cooperation. In many cases such cooperation considers several humans and computer systems, for which multi-agent dynamic cooperative systems are appropriate models. Development of such systems requires very profound analysis of mutual interactions and conflicts that may arise in such systems. Additional testing is exhaustive and expensive for such systems. In the scope of the D3CoS project these problems are addressed from the modelling point of view with ambition to create tools that will simplify the development phase and replace parts of the testing phase. In this paper we investigate common flight procedures, for which computer assistance could be developed. We show how formal modelling of procedures allows us to inspect procedural inconsistencies and workload peaks before the development starts. We show how a computer cognitive architecture (a virtual pilot) can simulate human pilot behaviour in the cockpit to address questions typical for the early phase of the development. Analysis of these questions allows us to reduce the number of candidates for the final implementation without the need of expensive experiments with human pilots. This modelling approach is demonstrated on experiments undertaken both with human pilots and a virtual pilot. The quality of the outcome from both experimental settings remains conserved as shown by physiological assessment of pilot workload, which in turn justifies the use of the modelling approach for this type of problems.