Manned-Unmanned Teaming (MUM-T) represents the future of aerial operations by enabling an operator aboard a manned aircraft to control one or more Unmanned Aerial Vehicles (UAVs). As MUM-T interfaces are still developing, understanding how dynamic environments affect operators’ perception and control performance is essential. This study investigates the effect of load factor and visual perspective on teleoperation performance, workload, and motion sickness. Fourteen participants completed a simulated teleoperation task onboard a light Robin DR400 aircraft, subjected to turn angles of 30°, 45°, and 60° (load factors of 1.2, 1.4, and 2g respectively). Preliminary results reveal a significant increase in mental and physical demands under higher load factors without affecting teleoperation performance. No difference was observed between the two visual perspectives tested (first person and third person). This study shows that teleoperators can maintain their performance in real flight conditions at the cost of a higher subjective workload.

The study investigates the impact of vestibular and somatosensory graviceptive signals on spatial perception and motor planning/control. Specifically, it explores the consequences of spatial compatibility between (1) the tilts of an observer’s body and those of an object perceived in virtual reality, and (2) the directions of body rotation and manual movements, on visual perception and object orientation control. We hypothesize that intersensory conflicts resulting from performing a task with a head-mounted display while tilted may increase susceptibility to spatial incompatibilities, further impairing teleoperation performance. Fifty-four participants controlled and estimated the orientation of an object in virtual reality while their body tilt and movement could be spatially compatible or incompatible with the object’s orientation and manual movement direction. The results indicate that spatial incompatibilities between the direction of participants’ whole-body rotation and manual movement negatively impacted the control of the object orientation, as indexed by accuracy, precision, and response times. Moreover, body roll tilt decreased operators’ confidence in their visual perception of the remotely controlled object, surprisingly only in spatially compatible conditions. These findings highlight teleoperation challenges in dynamic environments. Research on countermeasures for safer teleoperation is essential for field advancement.

This study investigates the mitigation of cognitive flexibility decrements during long-endurance Uncrewed Aerial Systems (UAS) missions through visual alerts. UAS mishaps are likely partially attributable to decreased cognitive flexibility caused by mental fatigue. Two groups of participants performed a 2-hour UAS simulation. One group was supported by visual alerts when switching between tasks, while the other was not. To further investigate the effect of visual alerts, the participant’s cerebral and cardiac activity during the task was recorded using electroencephalography (EEG) and electrocardiography (ECG). Results showed a weaker subjective increase of mental fatigue across time with visual alerts than the control group. Furthermore, behavioural results showed improved reaction time and accuracy in some performance metrics. Future work may improve this system’s efficacy by implementing it with an adaptive interface.

Mental fatigue from continuous operations without breaks represents a safety issue for military drone operations, as these systems are complex and operate during long shifts. Military operations are hard to study due to their sensitive nature. The open-access program UASOS serves as a testbed to examine the effects of mental fatigue in an ecologically valid environment. UASOS recreates fundamental aspects of military drone operations in a controllable environment that is easy enough for novices to understand but demanding enough to elicit mental fatigue. Participants alternate between navigating a drone-using either a trackball/mouse or a joystick-and searching for visual targets. The protocol is set up in a way that taxes the cognitive flexibility of participants by constantly requiring them to alternate between tasks. In addition, several parameters such as difficulty, duration, questionnaires, training phases, and more can be adapted. The task also allows for synchronization with physiological data using LabStreamingLayer. Implemented in python, the code is set up to be easily installed.

As teleoperated robotic units operate in remote and distant environments such as outer space, the communication latency impacts the operator’s performance. Yet, the delay impact on teleoperation performance and mental workload has scarcely been evaluated. Human temporal sensitivity appears to modulate the impact of latency on operators’ performance but no joint assessment of temporal sensitivity and subjective workload has been reported. In this study, we assess the relationship between the impact of communication delay on teleoperation performance, mental workload, and operators’ temporal sensitivity. Sixteen participants completed two online tasks: a duration reproduction task in which they were asked to reproduce the duration of previously presented visual stimuli, and an egocentric maze navigation task which required participants to escape a static maze, under an input latency of 0, 400, and 3000 ms. Completion time, move count, and error rate were recorded for each trial, along with perceived workload using the NASA-TLX questionnaire. The results showed that performance was significantly deteriorated by an increase in communication delay. Moreover, participants’ self-rated performance decreased with a larger communication delay, while their reported frustration, effort, and mental demands significantly increased. Interestingly, a possible effect of the temporal sensitivity profile on teleoperation performance - number of moves - was found, with a reduced number of moves for sensitive participants compared to insensitive ones, following a speed/accuracy trade-off (yet not significant). Hence, different operators’ strategies were uncovered, depending on their temporal sensitivity profile, to mitigate the impact of communication delay on the mission outcome.

This study aims at investigating the neural and physiological correlates of human-human and human-AI interactions under ecological settings. We designed a scenario in which a ground controller had to guide his/her pilot to reach a location. We also implemented a Controller-Bot and a Pilot-Bot using AI techniques to behave like real human operators. The cooperation between controllers and pilots were either genuine (‘Coop scenarios’ – four missions), explicitly notified as pilot-Bot and controller-Bot interactions (‘No coop scenarios’ – two missions), or with no notification that they were actually collaborating with their AI counterparts (‘fake coop scenarios’ – two missions). Sixteen participants (8 dyads) equipped with EEG and ECG took part in this experiment. Our findings disclosed that Human-Human dyads exhibited similar performance to Human-Bots dyads whether the human participants were aware that they were playing with a bot or not. Our participants declared that they did not realize they were playing with an AI in the fake cooperation condition. These findings indicate that 1) humans can be fooled by AI, and that 2) humans can behave in a natural way with AI. Interestingly enough, our analyses revealed that the cardiac activity of controllers and pilots was more synchronized when they were collaborating together than when they were playing with AI (being aware or not). Similarly, EEG analyses disclosed a higher cerebral efficiency and connectivity between the two brains when teammates were interacting together than when cooperating with AI.

Synchrony at the physiological level is an objective measure that can be used to investigate cooperation between human agents. This physiological synchrony has been experimentally observed in different dyadic contexts through measures of the autonomous system such as cardiac measures. Various metrics are used to characterize synchrony between participants such as crosscorrelation, weighted coherence, or cross recurrence quantification analysis and with a wide variety of paradigms. We propose the delayed coincidence count as a new method for assessing cardiac synchrony. Delayed coincidence count has already been used to characterize synchrony in firing neurons populations. While being straightforward and computationally light, this method has already been formally proven to be statistically robust. A complex dynamic microworld is designed with two difficulty levels and two cooperation conditions. A total of 40 participants, i.e., 20 teams, voluntarily has conducted the experiment. The delayed coincidence count method (with a coincidence threshold δ of 20 ms) reveals a significant synchrony (p <; .01) during the cooperative and high difficulty condition only, while the other methods did not. The results are interpreted in terms of interaction intensity in accordance with recent literature.

Accident analyses have revealed that pilots can fail to process auditory stimuli such as alarms, a phenomenon known as inattentional deafness. The motivation of this research is to develop a passive brain computer interface that can predict the occurence of this critical phenomenon during real flight conditions. Ten volunteers, equipped with a dry-EEG system, had to fly a challenging flight scenario while responding to auditory alarms by button press. The behavioral results disclosed that the pilots missed 36% of the auditory alarms. ERP analyses confirm that this phenomenon affects auditory processing at an early (N100) and late (P300) stages as the consequence of a potential attentional bottleneck mechanism. Intersubject classification was carried out over frequency features extracted three second epochs before the alarms’ onset using sparse representation for classification (SRC), sparse and dense representation (SDR) and more conventional approach such as linear discriminant analysis (LDA), shrinkage LDA and nearest neighbor (1NN). In the best case, SRC and SDR gave respectively a performance of 66.9% and 65.4% of correct mean classification rate to predict the occurrence of inattentional deafness, outperforming LDA (60.6%), sLDA (60%) and 1 NN (59.6%). These results open promising perspectives for the implementation of neuroadaptive automation with as ultimate goal to enhance alarm stimulation delivery so that it is perceived and acted upon.

There is a growing momentum to design online tools to measure mental workload for neuroergonomic purposes. Most of the research focuses on the monitoring of a single human operator. However, in real-life situations, human operators work in cooperation to optimize safety and performance. This is particularly the case in aviation whereby crews are composed of a pilot flying and a pilot monitoring. The motivation of this study is to evaluate the possibility to apply an hyperscanning approach to estimate the mental workload of crews composed of two operators. We designed an experimental protocol in which ten crews (i.e. 20 subjects) had to perform a modified version of the NASA MATBII during 8 five-minute blocks (i.e. 4 mental workload level configurations * 2 cooperation v. non cooperation conditions). Mental workload and cooperation level were classified using a traditional passive brain-computer interface pipeline that includes a spatial filtering step on frequency features. Our results disclosed that all mental states' estimations were significantly above chance level. Intra-subject classification accuracy for mental workload (2 classes) was 63% for the pilot flying and 58% for the pilot monitoring. As for cooperation level, the binary classification reached 57% for the pilot flying and 60% for the pilot monitoring. Regarding the team, intra-team classification accuracy of the workload configuration of the team (4-class) reached 35%. As for the team cooperation level, the binary classifier reached 60% of accuracy. The results are discussed in terms of hyperscanning applications.

There is growing interest for implementing tools to monitor cognitive performance in naturalistic environments. Recent technological progress has allowed the development of new generations of brain imaging systems such as dry electrode electroencephalography (EEG) and functional near infrared spectroscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. These highly portable brain imaging devices offer interesting prospects to implement passive brain computer interfaces (pBCI) and neuroadaptive technology. We developed a fNIRS-EEG based pBCI to monitor cognitive fatigue using engagement related features (EEG engagement ratio and wavelet coherence fNIRS based metrics). This mental state is known to impair cognitive performance and can jeopardize flight safety. In this preliminary study, four participants were asked to perform four traffic patterns along with a secondary auditory task in a flight simulator and in an actual light aircraft. The two first traffic patterns were considered as the low cognitive fatigue class, whereas the two last traffic patterns were considered as the high cognitive fatigue class. As expected, the pilots missed more auditory targets in the second part than in the first part of the experiment. Classification accuracy reached 87.2% in the flight simulator condition and 87.6% in the actual flight conditions when combining the two modalities. This study demonstrates that fNIRS and EEG-based pBCIs can monitor mental states in operational and noisy environments.

Long monitoring tasks without regular actions, are becoming increasingly common from aircraft pilots to train conductors as these systems grow more automated. These task contexts are challenging for the human operator because they require inputs at irregular and highly interspaced moments even though these actions are often critical. It has been shown that such conditions lead to divided and distracted attentional states which in turn reduce the processing of external stimuli (e.g. alarms) and may lead to miss critical events. In this study we explored to which extent it is possible to predict an operator's behavioural performance in a Unmanned Aerial Vehicle (UAV) monitoring task using electroencephalographic (EEG) activity. More specifically we investigated the relevance of large-scale EEG connectivity for performance prediction by correlating relative coherence with reaction times (RT). We show that long-range EEG relative coherence, i.e. between occipital and frontal electrodes, is significantly correlated with RT and that different frequency bands exhibit opposite effects. More specifically we observed that coherence between occipital and frontal electrodes was: negatively correlated with RT at 6Hz (θ band), more coherence leading to better performance, and positively correlated with RT at 8Hz (lower α band), more coherence leading to worse performance. Our results suggest that EEG connectivity measures could be useful in predicting an operator's attentional state and her/his performances in ecological settings. Hence these features could potentially be used in a neuro-adaptive interface to improve operator-system interaction and safety in critical systems.