Manned-Unmanned Teaming (MUM-T) represents the future of aerial operations by enabling an operator aboard a manned aircraft to control one or more Unmanned Aerial Vehicles (UAVs). As MUM-T interfaces are still developing, understanding how dynamic environments affect operators’ perception and control performance is essential. This study investigates the effect of load factor and visual perspective on teleoperation performance, workload, and motion sickness. Fourteen participants completed a simulated teleoperation task onboard a light Robin DR400 aircraft, subjected to turn angles of 30°, 45°, and 60° (load factors of 1.2, 1.4, and 2g respectively). Preliminary results reveal a significant increase in mental and physical demands under higher load factors without affecting teleoperation performance. No difference was observed between the two visual perspectives tested (first person and third person). This study shows that teleoperators can maintain their performance in real flight conditions at the cost of a higher subjective workload.

The study investigates the impact of vestibular and somatosensory graviceptive signals on spatial perception and motor planning/control. Specifically, it explores the consequences of spatial compatibility between (1) the tilts of an observer’s body and those of an object perceived in virtual reality, and (2) the directions of body rotation and manual movements, on visual perception and object orientation control. We hypothesize that intersensory conflicts resulting from performing a task with a head-mounted display while tilted may increase susceptibility to spatial incompatibilities, further impairing teleoperation performance. Fifty-four participants controlled and estimated the orientation of an object in virtual reality while their body tilt and movement could be spatially compatible or incompatible with the object’s orientation and manual movement direction. The results indicate that spatial incompatibilities between the direction of participants’ whole-body rotation and manual movement negatively impacted the control of the object orientation, as indexed by accuracy, precision, and response times. Moreover, body roll tilt decreased operators’ confidence in their visual perception of the remotely controlled object, surprisingly only in spatially compatible conditions. These findings highlight teleoperation challenges in dynamic environments. Research on countermeasures for safer teleoperation is essential for field advancement.

Eye movements analysis has great potential for understanding operator behaviour in many safety-critical domains, including aviation. In addition to traditional eye-tracking measures on pilots’ visual behavior, it seems promising to incorporate machine learning approaches to classify pilots’ visual scanning patterns. However, given the multitude of pattern measures, it is unclear which are better suited as predictors. In this study we analyzed the visual behaviour of eight pilots, flying different flight phases in a moving-base flight simulator. With this limited dataset we present a methodological approach to train linear Support Vector Machine models, using different combinations of the attention ratio and scanning pattern features. The results show that the overall accuracy to classify the pilots’ visual behaviour in different flight phases, improves from 51.6% up to 64.1% when combining the attention ratio and instrument scanning sequence in the classification model.

The Low/High Index of Pupillary Activity (LHIPA), an eye-tracked measure of pupil diameter oscillation, is redesigned and implemented to function in real-time. The novel Real-time IPA (RIPA) is shown to discriminate cognitive load in re-streamed data from earlier experiments. Rationale for the RIPA is tied to the functioning of the human autonomic nervous system yielding a hybrid measure based on the ratio of Low/High frequencies of pupil oscillation. The paper's contribution is drawn from provision of documentation of the calculation of the RIPA. As with the LHIPA, it is possible for researchers to apply this metric to their own experiments where a measure of cognitive load is of interest.

As teleoperated robotic units operate in remote and distant environments such as outer space, the communication latency impacts the operator’s performance. Yet, the delay impact on teleoperation performance and mental workload has scarcely been evaluated. Human temporal sensitivity appears to modulate the impact of latency on operators’ performance but no joint assessment of temporal sensitivity and subjective workload has been reported. In this study, we assess the relationship between the impact of communication delay on teleoperation performance, mental workload, and operators’ temporal sensitivity. Sixteen participants completed two online tasks: a duration reproduction task in which they were asked to reproduce the duration of previously presented visual stimuli, and an egocentric maze navigation task which required participants to escape a static maze, under an input latency of 0, 400, and 3000 ms. Completion time, move count, and error rate were recorded for each trial, along with perceived workload using the NASA-TLX questionnaire. The results showed that performance was significantly deteriorated by an increase in communication delay. Moreover, participants’ self-rated performance decreased with a larger communication delay, while their reported frustration, effort, and mental demands significantly increased. Interestingly, a possible effect of the temporal sensitivity profile on teleoperation performance - number of moves - was found, with a reduced number of moves for sensitive participants compared to insensitive ones, following a speed/accuracy trade-off (yet not significant). Hence, different operators’ strategies were uncovered, depending on their temporal sensitivity profile, to mitigate the impact of communication delay on the mission outcome.

Increasing workload is a central notion in human factors research that can decrease the performance and yield accidents. Thus, it is crucial to understand the impact of different internal operator’s factors including eye movements, memory and audio-visual integration. Here, we explored the relationship between cognitive workload (low vs. high) and eye movements (saccades, fixations and smooth pursuit). The task difficulty was induced by auditory noise, arithmetical count and working memory load. We estimated cognitive workload using EOG and EEG-based mental state monitoring. One novelty consists in recording the EOG around the ears (alternative EOG) and around the eyes (conventional EOG). The number of blinks and saccades amplitude increased along with the difficulty increase (p ≤ 0.05). We found significant correlations between EOG and EEG (theta/alpha ratio) and between conventional and alternative EOG signal. The increase in cognitive load may disturb the coding and maintenance of related visual information. Alternative EOG metrics could be a valuable tool for detecting workload.

Brushing techniques have a long history with the first interactive selection tools appearing in the 1990s. Since then, many additional techniques have been developed to address selection accuracy, scalability and flexibility issues. Selection is especially difficult in large datasets where many visual items tangle and create overlapping. Existing techniques rely on trial and error combined with many view modifications such as panning, zooming, and selection refinements. For moving object analysis, recorded positions are connected into line segments forming trajectories and thus creating more occlusions and overplotting. As a solution for selection in cluttered views, this paper investigates a novel brushing technique which not only relies on the actual brushing location but also on the shape of the brushed area. The process can be described as follows. Firstly, the user brushes the region where trajectories of interest are visible (standard brushing technique). Secondly, the shape of the brushed area is used to select similar items. Thirdly, the user can adjust the degree of similarity to filter out the requested trajectories. This brushing technique encompasses two types of comparison metrics, the piecewise Pearson correlation and the similarity measurement based on information geometry. To show the efficiency of this novel brushing method, we apply it to concrete scenarios with datasets from air traffic control, eye tracking, and GPS trajectories.

We discuss several techniques for mapping the gaze point to fiducial markers detected in a video stream, as commonly used in Augmented Reality eye-tracking applications. Specifically, we focus on using the recovered camera rotation and translation determined by the apparent distortion of the marker. We exploit this to map Areas Of Interest (AOIs) in the plane relative to the given marker. This gives the advantage of not needing any more than a single marker. We also review how the recovered homography is transformed to the graphics world coordinates so that AOI visualization can be performed in world space instead of camera space.

We investigate the smooth pursuit eye movement based interaction using an unmodified off-the-shelf RGB camera. In each pair of sequential video frames, we compute the indicative direction of the eye movement by analyzing flow vectors obtained using the Lucas-Kanade optical flow algorithm. We discuss how carefully selected low vectors could replace the traditional pupil centers detection in smooth pursuit interaction. We examine implications of unused features in the eye camera imaging frame as potential elements for detecting gaze gestures. This simple approach is easy to implement and abstains from many of the complexities of pupil based approaches. In particular, EyeFlow does not call for either a 3D pupil model or 2D pupil detection to track the pupil center location. We compare this method to state-of-the-art approaches and ind that this can enable pursuit interactions with standard cameras. Results from the evaluation with 12 users data yield an accuracy that compares to previous studies. In addition, the benefit of this work is that the approach does not necessitate highly matured computer vision algorithms and expensive IR-pass cameras.

The virtual reality (VR) has nowadays numerous applications in training, education, and rehabilitation. To efficiently present the immersive 3D stimuli, we need to understand how spatial attention is oriented in VR. The efficiency of different cues can be compared using the Posner paradigm. In this study, we designed an ecological environment where participants were presented with a modified version of the Posner cueing paradigm. Twenty subjects equipped with an eye-tracking system and VR HMD performed a sandwich preparation task. They were asked to assemble the ingredients which could be either endogenously and exogenously cued in both auditory and visual modalities. The results showed that all valid cues made participants react faster. While directional arrow (visual endogenous) and 3D sound (auditory exogenous) oriented attention globally to the entire cued hemifield, the vocal instruction (auditory endogenous) and object highlighting (visual exogenous) allowed more local orientation, in a specific region of space. No differences in gaze shift initiation nor time to fixate the target were found suggesting the covert orienting.

We investigate the mechanisms of attentional orienting in a 360-degree virtual environments. Through the use of Posner's paradigm, we study the effects of different attentional guidance techniques designed to improve information processing. The most efficient technique will be applied to a procedure learning tool in virtual reality and a remote air traffic control tower. The eye-tracker allows us to explore the differential effects of overt and covert orienting, to estimate the effectiveness of visual research and to use it as a technique for interaction in virtual reality.

In this research, we use eye tracking to monitor the attentional behavior of pilots in the cockpit. We built a cockpit monitoring database that serves as a reference for real-time assessment of the pilot's monitoring strategies, based on numerous flight simulator sessions with eye-tracking recordings. Eye tracking may also be employed as a passive input for assistive system, future studies will also explore the possibility to adapt the notifications' modality using gaze.

We present PathWord (PATH passWORD), a multimodal digit entry method for ad-hoc authentication based on known digits shape and user relative eye movements. PathWord is a touch-free, gaze-based input modality, which attempts to decrease shoulder surfing attacks when unlocking a system using PINs. The system uses a modified web camera to detect the user's eye. This enables suppressing direct touch, making it difficult for passer-bys to be aware of the input digits, thus reducing shoulder surfing and smudge attacks. In addition to showing high accuracy rates (Study 1: 87.1% successful entries) and strong confidentiality through detailed evaluations with 42 participants (Study 2), we demonstrate how PathWord considerably diminishes the potential of stolen passwords (on average 2.38% stolen passwords with PathWord vs. over 90% with traditional PIN screen). We show use-cases of PathWord and discuss its advantages over traditional input modalities. We envision PathWord as a method to foster confidence while unlocking a system through gaze gestures.

Edge bundling methods reduce visual clutter of dense and occluded graphs. However, existing bundling techniques either ignore edge properties such as direction and data attributes, or are otherwise computationally not scalable, which makes them unsuitable for tasks such as exploration of large trajectory datasets. We present a new framework to generate bundled graph layouts according to any numerical edge attributes such as directions, timestamps or weights. We propose a GPU-based implementation linear in number of edges, which makes our algorithm applicable to large datasets. We demonstrate our method with applications in the analysis of aircraft trajectory datasets and eye-movement traces.

Conflicts between the pilot and the automation, when pilots detect but do not understand them, cause "automation surprise" situations and jeopardize flight safety. We conducted an experiment in a 3-axis motion flight simulator with 16 pilots equipped with an eye-tracker to analyze their behavior and eye movements during the occurrence of such a situation. The results revealed that this conflict engages participant's attentional abilities resulting in excessive and inefficient visual search patterns. This experiment confirmed the crucial need to design solutions for detecting the occurrence of conflictual situations and to assist the pilots. We therefore proposed an approach to formally identify the occurrence of "automation surprise" conflicts based on the analysis of "silent mode changes" of the autopilot. A demonstrator was implemented and allowed for the automatic trigger of messages in the cockpit that explains the autopilot behavior. We implemented a real-time demonstrator that was tested as a proof-of-concept with 7 subjects facing 3 different conflicts with automation. The results shown the efficacy of this approach which could be implemented in existing cockpits.