Eye-hand coordination under the microscope is a fundamental skill to many microsurgical specializations. In this work, we investigate the oculomotor and motor differences between novice and expert surgeons in the classical eye-hand coordination Fitts’ law task. However, the field of view under the microscope is never static or stable due to inherent internal and external artifacts. In this work, we present methodological steps necessary for stabilizing the operation field of view using RANSAC and for defining static areas of interest (i.e., task targets) and dynamic areas of interest using YOLOv8 (i.e., instrument movement). When combined, we observed that under the microscope, eye movements were spatially more focused than hand movements and that experts relied on parafoveal vision when targeting large targets. Our work opens the path to objective and autonomous skill-acquisition assessment in microsurgical expertise.

Microsurgery requires visuomotor coordination under high magnification, where depth and scale are distorted. Understanding how surgeons adapt eye-hand coordination under the surgical microscope is key to advancing microsurgical education. We investigated gaze behavior and performance differences between expert plastic surgeons and novices during microsurgical suturing. An ocular-mounted eye-tracking system magnetically attached to the microscope preserved naturalistic gaze patterns while maintaining authentic spatial and visual conditions. Suturing was decomposed into phases: needle alignment and tissue piercing, pull-through, and knot tying, for phase-specific analysis. Performance was evaluated using the UWOMSA, and perceived workload was measured with the SURG-TLX. Experts completed the overall task faster than novices, exhibited shorter mean fixations overall, and showed the largest event-level advantage during knot tying. Findings indicate that visual strategies in microsurgery are phase-dependent and expertise-specific. In conclusion, ocular-mounted eye-tracking offers a valuable framework for studying microsurgical visuomotor control and developing objective, data-driven approach for microsurgical training.

Traditional baseline-subtraction methods in calculating pupillary response have short-comings in assessing workloads. The enhanced algorithms based on pupil oscillation, such as the Index of Cognitive Activity (ICA), Index of Pupillary Activity (IPA), and the enhanced version, Low/High Index of Pupillary Activity (LHIPA), utilizing wavelet analysis, have been developed to measure cognitive loads in scenarios where the gaze is relatively static. The applicability of these algorithms in motor tasks, where eye-hand coordination plays a significant role, has not been sufficiently studied. This research explores the effectiveness of LHIPA in measuring task workloads of users in a simulated surgical task. The task’s complexity was determined using Fitts’ index of difficulty, which varied by target size and distance. We found that LHIPA effectively differentiates task workloads across various levels of task difficulties in the surgical task. Results are crucial for creating objective methods to assess task workloads using pupil-based metrics during goal-oriented movements.

Research has found that changes in mental workload during both cognitive and motor tasks can be indicated by pupil dilation. To measure cognitive workload, researchers have developed tools such as the wavelet analysis-based index of cognitive activity (ICA) and index of pupil activity (IPA). However, it is still unclear whether these tools can accurately measure workload during motor tasks. This study aims to investigate whether the IPA can differentiate task workload during a motor task involving aiming in a tele-operation setting, where the task requirements were quantified using Fitts’ index of difficulty(ID) based on target size and distance. The study found that the IPA can differentiate between different levels of motor task workload, provided the proper window is used, namely the period before the tool touches the targets. This finding is significant as it can aid in developing objective methods to evaluate task workload using pupil parameters during goal-directed movements.

Blinks have been widely studied in various fields including medical and human computer interactions, and in driver fatigue. Automatic detection of blinks has valuable practical importance. While existing deep neural networks excel in extracting spatial features from images and demonstrate impressive performance in visual object recognition, their application for blink detection in videos on a frame-by-frame basis is suboptimal, as they only consider spatial features from single images. In this paper, we developed a spatial-temporal-aware deep learning framework that capitalizes on the rapid advancements of the existing state-of-the-art visual object recognition networks, aiming to enhance their performance specifically in blink detection. Our framework takes consecutive frames as input to extract spatial and temporal features simultaneously for better detection of eye movements. We also propose a sliding window re-sampling strategy to mitigate overfitting on training data. Extensive experimental evaluations and comparisons demonstrate the feasibility of the proposed algorithm, which delivers excellent performance for detecting blinks.

We examined whether human operators move their eyes earlier to a target before hands when the level of task difficulty increases. We hypothesized that participants would perform less proactive eye movements in the difficult task than in the easy one, as they would need to focus more on their current hand movements. Sixteen university students were recruited to perform the aiming and touching task reciprocally on three paired targets (circles) differing in sizes and distances, while had their eye movements tracked. The movement time, the early eye engagement time (EEET), and the number of eye adjustments were recorded. The EEET was defined as the time that a subject’s eyes fix on a target to the moment that the tool reaches out for it. The movement time increased as the index of difficulty (ID) value increased, echoing with the Fitts’ Law prediction. When aiming to a target with a higher ID, participants’ EEET was longer comparing to when reaching for a target with a lower ID. Participants reduced the movement speed to give themselves a longer time in searching visually for the target information before moving their hands. In contrast to our hypothesis, results suggested a proportional relationship between the task difficulty and the early eye engagement time. Participants also performed an increasing number of eye adjustments over the course of moving from the easy to the hard target. Future research is needed to examine eye hand coordination under the regulation of Fitts’ Law.

A large part of health education involves acquiring operational skills that require continued cooperation from the sense of sight. Traditional skill-training systems heavily rely on post-task evaluation and cannot automatically identify the tacit knowledge (e.g., visual attention allocation) that is used to support the decision-making and tool manipulation process. Recent research has started addressing this issue by identifying learner behaviours to understand learner needs and respond with personalized real-time support. To enable this type of adaptivity when training people in how to give injections, we conducted a study to investigate user's visual attention. We analyzed the eye-tracking data of 8 skilled medical practitioners and 10 student nurses from when they were performing an injection task on a simulated abdominal pad. Differences in the detected visual attention and eye-scan patterns across expertise levels create an opportunity for using overlay models or other diagnosis approaches to provide adaptive support to nurse trainees.

Surgery is a team effort. In this video, we display how we record two surgeons' eye motions during a simulated surgical operation, then performed Cross Recurrence Analysis (CRA) on the dual eye-tracking data to develop a valid technology to assess shared cognition. Twenty-two dyad teams were recruited to perform object transportation task using laparoscopic techniques. Outputs from CRA, including overlapping, recurrence rate and phase delay were correlated with team performance measured by the task time, errors made, and movement de-synchronization. Gaze behaviors between the two team members recorded in the surgical videos correlated positively with team performance. Elite teams were overlapping gaze more with higher recurrence rate than the poor teams. Dual eye-tracking analysis can be a useful tool for assessing team cognition and evaluating the team training.

Pupil size is known to correlate with the changes of cognitive task workloads, but how the pupil responds to requirements of basic goal-directed motor tasks involved in human-machine interactions is not yet clear. This work conducted a user study to investigate the pupil dilations during aiming in a tele-operation setting, with the purpose of better understanding how the changes in task requirements are reflected by the changes of pupil size. The task requirements, managed by Fitts' index of difficulty (ID), i.e. the size and distance apart of the targets, were varied between tasks, and pupil responses to different task IDs were recorded. The results showed that pupil diameter can be employed as an indicator of task requirements in goal-directed movements-higher task difficulty evoked higher valley to peak pupil dilation, and the peak pupil dilation occurred after a longer delay. These findings contribute to the foundation for developing methods to objectively evaluate interactive task requirements using pupil parameters during goal-directed movements in HCI.

Pupil size is known to correlate with changes of cognitive task workloads, but the pupillary response to requirements of basic goal-directed motor tasks is not yet clear, although pointing with tools is a ubiquitous human task. This work describes a user study to investigate the pupil dilations during aiming in two tele-operation tasks with different target settings, one aiming at targets with different sizes located at constant distance apart, and the other aiming at targets varying in different distances. The task requirements in each task were defined by Fitts' index of difficulty (ID). The purpose of this work is to further explore how the changes in task requirements are reflected by the changes of pupil size, i.e., whether the pupil responds to either target size or target distance, or to both of them. Pupil responses to different task IDs were recorded in each task. The results showed that the pupil responds to the changes of ID, not just to the change of target size. This implies that pupil diameter can be employed as an indicator of task requirement in goal-directed movements, because higher task difficulty evoked higher peak pupil dilation which occurred with longer delay. These findings can be used for detailed understanding of eye-hand coordination mechanisms in interactive systems and contribute to the foundation for developing methods to objectively evaluate interactive task requirements using pupil parameters during goal-directed movements.

Novices were trained to perform a unimanual peg transport task in a laparoscopic training box with an illuminated interior displayed on a monitor. Subjects were divided into two groups; one group was verbally instructed to direct their gaze at distant targets, while the other group had their gaze behaviour implicitly manipulated using distant target illumination. Both groups achieved similar task completion times post-training and developed peripheral vision strategies leading to delayed foveation on targets until the instrument was closer to its destination, although the ability to focus on targets earlier during manual movements as done by an expert surgeon was quickly regained by the verbal instruction group post-training. This suggests that care should be taken when employing visual attention cuing methods such as target highlighting for training eye-hand coordination skills, as simple verbal instruction may be sufficient to help trainees to adopt more expert-like gaze behaviours.

To observe whether there is a difference in eye gaze between doing a task, and watching a video of the task, we recorded the gaze of 17 subjects performing a simple surgical eye-hand coordination task. We also recorded eye gaze of the same subjects later while they were watching videos of their performance.

For gaze-based training in surgery to be meaningful, the similarity between a trainee's gaze and an expert's gaze during performance of surgical tasks must be assessed. As it is difficult to record two people's gaze simultaneously, we produced task videos made by experts, and measured the amount of overlap between the gaze path of the expert surgeon and third-party observers while watching the videos. For this investigation, we developed a new, simple method for displaying and summarizing the proportion of time during which two observers' points of gaze on a common stimulus were separated by no more than a specified visual angle.

The study of surgeons' eye movements is an innovative way of assessing skill and situation awareness, in that a comparison of eye movement strategies between expert surgeons and novices may show differences that can be used in training.