Eye-hand coordination under the microscope is a fundamental skill to many microsurgical specializations. In this work, we investigate the oculomotor and motor differences between novice and expert surgeons in the classical eye-hand coordination Fitts’ law task. However, the field of view under the microscope is never static or stable due to inherent internal and external artifacts. In this work, we present methodological steps necessary for stabilizing the operation field of view using RANSAC and for defining static areas of interest (i.e., task targets) and dynamic areas of interest using YOLOv8 (i.e., instrument movement). When combined, we observed that under the microscope, eye movements were spatially more focused than hand movements and that experts relied on parafoveal vision when targeting large targets. Our work opens the path to objective and autonomous skill-acquisition assessment in microsurgical expertise.

Pupillary responses have long been used in a range of psycho-socio-cognitive and motor-control studies of human effort, such as listening effort during diagnostic tests, driving fatigue, surgical training workload, or as a basis for intelligent tutor supports. When pupillary metrics get systematically surveyed, however, the strength of association with valued outcomes is often inconsistent. Furthermore, the complex origin of pupillary responses makes their measurement especially susceptible to confounds, which increases the odds of false positive and false negative findings. In this work, we address the "natural selection of bad science" and open the discussion on the underlying flaws of pupillary metrics and, oftentimes, covert decisions preceding their calculations. We advocate for bare-minimum reporting for explainable and reproducible pupillary metrics.

Surgical training is inherently dual endeavor, where the instructor navigates the trainee in the complex human anatomy while the trainee follows and implements these steps. To understand the training process and skill acquisition, however, eye-tracking studies mainly focused on the attention of individual medical students and in more controlled simulation training observed on an attached monitor. In this work, we explored the authentic surgical training of tracheostomy and developed customized eye-tracking glasses and experimental protocol specifically for use in surgical training. The customized eye trackers were tested with surgical instructors and medical students. We discuss the challenges posed for eye tracking studies in situ and present eye-tracking customization necessary for the dual eye tracking in the operating field of view (FOV).

Microsurgery requires visuomotor coordination under high magnification, where depth and scale are distorted. Understanding how surgeons adapt eye-hand coordination under the surgical microscope is key to advancing microsurgical education. We investigated gaze behavior and performance differences between expert plastic surgeons and novices during microsurgical suturing. An ocular-mounted eye-tracking system magnetically attached to the microscope preserved naturalistic gaze patterns while maintaining authentic spatial and visual conditions. Suturing was decomposed into phases: needle alignment and tissue piercing, pull-through, and knot tying, for phase-specific analysis. Performance was evaluated using the UWOMSA, and perceived workload was measured with the SURG-TLX. Experts completed the overall task faster than novices, exhibited shorter mean fixations overall, and showed the largest event-level advantage during knot tying. Findings indicate that visual strategies in microsurgery are phase-dependent and expertise-specific. In conclusion, ocular-mounted eye-tracking offers a valuable framework for studying microsurgical visuomotor control and developing objective, data-driven approach for microsurgical training.

Hyperspectral imaging (HSI) can enhance the recognition of normal and pathological tissues exposed during microscopic or endoscopic surgeries. However, robust HSI classification models would require meticulous documentation of the tissue-specific optical properties to account for individual variation and intraoperative factors. Publicly available HSI databases are yet scarce or lack relevant metadata, anatomical accuracy, and patients' characteristics which limits the clinical utility of the data. The essential problem is that clinical standards for HSI acquisition and archival do not exist. We collected a total of 52 microsurgical HSI images from 10 patients using our customized HSI system for the operation microscopes. We annotated the relevant microanatomical structures and labeled the tissue areas intended for HSI analyses. Using the collected HSI data, we developed the initial design of the microneurosurgical HSI database. The HSI database allows to display and query anatomical annotations, localizing magnetic resonance imaging (MRI) scans, operation videos, tissue labels, and HSI spectra per individual patient. Here we present the fundamental structures and functions of the HSI database in development. Our clinical HSI database will provide grounds for further development of HSI algorithms and machine-learning applications in microscopic and endoscopic surgery. Future collaborative research will establish clinical HSI standards with approved supporting technologies.

Given the widespread adverse outcomes of stress – exacerbated by the current pandemic – wearable sensing provides unique opportunities for automated stress tracking to inform well-being interventions. However, its success in the wild and at scale depends on the robustness and validity of automated stress inference, which is limited in current systems. In this work, we enumerate the properties of robustness and validity necessary for achieving viable automated stress inference using wearable sensors, and we underscore present challenges to constructing and evaluating these systems. Using these criteria as guiding principles, we present automated stress inference results from a large (N=606)in situlongitudinal wearable and contextual sensing study of information workers. Using a multimodal approach encompassing a wearable sensor, relative location tracking, smartphone usage, and environmental sensing, we trained regression models to predict daily self-reported perceived stress in a participant-independent fashion. Our models significantly outperformed baseline variants with shuffled stress scores and were consistent with small-to-moderate effects. Our findings highlight the performance disparity between robust and valid approaches to automated perceived stress inference and current approaches and suggest that further performance gains might require additional sensing modalities and enhanced contextual awareness than existing approaches.

Optic image-guidance systems enable minimally invasive (MIS) approaches in surgery. However, available MIS-techniques limits both ergonomics and field of view (FoV), which can be detrimental for anatomical awareness and safe manipulation with tissues. Contemporary navigation techniques (i.e. neuronavigation) support spatial awareness during surgery. However, these techniques require time-consuming instrumentation and lack real-time precision needed in soft-tissue surgery. In this work, we utilize operative microscopes FoV as an unobtrusive source to support MIS-navigation with micro-instrument tracking. The FoV instrument tracking has been investigated in laparoscopy, however, high magnification, selection of instruments and bimanually variant characteristics of microneurosurgery make the current computational approaches challenging to adopt. In this work, we investigate potentials of spectral imaging for micro-instrument tracking. We present a spectral-imaging system suitable for the use at the operation rooms. Using a hyperspectral camera mounted to the side ocular of operation microscope and Xenon white light illumination, we collected samples of standard microsurgical instruments (reflective and non-reflective) that were positioned in a biological tissue (placenta). In the analysis of contrasts, we compared spectral images to traditional RGB. We observed 8-13% contrast enhancement with the optimal wavelength bands and 20.4% improvement in instrument-tracking time. Our results encourage application of wavelength-tuned cameras to improve efficiency of optic tracking in MIS-systems.

In this work-in-progress, we present the design and implementation of a novel higher education medical computing continuous program that addresses on-demand, relevant clinical and health-care industry needs. A consortium of computing science and engineering academics and researchers, entrepreneurship and industry representatives, and clinicians is jointly designing an ecosystem that channels relevant needs from medical, clinical, and industrial environments into requirements for medical computing academic training. The design of this interdisciplinary program adheres to participatory design principles. Here, we outline the motivation and demands of such program, the design of the program, the principles and practices planned and implemented, and we report on first implementation steps. Since the graduate programs of computing and engineering mostly provide broader education, the graduates from these programs lack relevant clinical perspectives, necessary to work in- and advance the domain of medical computing. Likewise, the graduates from clinical and medical programs are often missing up-to-date computing skills, essential to comprehend and manage the fast-developing industry of medical computing technology. By implementing the processes of funnelling industrial and clinical aspects, we ensure that the graduates are equipped with relevant knowledge, and become a part of germane networks. We also report on challenges, experience, and lessons learned.

Collaborative problem solving (CPS) in virtual environments is an increasingly important context of 21st century learning. However, our understanding of this complex and dynamic phenomenon is still limited. Here, we examine unimodal primitives (activity on the screen, speech, and body movements), and their multimodal combinations during remote CPS. We analyze two datasets where 116 triads collaboratively engaged in a challenging visual programming task using video conferencing software. We investigate how UI-interactions, behavioral primitives, and multimodal patterns were associated with teams' subjective and objective performance outcomes. We found that idling with limited speech (i.e., silence or backchannel feedback only) and without movement was negatively correlated with task performance and with participants' subjective perceptions of the collaboration. However, being silent and focused during solution execution was positively correlated with task performance. Results illustrate that in some cases, multimodal patterns improved the predictions and improved explanatory power over the unimodal primitives. We discuss how the findings can inform the design of real-time interventions for remote CPS.

Whereas social visual attention has been examined in computer-mediated (e.g., shared screen) or video-mediated (e.g., FaceTime) interaction, it has yet to be studied in mixed-media interfaces that combine video of the conversant along with other UI elements. We analyzed eye gaze of 37 dyads (74 participants) who were tasked with negotiating the price of a new car (as a buyer and seller) using mixed-media video conferencing under competitive or cooperative negotiation instructions (experimental manipulation). We used multidimensional recurrence quantification analysis to extract spatio-temporal patterns corresponding to mutual gaze (individuals look at each other), joint attention (individuals focus on the same elements of the interface), and gaze aversion (an individual looks at their partner, who is looking elsewhere). Our results indicated that joint attention predicted the sum of points attained by the buyer and seller (i.e., the joint score). In contrast, gaze aversion was associated with faster time to complete the negotiation, but with a lower joint score. Unexpectedly, mutual gaze was highly infrequent and unrelated to the negotiation outcomes and none of the gaze patterns predicted subjective perceptions of the negotiation. There were also no effects of gender composition or negotiation condition on the gaze patterns or negotiation outcomes. Our results suggest that social visual attention may operate differently in mixed-media collaborative interfaces than in face-to-face interaction. As mixed-media collaborative interfaces gain prominence, our work can be leveraged to inform the design of gaze-sensitive user interfaces that support remote negotiations among other tasks.

A large dataset that contains the eye movements of N=216 programmers of different experience levels captured during two code comprehension tasks is presented. Data are grouped in terms of programming expertise (from none to high) and other demographic descriptors. Data were collected through an international collaborative effort that involved eleven research teams across eight countries on four continents. The same eye tracking apparatus and software was used for the data collection. The Eye Movements in Programming (EMIP) dataset is freely available for download. The varied metadata in the EMIP dataset provides fertile ground for the analysis of gaze behavior and may be used to make novel insights about code comprehension.

Multiparty collaborative problem solving - an increasingly important context in the 21st century workforce - suffers from a degradation of social and behavioral signals when attempted remotely, resulting in suboptimal outcomes. We investigate teams' multidimensional patterns of visual attention during a collaborative problem-solving task with an eye for leveraging insights to improve collaborative interfaces. Fifty-seven novices (forming 19 triads) engaged in a challenging programming task (Minecraft Hour of Code) using videoconferencing software with screen sharing. To discover patterns of individual-level gaze-UI coupling(coordination of a teammate's attention with respect to changes in the user interface) and team-level gaze-UI regularity (dynamics of teams' collective attention in context with changes in the user interface), we applied cross- and multidimensional recurrence quantification analyses, respectively. Individuals' eye gaze was significantly coupled with the ongoing screen activity whereas teams displayed significant patterns of gaze regularity, suggesting repetitive patterns in teams' attention. These measures predicted expert-coded collaborative processes of constructing shared knowledge and negotiation and coordination (but not maintaining team function) and correlated with task score (r = .425). They also predicted individually assessed subjective perceptions of team performance and the collaboration process, but not individual's learning or team's task scores. We discuss implications of our findings for the design of intelligent collaborative interfaces.

Collaborative problem solving (CPS) is a crucial 21st century skill; however, current technologies fall short of effectively supporting CPS processes, especially for remote, computer-enabled interactions. In order to develop next-generation computer-supported collaborative systems that enhance CPS processes and outcomes by monitoring and responding to the unfolding collaboration, we investigate automated detection of three critical CPS process ? construction of shared knowledge, negotiation/coordination, and maintaining team function ? derived from a validated CPS framework. Our data consists of 32 triads who were tasked with collaboratively solving a challenging visual computer programming task for 20 minutes using commercial videoconferencing software. We used automatic speech recognition to generate transcripts of 11,163 utterances, which trained humans coded for evidence of the above three CPS processes using a set of behavioral indicators. We aimed to automate the trained human-raters' codes in a team-independent fashion (current study) in order to provide automatic real-time or offline feedback (future work). We used Random Forest classifiers trained on the words themselves (bag of n-grams) or with word categories (e.g., emotions, thinking styles, social constructs) from the Linguistic Inquiry Word Count (LIWC) tool. Despite imperfect automatic speech recognition, the n-gram models achieved AUROC (area under the receiver operating characteristic curve) scores of .85, .77, and .77 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively; these reflect 70%, 54%, and 54% improvements over chance. The LIWC-category models achieved similar scores of .82, .74, and .73 (64%, 48%, and 46% improvement over chance). Further, the LIWC model-derived scores predicted CPS outcomes more similar to human codes, demonstrating predictive validity. We discuss embedding our models in collaborative interfaces for assessment and dynamic intervention aimed at improving CPS outcomes.

Imaging is an integral part of most operating room procedures and is used on a daily basis in the operating theater. Near real-time imaging during surgical procedures can significantly enhance the procedures by providing information about the anatomy and pathological conditions. So far, customizable spectral-imaging systems were only rarely investigated, developed and even less often installed in operational settings, especially microsurgery. In this paper we describe a design of portable imaging system for high throughput spectral characterization of ex vivo samples. The setup presented in this work was used for non-invasive collection of spectral signatures from a set of biological tissues.

Numerous visualization tools are involved in surgeon's decision making during a procedure. Image-guided navigation systems such as computed tomography and magnetic resonance have become an integral part of many surgical procedures. In tumor removal microsurgeries, the distinction between a tumor tissue and surrounding normal tissues are often negligible, highly impacted by color contrast and illumination of operative field. To enhance surgical decision making, we investigate the use of real-time spectral imaging in operating room. Coupling a spectral camera with a surgical microscope, however, is challenging due to numerous standards and ergonomic requirements of operating room. In this paper, we fulfill these ergonomic considerations and describe the process of integration of spectral camera in the clinical environment.

Eye-hand coordination is a central skill in microsurgery. To develop efficacious microsurgical training environments to support development of eye-hand coordination, it is important to understand the workload associated with visuo-motor tasks in microsurgery. We embedded an eye-tracker to a surgical microscope and collected eye-blink data of 10 participants during a microsuture training task. Blink-rate was shown to drop to low levels compared to a resting-state rate and be sensitive to the phases of microsurgical suture. We discuss these findings in the light of operator training in microsurgical environments.

In video-based training, clinicians practice and advance their skills on surgeries performed by their colleagues and themselves. Although microsurgeries are recorded daily, training centers are lacking the workforce to manually annotate the segments important for practitioners, such as instrument presence and position. In this work, we propose intelligent instrument detection using Convolutional Neural Network (CNN) to augment microsurgical training. The network was trained on real microsurgical practice videos for which human annotators manually gathered a large corpus of instrument positions. Under challenging conditions of highly magnified and often blurred view, the CNN was capable to correctly detect a needle-holder (a dominant tool in suturing practice) with 78.3% accuracy (F-score = 0.84) with recognition speed above 15 FPS. The result is promising in the emerging domain of augmented medical training where instrument recognition presents benefits to the microsurgical training.

Developing cross-device multi-user interfaces (UIs) is a challenging problem. There are numerous ways in which content and interactivity can be distributed. However, good solutions must consider multiple users, their roles, their preferences and access rights, as well as device capabilities. Manual and rule-based solutions are tedious to create and do not scale to larger problems nor do they adapt to dynamic changes, such as users leaving or joining an activity. In this paper, we cast the problem of UI distribution as an assignment problem and propose to solve it using combinatorial optimization. We present a mixed integer programming formulation which allows real-time applications in dynamically changing collaborative settings. It optimizes the allocation of UI elements based on device capabilities, user roles, preferences, and access rights. We present a proof-of-concept designer-in-the-loop tool, allowing for quick solution exploration. Finally, we compare our approach to traditional paper prototyping in a lab study.

Pupillary responses have been for long linked to cognitive workload in numerous tasks. In this work, we investigate the role of pupil dilations in the context of microsurgical training, handling of microinstruments and the suturing act in particular. With an eye-tracker embedded on the surgical microscope oculars, eleven medical participants repeated 12 sutures of artificial skin under high magnification. A detailed analysis of pupillary dilations in suture segments revealed that pupillary responses indeed varied not only according to the main suture segments but also in relation to participants' expertise.

When modeling natural conversational behavior of an agent, a head direction becomes an intuitive proxy to visual attention. We examine this assumption and carefully investigate the relationship between head directions and gaze dynamics through the use of eye-movement tracking. In a group conversation settings, we analyze relationships of the two nonverbal social signals - head directions and gaze dynamics - linked to influential and non-influential statements. We develop a clustering method to estimate the number of gaze targets. We employ this method to show that head and gaze dynamic behaviors are not correlated, and thus head cannot be used as a direct proxy to a person's gaze in the context of conversations. We also describe in detail how influential statements affect head and gaze behaviors. The findings have implications on methodology, modeling and design of natural conversational agents and present a supportive evidence for employing gaze-tracking into the future conversational technologies.

Heatmap is one of the most popular visualizations of gaze behavior, however, increasingly voluminous streams of eye-tracking data make processing of such visualization computationally demanding. Because of high requirements on a single processing machine, real-time visualizations from multiple users are unfeasible if rendered locally. We designed a framework that collects data from multiple eye-trackers regardless of their physical location, analyses these streams, and renders heatmaps in real-time. We propose a cloud computing architecture (EyeCloud) consisting of master and slave nodes on a cloud cluster, and a web interface for fast computation and effective aggregation of the large volumes of eye-tracking data. In experimental studies of the feasibility and effectiveness, we built a cloud cluster on a well-known service, implemented the architecture and reported on a comparison between the proposed system and traditional local processing. The results showed efficiency of the EyeCloud when recordings vary in durations. To our knowledge, this is the first solution to implement cloud computing for gaze visualization.

Interaction intent prediction and the Midas touch have been a longstanding challenge for eye-tracking researchers and users of gaze-based interaction. Inspired by machine learning approaches in biometric person authentication, we developed and tested an offline framework for task-independent prediction of interaction intents. We describe the principles of the method, the features extracted, normalization methods, and evaluation metrics. We systematically evaluated the proposed approach on an example dataset of gaze-augmented problem-solving sessions. We present results of three normalization methods, different feature sets and fusion of multiple feature types. Our results show that accuracy of up to 76% can be achieved with Area Under Curve around 80%. We discuss the possibility of applying the results for an online system capable of interaction intent prediction.