Atypical gaze behavior is a diagnostic hallmark of Autism Spectrum Disorder (ASD), playing a substantial role in the social and communicative challenges that individuals with ASD face. This study explores the impacts of a month-long, in-home intervention designed to promote triadic interactions between a social robot, a child with ASD, and their caregiver. Our results indicate that the intervention successfully promoted appropriate gaze behavior, encouraging children with ASD to follow the robot's gaze, resulting in more frequent and prolonged instances of spontaneous eye contact and joint attention with their caregivers. Additionally, we observed specific timelines for behavioral variability and novelty effects among users. Furthermore, diagnostic measures for ASD emerged as strong predictors of gaze patterns for both caregivers and children. These results deepen our understanding of ASD gaze patterns and highlight the potential for clinical relevance of robot-assisted interventions.

The International Society for Clinical Eye Tracking (ISCET) serves as a global platform for promoting international consensus on open standards on clinical eye tracking. Originally formed in March 2023, ISCET was created to facilitate collaboration, knowledge exchange, and advancements in clinical eye tracking applications, with the ultimate goal of fostering interdisciplinary research and improving clinical outcomes. Through collaborative and interdisciplinary efforts, ISCET’s current mission is to provide guidance on conducting eye-tracking tasks in clinical settings, unify clinicians’ voices, and maintain reference datasets for normative comparisons. ISCET now has 80 members, spread across the globe. In the Europe/Africa region meeting on January 24, 2024, ISCET established subcommittees to address specific needs. This paper outlines the rationale behind ISCET’s formation, its mission, objectives, ongoing initiatives, and organizational structure. Ongoing work focuses on surveying current international clinical eye tracker usage to inform standards development.

The International Society for Clinical Eye Tracking (ISCET) serves as a global platform for promoting international consensus on open standards in the application of eye tracking to clinical areas of focus. Formed in March 2023, ISCET facilitates collaboration, knowledge exchange, and advancements in clinical eye-tracking applications, with the goal of fostering interdisciplinary research and improving clinical outcomes. Through collaborative and interdisciplinary efforts, ISCET aims to provide guidance on conducting eye-tracking tasks in clinical settings, unify clinicians’ voices, and maintain reference datasets for normative comparisons. Convening during its latest meeting for the Europe/Africa region on January 24, 2024, ISCET established subcommittees to address specific needs. This paper outlines the rationale behind ISCET’s formation, its mission, objectives, ongoing initiatives, and organizational structure. Ongoing work focuses on surveying current international clinical eye tracker usage to inform standards development.

This study tracked eye movement in children with and without autism spectrum disorder (ASD) watching emotional biological and non-biological motion point-light-displays (PLDs). Older children with ASD focused on extremities while older typically developing (TD) children looked at figure’s heads, whereas not evident in the younger groups. These results suggest developmental advances in social-information biases in TD children not evident in children with ASD, together with atypical and potentially adaptive increases in attentional biases towards local motion cues with age in ASD. Potential avenues for future computational and methodological analyses are discussed.

Time of Day (ToD) of eye-tracking data acquisition may be a confounding variable that disproportionately impacts certain groups. This research examines attentional differences between infants with Lower-Likelihood (LL) and Higher-Likelihood (HL) for atypical developmental outcomes. We find that LL infants tend to pay more overall attention to eye-tracking probes during midday than their HL counterparts. Future research should examine and address factors underlying ToD-associated group differences and explore frameworks for systematically addressing additional eye-tracking confounding variables.

Data loss in eye-tracking studies is often considered a nuisance variable or noise. This study examined the value of data loss in eye tracking and proposed a new method to utilize lost data in predicting the clinical characteristics of autism spectrum disorder (ASD). We used eye tracking to confirm previous findings on atypical attention patterns and further utilized behavior coding to examine the three types of causes of data loss including blinks, non-compliant behaviors, and technical errors. We discovered that data loss due to blinking was associated with a lack of interest in social cues, and data loss due to non-compliance predicted a greater severity of ASD symptoms. These results suggest that the loss of data in eye tracking is meaningful as a measure of diminished social attention and a reflection of clinical characteristics in ASD.

Face swapping combines one face’s identity with another face’s non-appearance attributes (expression, head pose, lighting) to generate a synthetic face. This technology is rapidly improving, but falls flat when reconstructing some attributes, particularly gaze. Image-based loss metrics that consider the full face do not effectively capture the perceptually important, yet spatially small, eye regions. Improving gaze in face swaps can improve naturalness and realism, benefiting applications in entertainment, human computer interaction, and more. Improved gaze will also directly improve Deepfake detection efforts, serving as ideal training data for classifiers that rely on gaze for classification. We propose a novel loss function that leverages gaze prediction to inform the face swap model during training and compare against existing methods. We find all methods to significantly benefit gaze in resulting face swaps.

Background: Looking pattern differences are shown to separate individuals with Autism Spectrum Disorder (ASD) and Typically Developing (TD) controls. Recent studies have shown that, in children with ASD, these patterns change with intellectual and social impairments, suggesting that patterns of social attention provide indices of clinically meaningful variation in ASD. Method: We conducted a naturalistic study of children with ASD (n = 55) and typical development (TD, n = 32). A battery of eye-tracking video stimuli was used in the study, including Activity Monitoring (AM), Social Referencing (SR), Theory of Mind (ToM), and Dyadic Bid (DB) tasks. This work reports on the feasibility of spatial and spatiotemporal scanpaths generated from eye-gaze patterns of these paradigms in stratifying ASD and TD groups. Algorithm: This article presents an approach for automatically identifying clinically meaningful information contained within the raw eye-tracking data of children with ASD and TD. The proposed mechanism utilizes combinations of eye-gaze scan-paths (spatial information), fused with temporal information and pupil velocity data and Convolutional Neural Network (CNN) for stratification of diagnosis (ASD or TD). Results: Spatial eye-gaze representations in the form of scanpaths in stratifying ASD and TD (ASD vs. TD: DNN: 74.4%) are feasible. These spatial eye-gaze features, e.g., scan-paths, are shown to be sensitive to factors mediating heterogeneity in ASD: age (ASD: 2–4 y/old vs. 10–17 y/old CNN: 80.5%), gender (Male vs. Female ASD: DNN: 78.0%) and the mixture of age and gender (5–9 y/old Male vs. 5–9 y/old Female ASD: DNN:98.8%). Limiting scan-path representations temporally increased variance in stratification performance, attesting to the importance of the temporal dimension of eye-gaze data. Spatio-Temporal scan-paths that incorporate velocity of eye movement in their images of eye-gaze are shown to outperform other feature representation methods achieving classification accuracy of 80.25%. Conclusion: The results indicate the feasibility of scan-path images to stratify ASD and TD diagnosis in children of varying ages and gender. Infusion of temporal information and velocity data improves the classification performance of our deep learning models. Such novel velocity fused spatio-temporal scan-path features are shown to be able to capture eye gaze patterns that reflect age, gender, and the mixed effect of age and gender, factors that are associated with heterogeneity in ASD and difficulty in identifying robust biomarkers for ASD.

Gaze calibration is common in traditional infrared oculographic eye tracking. However, it is not well studied in visible-light mobile/remote eye tracking. We developed a lightweight real-time gaze error estimator and analyzed calibration errors from two perspectives: facial feature-based and Monte Carlo-based. Both methods correlated with gaze estimation errors, but the Monte Carlo method associated more strongly. Facial feature associations with gaze error were interpretable, relating movements of the face to the visibility of the eye. We highlight the degradation of gaze estimation quality in a sample of children with autism spectrum disorder (as compared to typical adults), and note that calibration methods may improve Euclidean error by 10%.

Identifying oculomotor behaviors relevant for eye-tracking applications is a critical but often challenging task. Aiming to automatically learn and extract knowledge from existing eye-tracking data, we develop a novel method that creates rich representations of oculomotor scanpaths to facilitate the learning of downstream tasks. The proposed stimulus-agnostic Oculomotor Behavior Framework (OBF) model learns human oculomotor behaviors from unsupervised and semi-supervised tasks, including reconstruction, predictive coding, fixation identification, and contrastive learning tasks. The resultant pre-trained OBF model can be used in a variety of applications. Our pre-trained model outperforms baseline approaches and traditional scanpath methods in autism spectrum disorder and viewed-stimulus classification tasks. Ablation experiments further show our proposed method could achieve even better results with larger model sizes and more diverse eye-tracking training datasets, supporting the model’s potential for future eye-tracking applications. Open source code: http://github.com/BeibinLi/OBF.

Eye tracking has become a powerful tool in the study of autism spectrum disorder (ASD). Current, large-scale efforts aim to identify specific eye-tracking stimuli to be used as biomarkers for ASD, with the intention of informing the diagnostic process, monitoring therapeutic response, predicting outcomes, or identifying subgroups with the spectrum. However, there are hundreds of candidate experimental paradigms, each of which contains dozens or even hundreds of individual stimuli. Each stimuli is associated with an array of potential derived outcome variables, thus the number of variables to consider can be enormous. Standard variable selection techniques are not applicable to this problem, because selection must be done at the level of stimuli and not individual variables. In other words, this is a grouped variable selection problem. In this work, we apply lasso, group lasso, and a new technique, Sparsely Grouped Input Variables for Neural Network (SGIN), to select experimental stimuli for group discrimination and regression with clinical variables. Using a dataset obtained from children with and without ASD who were administered a battery containing 109 different stimuli presentations involving 9647 features, we are able to retain strong group separation even with only 11 out of the 109 stimuli. This work sets the stage for concerted techniques designed around engines to iteratively refine and define next-generation biomarkers using eye tracking for psychiatric conditions. http://github.com/beibinli/SGIN

In this paper, we introduce an end-to-end machine learning-based system for classifying autism spectrum disorder (ASD) using facial attributes such as expressions, action units, arousal, and valence. Our system classifies ASD using representations of different facial attributes from convolutional neural networks, which are trained on images in the wild. Our experimental results show that different facial attributes used in our system are statistically significant and improve sensitivity, specificity, and F1 score of ASD classification by a large margin. In particular, the addition of different facial attributes improves the performance of ASD classification by about 7% which achieves a F1 score of 76%.

Most studies of executive function (EF) in Autism Spectrum Disorder (ASD) focus on cognitive information processing, emphasizing less the social interaction deficits core to ASD. We designed a mobile game that uses social and nonsocial stimuli to assess children's EF skills. The game comprised three components involving different EF skills: cognitive flexibility (shifting/inference), inhibitory control, and short-term memory. By recruiting 65 children with and without ASD to play the mobile game, we investigated the potential of such platforms for capturing important phenotypic characteristics of individuals with autism. Results highlighted between-diagnostic-group differences in playing patterns with children with ASD showing broad patterns of EF deficits, but with relative strengths in nonsocial short-term memory, and preserved response to emotional inhibition cues. We showed the system could predict IQ, an important target for clinical treatment, towards the goal of developing platforms to act as long-term, efficient, and effective behavioral biomarkers for ASD.

Augmentative and Alternative Communication (AAC) apps are apps that enable non-speech communicative forms. One class of AAC apps are speech-generating devices (SGDs), where icons/pictures are tapped to produce spoken words. These apps are widely used to support communication and language learning for individuals with disabilities such as autism spectrum disorder (ASD). Given that these apps are used in everyday scenarios, they can generate massive streams of data, providing a wealth of information regarding individual usage patterns and for developing usage model profiles. However, the utility and potential of these streams of data has been little explored from a data mining perspective. The objective of this study is to evaluate several feature representations of usage patterns, coupled with data mining and data modelling techniques, for identifying differences in AAC usage patterns between users with and without ASD. The study is conducted using data streams aggregated from an AAC app called FreeSpeech, specifically designed for individuals with learning disabilities and ASD. Several feature representations for modeling usage profiles based on temporal, behavioral and frequency of usage, are investigated. The potential of each usage representation is assessed using a collection of well-known and well-established learning methods such as support vector machine and ensemble learning. While, in general, prediction performance was only slightly above chance in most representations, results from unsupervised class labeling experiments showed promising results regarding the potential of stationary keypress usage representations with bootstrapped ensembles for separating ASD from non-ASD users.

This paper modifies the DBSCAN algorithm to identify fixations and saccades. This method combines advantages from dispersion-based algorithms, such as resilience to noise and intuitive fixational structure, and from velocity-based algorithms, such as the ability to deal appropriately with smooth pursuit (SP) movements.

Researchers use fixation identification algorithms to parse eye movement trajectories into a series of fixations and saccades, simplifying analyses and providing measures which may relate to cognition. The Distance Dispersion (I-DD) a widely-used elementary fixation identification algorithm. Yet the "optimality" properties of its most popular greedy implementation have not been described. This paper: (1) asks how "optimal" should be defined, and advances maximizing total fixation time and minimizing number of clusters as a definition; (2) asks whether the greedy implementation of I-DD is optimal, and shows that it is when no fixations are rejected for being too short; and (3) we show that when fixation time rejection criterion are enabled, the greedy algorithm is not optimal. We propose an O(n2) algorithm which is.

Far infrared thermography, which can be used to detect thermal radiation emitted by humans, has been used to detect physical disease, physiological changes relating to emotion, and polygraph testing, but has not been used for eye tracking. However, because the surface temperature of the cornea is colder than the limbus, it is theoretically possible to track corneal movements through thermal imaging. To explore the feasibility of thermal eye tracking, we invited 10 adults and tracked their corneal movements with passive thermal imaging at 60 Hz. We combined shape models of eyes with intensity threshold to segment the cornea from other parts of the eye in thermal images. We used an animation sequence as a calibration target for 5 point calibration/validation 5 times. Our results were compared to simultaneously collected data using an SR EyeLink eye tracker at 500 Hz, demonstrating the feasibility of eye tracking with thermal images. Blinking and breathing frequencies, which reflect the psychophysical status of the participants, were also robustly detected during thermal eye tracking.

Robots are often employed to proactively engage children with Autism Spectrum Disorder (ASD) in well-defined physical or social activities to promote specific educational or therapeutic outcomes. However, much can also be learned by leveraging a robot's unique ability to objectively deliver stimuli in a consistent, repeatable way and record child-robot interactions that may be indicative of developmental ability and autism severity in this population. In this study, we elicited affective responses with an emotion-simulating robot and recorded child-robot interactions and child-other interactions during robot emotion states. This research makes two key contributions. First, we analyzed child-robot interactions and affective responses to an emotion-simulating robot to explore differences between the responses of typically developing children and children with Autism Spectrum Disorder (ASD). Next, we characterized play and affective responsivity and its connection to severity of autism symptoms using the Autism Diagnostic Observation Schedule (ADOS) calibrated severity scores. This preliminary work delivers a novel and robust robot-enabled technique for (1) differentiating child-robot interactions of a group of very young children with ASD (n=12) from a group of typically developing children (n=15) and, (2) characterizing within-group differences in play and affective response that may be associated with symptoms of autism severity.

In their expanding role as tutors, home and healthcare assistants, robots must effectively interact with individuals of varying ability and temperament. Indeed, deploying robots in long-term social engagements will almost certainly require robots to reliably detect and adapt to changes in the demeanor of social partners to promote trust and more productive collaboration. However, the recognition of emotional state typically relies on the interpretation of very subtle cues, often varying from one person to the next. In addition, while facial expressions, body posture and features of speech have been used to detect affective changes, the robustness of these measures is often hindered by cultural and age differences. Recently, infrared thermography has shown promise in detecting guilt, fear and stress, indicating that it may be a viable sensing modality for improved human-robot interaction. In this study, we evaluated the efficacy of using a far infrared (FIR) camera for detecting robot-elicited affective response compared to video-elicited affective response by tracking thermal changes in five areas of the face. Further, we analyzed localized changes in the face to assess whether thermal and electrodermal responses to emotions elicited by traditional video techniques and by robots are similar. Finally, we performed principal component analysis to reduce the dimensionality of data and evaluated the performance using machine learning techniques for classifying thermal data by emotion state, resulting in a thermal classifier with a performance accuracy of 77.5%.

This research employs an expressive robot to elicit affective response in young children and explore correlations between autonomously-detected play, affective response and developmental ability. In this study, we introduce a new, affective interface that combines sound, color, movement and context to simulate the expression of emotions. Our approach exploits social contingencies to emphasize the importance of situational cues in the proper interpretation of affective state. We studied a group of young children at various ages and stages of cognitive development, to: (1) evaluate the efficacy of using captured motion data to autonomously detect physical patterns of play while interacting with a robot, (2) examine relationships between physical play patterns and observed affective response and, (3) explore associations between developmental ability and play or affective response. This pilot study demonstrates that aggregate patterns of physical interaction with a robot are distinguishable through autonomous data collection. Further, statistical analyses demonstrates that developmental ability may be directly related to how a child interacts with and responds to an affective robot.

While interactive technologies frequently are designed to be enjoyable, there are particular reasons to prioritize this for technologies intended to support autism interventions. Most broadly, enjoyment of activities or materials used in interventions has been associated with heightened improvements in the behaviors targeted by the interventions. In the largest group study to date of school-aged children with high-functioning autism (N=24), we present evidence of more positive affect elicited with a robot than with an adult, during matched triadic interactions designed to facilitate social and conversational interaction with a clinician. Robot-mediated increases in positive affect were found to be associated with production of spoken language directed to the clinician during robot interaction. We further found that robot-mediated increases in positive affect were associated with greater autism severity, particularly in the social affect domain, and with lower nonverbal IQ. Our findings suggest that robots may have a unique advantage in interventions for children with autism spectrum disorders by eliciting more positive affect, and that we should explore robot support for interventions with lower-functioning, affected individuals.

This study presents a proof-of-concept study to assess the prediction of emotional perceptive competency and implicit affective preferences from each other. Predictions are made using linear regression, principal component analysis with linear regression, and support vector machines. Results point to a strong, bidirectional relationship between preference for emotional stimuli and affective competency. This work informs future studies predicting emotion processing abilities in humans and highlights the importance of tailoring interfaces to meet the emotional abilities of the user.

Emotional intelligence has been positively associated with social competence. In addition, attentional responses to emotional information have been associated with psychological characteristics related to mental health. In this study, we used the Microsoft Kinect platform as a tool to examine relationships between responses to emotional information in a gameplay environment and psychological factors. 45 typically developing individuals participated in the study, which involved 1) the Kinect game, requiring participants to engage in unprompted volitional whole-body responses to emotional stimuli, and 2) psychosocial assessments such as the Broader Autism Phenotype Questionnaire (BAPQ) and the Brief Symptom Inventory (BSI). Principal component analysis revealed patterns of gameplay that were associated with psychological characteristics of individuals. Preference for emotional content in general was associated with fewer social difficulties. The present work offers preliminary support for utilizing Kinect video games to understand emotion orienting and social capabilities, and shows that implicit patterns of preference identified during gameplay may relate to psychological and psychiatric phenomena. While the issues are complex and further research is needed, this work may inform the development of novel approaches to diagnostic and therapeutic tools.

Atypical looking behaviors in neuropsychiatric conditions such as autism spectrum disorders (ASD) are not only a reflection of inherently abnormal neuropsychological processes, but also suggest that future access to observational learning opportunities may be limited. The work presented in this paper uses interactive eye tracking as a first step towards the development of automated tools that can help toddlers and young children with atypical visual attention learn to attend to social information in a more typical fashion. In our study, we designed an automated visual strategy training system that would redirect a viewers' attention to locations highly salient to the normative control group when the viewer drifted from those locations for a significant period of time. We evaluated our experimental technique on typically-developing adults, obtaining results that suggest that looking patterns can be altered to be more similar to those evidenced by a normative group of young children. Furthermore, these alterations appear be retained in post-training sessions when considering new presentations of videos participants had been trained upon, and, on more sensitive outcome measures based on integrated scanpath probabilities (heatmaps), seemed to generalize presentations not trained upon as well. The development of these techniques may provide a new model for modifying attentional biases not only in toddlers with ASD, but also in children affected by other neuropsychiatric conditions, and may thus lead to new therapeutic interventions as well as more efficacious methods for identifying the patterns associated with abnormal, attention-driven experience.

Most analytic approaches for eye-tracking data focus either on identification of fixations and saccades, or on estimating saliency properties. Analyzing both aspects of visual attention simultaneously provides a more comprehensive view of strategies used to process information. This work presents a method that incorporates both aspects in a unified Bayesian model to jointly estimate dynamic properties of scanpaths and a saliency map. Performance of the model is assessed on simulated data and on eye-tracking data from 15 children with autism spectrum disorder and 13 control children. Saliency differences between ASD and TD groups were found for both social and non-social images, but differences in dynamic gaze features were evident in only a subset of social images. These results are consistent with previous region-based analyses as well as previous fixation parameter models, suggesting that the new approach may provide synthesizing and statistical perspectives on eye-tracking analyses.

Many different eye-tracking calibration techniques have been developed [e.g. see Talmi and Liu 1999; Zhu and Ji 2007]. A community standard is a 9-point-sparse calibration that relies on sequential presentation of known scene targets. However, fixating different points has been described as tedious, dull and tiring for the eye [Bulling, Gellersen, Pfeuffer, Turner and Vidal 2013].

Head-mounted eye-tracking systems allow us to observe participants' gaze behaviors in largely unconstrained, real-world settings. We have developed novel, untethered, mobile, low-cost, lightweight, easily-assembled head-mounted eye-tracking devices, comprised entirely of off-the-shelf components, including untethered, point-of-view, sports cameras. In total, the parts we have used cost ~$153, and we suggest untested alternative components that reduce the cost of parts to ~$31. Our device can be easily assembled using hobbying skills and techniques. We have developed hardware, software, and methodological techniques to perform point-of-regard estimation, and to temporally align scene and eye videos in the face of variable frame rate, which plagues low-cost, lightweight, untethered cameras. We describe an innovative technique for synchronizing eye and scene videos using synchronized flashing lights. Our hardware, software, and calibration designs will be made publicly available, and we describe them in detail here, to facilitate replication of our system. We also describe novel smooth-pursuit-based calibration methodology, which affords rich sampling of calibration data while compensating for lack of information regarding the extent of visibility on participants' scene recordings. Validation experiments indicate accuracy within 0.752 degrees of visual angle on average.

Fixation identification algorithms facilitate data comprehension and provide analytical convenience in eye-tracking analysis. However, current fixation algorithms for eye-tracking analysis are heavily dependent on parameter choices, leading to instabilities in results and incompleteness in reporting. This work examines the nature of human scanning patterns during complex scene viewing. We show that standard implementations of the commonly used distance-dispersion algorithm for fixation identification are functionally equivalent to greedy spatiotemporal tiling. We show that modeling the number of fixations as a function of tiling size leads to a measure of fractal dimensionality through box counting. We apply this technique to examine scale-free gaze behaviors in toddlers and adults looking at images of faces and blocks, as well as large number of adults looking at movies or static images. The distributional aspects of the number of fixations may suggest a fractal structure to gaze patterns in free scanning and imply that the incompleteness of standard algorithms may be due to the scale-free behaviors of the underlying scanning distributions. We discuss the nature of this hypothesis, its limitations, and offer directions for future work.

While there is a rich history of studies involving robots and individuals with autism spectrum disorders (ASD), few of these studies have made substantial impact in the clinical research community. In this paper we first examine how differences in approach, study design, evaluation, and publication practices have hindered uptake of these research results. Based on ten years of collaboration, we suggest a set of design principles that satisfy the needs (both academic and cultural) of both the robotics and clinical autism research communities. Using these principles, we present a study that demonstrates a quantitatively measured improvement in human-human social interaction for children with ASD, effected by interaction with a robot.

In this paper we evaluate several of the most popular algorithms for segmenting fixations from saccades by testing these algorithms on the scanning patterns of toddlers. We show that by changing the parameters of these algorithms we change the reported fixation durations in a systematic fashion. However, we also show how choices in analysis can lead to very different interpretations of the same eye-tracking data. Methods for reconciling the disparate results of different algorithms as well as suggestions for the use of fixation identification algorithms in analysis, are presented.

Computational models of development aim to describe the mechanisms that underlie the acquisition of new skills or the emergence of new capabilities. The strength of a model is judged by both its ability to explain the phenomena in question as well as its ability to generate new hypotheses, generalize to new situations, and provide a unifying conceptual framework. Although often constructed using traditional engineering methodologies, evaluating the performance of a computational model of development in terms of traditional perspectives is a flawed approach. This paper addresses the fundamental issues that confound quantitative analysis of computational models of developmental systems. In particular we focus on the following recommendations: 1) don't equate the success of a developmental model with its peak performance at some task; 2) don't employ purely subjective or vague measures of model fitness; and 3) don't hide or reject variation as found in the computational model. Along the way, we discuss the aspects of computational models of development that lead to the requirements for specialized methods of analysis.