Background: Looking pattern differences are shown to separate individuals with Autism Spectrum Disorder (ASD) and Typically Developing (TD) controls. Recent studies have shown that, in children with ASD, these patterns change with intellectual and social impairments, suggesting that patterns of social attention provide indices of clinically meaningful variation in ASD. Method: We conducted a naturalistic study of children with ASD (n = 55) and typical development (TD, n = 32). A battery of eye-tracking video stimuli was used in the study, including Activity Monitoring (AM), Social Referencing (SR), Theory of Mind (ToM), and Dyadic Bid (DB) tasks. This work reports on the feasibility of spatial and spatiotemporal scanpaths generated from eye-gaze patterns of these paradigms in stratifying ASD and TD groups. Algorithm: This article presents an approach for automatically identifying clinically meaningful information contained within the raw eye-tracking data of children with ASD and TD. The proposed mechanism utilizes combinations of eye-gaze scan-paths (spatial information), fused with temporal information and pupil velocity data and Convolutional Neural Network (CNN) for stratification of diagnosis (ASD or TD). Results: Spatial eye-gaze representations in the form of scanpaths in stratifying ASD and TD (ASD vs. TD: DNN: 74.4%) are feasible. These spatial eye-gaze features, e.g., scan-paths, are shown to be sensitive to factors mediating heterogeneity in ASD: age (ASD: 2–4 y/old vs. 10–17 y/old CNN: 80.5%), gender (Male vs. Female ASD: DNN: 78.0%) and the mixture of age and gender (5–9 y/old Male vs. 5–9 y/old Female ASD: DNN:98.8%). Limiting scan-path representations temporally increased variance in stratification performance, attesting to the importance of the temporal dimension of eye-gaze data. Spatio-Temporal scan-paths that incorporate velocity of eye movement in their images of eye-gaze are shown to outperform other feature representation methods achieving classification accuracy of 80.25%. Conclusion: The results indicate the feasibility of scan-path images to stratify ASD and TD diagnosis in children of varying ages and gender. Infusion of temporal information and velocity data improves the classification performance of our deep learning models. Such novel velocity fused spatio-temporal scan-path features are shown to be able to capture eye gaze patterns that reflect age, gender, and the mixed effect of age and gender, factors that are associated with heterogeneity in ASD and difficulty in identifying robust biomarkers for ASD.

Gaze calibration is common in traditional infrared oculographic eye tracking. However, it is not well studied in visible-light mobile/remote eye tracking. We developed a lightweight real-time gaze error estimator and analyzed calibration errors from two perspectives: facial feature-based and Monte Carlo-based. Both methods correlated with gaze estimation errors, but the Monte Carlo method associated more strongly. Facial feature associations with gaze error were interpretable, relating movements of the face to the visibility of the eye. We highlight the degradation of gaze estimation quality in a sample of children with autism spectrum disorder (as compared to typical adults), and note that calibration methods may improve Euclidean error by 10%.

Identifying oculomotor behaviors relevant for eye-tracking applications is a critical but often challenging task. Aiming to automatically learn and extract knowledge from existing eye-tracking data, we develop a novel method that creates rich representations of oculomotor scanpaths to facilitate the learning of downstream tasks. The proposed stimulus-agnostic Oculomotor Behavior Framework (OBF) model learns human oculomotor behaviors from unsupervised and semi-supervised tasks, including reconstruction, predictive coding, fixation identification, and contrastive learning tasks. The resultant pre-trained OBF model can be used in a variety of applications. Our pre-trained model outperforms baseline approaches and traditional scanpath methods in autism spectrum disorder and viewed-stimulus classification tasks. Ablation experiments further show our proposed method could achieve even better results with larger model sizes and more diverse eye-tracking training datasets, supporting the model’s potential for future eye-tracking applications. Open source code: http://github.com/BeibinLi/OBF.

Eye tracking has become a powerful tool in the study of autism spectrum disorder (ASD). Current, large-scale efforts aim to identify specific eye-tracking stimuli to be used as biomarkers for ASD, with the intention of informing the diagnostic process, monitoring therapeutic response, predicting outcomes, or identifying subgroups with the spectrum. However, there are hundreds of candidate experimental paradigms, each of which contains dozens or even hundreds of individual stimuli. Each stimuli is associated with an array of potential derived outcome variables, thus the number of variables to consider can be enormous. Standard variable selection techniques are not applicable to this problem, because selection must be done at the level of stimuli and not individual variables. In other words, this is a grouped variable selection problem. In this work, we apply lasso, group lasso, and a new technique, Sparsely Grouped Input Variables for Neural Network (SGIN), to select experimental stimuli for group discrimination and regression with clinical variables. Using a dataset obtained from children with and without ASD who were administered a battery containing 109 different stimuli presentations involving 9647 features, we are able to retain strong group separation even with only 11 out of the 109 stimuli. This work sets the stage for concerted techniques designed around engines to iteratively refine and define next-generation biomarkers using eye tracking for psychiatric conditions. http://github.com/beibinli/SGIN

Most studies of executive function (EF) in Autism Spectrum Disorder (ASD) focus on cognitive information processing, emphasizing less the social interaction deficits core to ASD. We designed a mobile game that uses social and nonsocial stimuli to assess children's EF skills. The game comprised three components involving different EF skills: cognitive flexibility (shifting/inference), inhibitory control, and short-term memory. By recruiting 65 children with and without ASD to play the mobile game, we investigated the potential of such platforms for capturing important phenotypic characteristics of individuals with autism. Results highlighted between-diagnostic-group differences in playing patterns with children with ASD showing broad patterns of EF deficits, but with relative strengths in nonsocial short-term memory, and preserved response to emotional inhibition cues. We showed the system could predict IQ, an important target for clinical treatment, towards the goal of developing platforms to act as long-term, efficient, and effective behavioral biomarkers for ASD.

Augmentative and Alternative Communication (AAC) apps are apps that enable non-speech communicative forms. One class of AAC apps are speech-generating devices (SGDs), where icons/pictures are tapped to produce spoken words. These apps are widely used to support communication and language learning for individuals with disabilities such as autism spectrum disorder (ASD). Given that these apps are used in everyday scenarios, they can generate massive streams of data, providing a wealth of information regarding individual usage patterns and for developing usage model profiles. However, the utility and potential of these streams of data has been little explored from a data mining perspective. The objective of this study is to evaluate several feature representations of usage patterns, coupled with data mining and data modelling techniques, for identifying differences in AAC usage patterns between users with and without ASD. The study is conducted using data streams aggregated from an AAC app called FreeSpeech, specifically designed for individuals with learning disabilities and ASD. Several feature representations for modeling usage profiles based on temporal, behavioral and frequency of usage, are investigated. The potential of each usage representation is assessed using a collection of well-known and well-established learning methods such as support vector machine and ensemble learning. While, in general, prediction performance was only slightly above chance in most representations, results from unsupervised class labeling experiments showed promising results regarding the potential of stationary keypress usage representations with bootstrapped ensembles for separating ASD from non-ASD users.

This paper modifies the DBSCAN algorithm to identify fixations and saccades. This method combines advantages from dispersion-based algorithms, such as resilience to noise and intuitive fixational structure, and from velocity-based algorithms, such as the ability to deal appropriately with smooth pursuit (SP) movements.

Robots are often employed to proactively engage children with Autism Spectrum Disorder (ASD) in well-defined physical or social activities to promote specific educational or therapeutic outcomes. However, much can also be learned by leveraging a robot's unique ability to objectively deliver stimuli in a consistent, repeatable way and record child-robot interactions that may be indicative of developmental ability and autism severity in this population. In this study, we elicited affective responses with an emotion-simulating robot and recorded child-robot interactions and child-other interactions during robot emotion states. This research makes two key contributions. First, we analyzed child-robot interactions and affective responses to an emotion-simulating robot to explore differences between the responses of typically developing children and children with Autism Spectrum Disorder (ASD). Next, we characterized play and affective responsivity and its connection to severity of autism symptoms using the Autism Diagnostic Observation Schedule (ADOS) calibrated severity scores. This preliminary work delivers a novel and robust robot-enabled technique for (1) differentiating child-robot interactions of a group of very young children with ASD (n=12) from a group of typically developing children (n=15) and, (2) characterizing within-group differences in play and affective response that may be associated with symptoms of autism severity.

In their expanding role as tutors, home and healthcare assistants, robots must effectively interact with individuals of varying ability and temperament. Indeed, deploying robots in long-term social engagements will almost certainly require robots to reliably detect and adapt to changes in the demeanor of social partners to promote trust and more productive collaboration. However, the recognition of emotional state typically relies on the interpretation of very subtle cues, often varying from one person to the next. In addition, while facial expressions, body posture and features of speech have been used to detect affective changes, the robustness of these measures is often hindered by cultural and age differences. Recently, infrared thermography has shown promise in detecting guilt, fear and stress, indicating that it may be a viable sensing modality for improved human-robot interaction. In this study, we evaluated the efficacy of using a far infrared (FIR) camera for detecting robot-elicited affective response compared to video-elicited affective response by tracking thermal changes in five areas of the face. Further, we analyzed localized changes in the face to assess whether thermal and electrodermal responses to emotions elicited by traditional video techniques and by robots are similar. Finally, we performed principal component analysis to reduce the dimensionality of data and evaluated the performance using machine learning techniques for classifying thermal data by emotion state, resulting in a thermal classifier with a performance accuracy of 77.5%.