Co-Training has achieved significant success in the field of semi-supervised learning(SSL); however, the homogenization phenomenon, which arises from multiple models tending towards similar decision boundaries, remains inadequately addressed. To tackle this issue, we propose a novel algorithm called β-FFT from the perspectives of data processing and training structure. In data processing, we apply diverse augmentations to input data and feed them into two sub-networks. To balance the training instability caused by different augmentations during consistency learning, we introduce a nonlinear interpolation technique based on the Fast Fourier Transform (FFT). By swapping low-frequency components between variously augmented images, this method not only generates smooth and diverse training samples that bridge different augmentations but also enhances the model’s generalization capability while maintaining consistency learning stability. In training structure, we devise a differentiated training strategy to mitigate homogenization in co-training. Specifically, we use labeled data for additional training of one model within the co-training framework, while for unlabeled data, we employ linear interpolation based on the Beta(β) distribution as a regularization technique in additional training. This approach allows for more efficient utilization of limited labeled data and simultaneously improves the model’s performance on unlabeled data, optimizing overall system performance. Code is available at the following link. https://github.com/Xi-Mu-Yu/beta-FFT.

Gaze calibration is common in traditional infrared oculographic eye tracking. However, it is not well studied in visible-light mobile/remote eye tracking. We developed a lightweight real-time gaze error estimator and analyzed calibration errors from two perspectives: facial feature-based and Monte Carlo-based. Both methods correlated with gaze estimation errors, but the Monte Carlo method associated more strongly. Facial feature associations with gaze error were interpretable, relating movements of the face to the visibility of the eye. We highlight the degradation of gaze estimation quality in a sample of children with autism spectrum disorder (as compared to typical adults), and note that calibration methods may improve Euclidean error by 10%.

Identifying oculomotor behaviors relevant for eye-tracking applications is a critical but often challenging task. Aiming to automatically learn and extract knowledge from existing eye-tracking data, we develop a novel method that creates rich representations of oculomotor scanpaths to facilitate the learning of downstream tasks. The proposed stimulus-agnostic Oculomotor Behavior Framework (OBF) model learns human oculomotor behaviors from unsupervised and semi-supervised tasks, including reconstruction, predictive coding, fixation identification, and contrastive learning tasks. The resultant pre-trained OBF model can be used in a variety of applications. Our pre-trained model outperforms baseline approaches and traditional scanpath methods in autism spectrum disorder and viewed-stimulus classification tasks. Ablation experiments further show our proposed method could achieve even better results with larger model sizes and more diverse eye-tracking training datasets, supporting the model’s potential for future eye-tracking applications. Open source code: http://github.com/BeibinLi/OBF.

Most studies of executive function (EF) in Autism Spectrum Disorder (ASD) focus on cognitive information processing, emphasizing less the social interaction deficits core to ASD. We designed a mobile game that uses social and nonsocial stimuli to assess children's EF skills. The game comprised three components involving different EF skills: cognitive flexibility (shifting/inference), inhibitory control, and short-term memory. By recruiting 65 children with and without ASD to play the mobile game, we investigated the potential of such platforms for capturing important phenotypic characteristics of individuals with autism. Results highlighted between-diagnostic-group differences in playing patterns with children with ASD showing broad patterns of EF deficits, but with relative strengths in nonsocial short-term memory, and preserved response to emotional inhibition cues. We showed the system could predict IQ, an important target for clinical treatment, towards the goal of developing platforms to act as long-term, efficient, and effective behavioral biomarkers for ASD.

The ability to learn sequential behaviors is a fundamental property of our brains. Yet a long stream of studies including recent experiments investigating motor sequence learning in adult human subjects have produced a number of puzzling and seemingly contradictory results. In particular, when subjects have to learn multiple action sequences, learning is sometimes impaired by proactive and retroactive interference effects. In other situations, however, learning is accelerated as reflected in facilitation and transfer effects. At present it is unclear what the underlying neural mechanism are that give rise to these diverse findings. Here we show that a recently developed recurrent neural network model readily reproduces this diverse set of findings. The self-organizing recurrent neural network (SORN) model is a network of recurrently connected threshold units that combines a simplified form of spike-timing dependent plasticity (STDP) with homeostatic plasticity mechanisms ensuring network stability, namely intrinsic plasticity (IP) and synaptic normalization (SN). When trained on sequence learning tasks modeled after recent experiments we find that it reproduces the full range of interference, facilitation, and transfer effects. We show how these effects are rooted in the network's changing internal representation of the different sequences across learning and how they depend on an interaction of training schedule and task similarity. Furthermore, since learning in the model is based on fundamental neuronal plasticity mechanisms, the model reveals how these plasticity mechanisms are ultimately responsible for the network's sequence learning abilities. In particular, we find that all three plasticity mechanisms are essential for the network to learn effective internal models of the different training sequences. This ability to form effective internal models is also the basis for the observed interference and facilitation effects. This suggests that STDP, IP, and SN may be the driving forces behind our ability to learn complex action sequences.

This paper modifies the DBSCAN algorithm to identify fixations and saccades. This method combines advantages from dispersion-based algorithms, such as resilience to noise and intuitive fixational structure, and from velocity-based algorithms, such as the ability to deal appropriately with smooth pursuit (SP) movements.

Researchers use fixation identification algorithms to parse eye movement trajectories into a series of fixations and saccades, simplifying analyses and providing measures which may relate to cognition. The Distance Dispersion (I-DD) a widely-used elementary fixation identification algorithm. Yet the "optimality" properties of its most popular greedy implementation have not been described. This paper: (1) asks how "optimal" should be defined, and advances maximizing total fixation time and minimizing number of clusters as a definition; (2) asks whether the greedy implementation of I-DD is optimal, and shows that it is when no fixations are rejected for being too short; and (3) we show that when fixation time rejection criterion are enabled, the greedy algorithm is not optimal. We propose an O(n2) algorithm which is.

Far infrared thermography, which can be used to detect thermal radiation emitted by humans, has been used to detect physical disease, physiological changes relating to emotion, and polygraph testing, but has not been used for eye tracking. However, because the surface temperature of the cornea is colder than the limbus, it is theoretically possible to track corneal movements through thermal imaging. To explore the feasibility of thermal eye tracking, we invited 10 adults and tracked their corneal movements with passive thermal imaging at 60 Hz. We combined shape models of eyes with intensity threshold to segment the cornea from other parts of the eye in thermal images. We used an animation sequence as a calibration target for 5 point calibration/validation 5 times. Our results were compared to simultaneously collected data using an SR EyeLink eye tracker at 500 Hz, demonstrating the feasibility of eye tracking with thermal images. Blinking and breathing frequencies, which reflect the psychophysical status of the participants, were also robustly detected during thermal eye tracking.

In their expanding role as tutors, home and healthcare assistants, robots must effectively interact with individuals of varying ability and temperament. Indeed, deploying robots in long-term social engagements will almost certainly require robots to reliably detect and adapt to changes in the demeanor of social partners to promote trust and more productive collaboration. However, the recognition of emotional state typically relies on the interpretation of very subtle cues, often varying from one person to the next. In addition, while facial expressions, body posture and features of speech have been used to detect affective changes, the robustness of these measures is often hindered by cultural and age differences. Recently, infrared thermography has shown promise in detecting guilt, fear and stress, indicating that it may be a viable sensing modality for improved human-robot interaction. In this study, we evaluated the efficacy of using a far infrared (FIR) camera for detecting robot-elicited affective response compared to video-elicited affective response by tracking thermal changes in five areas of the face. Further, we analyzed localized changes in the face to assess whether thermal and electrodermal responses to emotions elicited by traditional video techniques and by robots are similar. Finally, we performed principal component analysis to reduce the dimensionality of data and evaluated the performance using machine learning techniques for classifying thermal data by emotion state, resulting in a thermal classifier with a performance accuracy of 77.5%.

This research employs an expressive robot to elicit affective response in young children and explore correlations between autonomously-detected play, affective response and developmental ability. In this study, we introduce a new, affective interface that combines sound, color, movement and context to simulate the expression of emotions. Our approach exploits social contingencies to emphasize the importance of situational cues in the proper interpretation of affective state. We studied a group of young children at various ages and stages of cognitive development, to: (1) evaluate the efficacy of using captured motion data to autonomously detect physical patterns of play while interacting with a robot, (2) examine relationships between physical play patterns and observed affective response and, (3) explore associations between developmental ability and play or affective response. This pilot study demonstrates that aggregate patterns of physical interaction with a robot are distinguishable through autonomous data collection. Further, statistical analyses demonstrates that developmental ability may be directly related to how a child interacts with and responds to an affective robot.

This study presents a proof-of-concept study to assess the prediction of emotional perceptive competency and implicit affective preferences from each other. Predictions are made using linear regression, principal component analysis with linear regression, and support vector machines. Results point to a strong, bidirectional relationship between preference for emotional stimuli and affective competency. This work informs future studies predicting emotion processing abilities in humans and highlights the importance of tailoring interfaces to meet the emotional abilities of the user.

Emotional intelligence has been positively associated with social competence. In addition, attentional responses to emotional information have been associated with psychological characteristics related to mental health. In this study, we used the Microsoft Kinect platform as a tool to examine relationships between responses to emotional information in a gameplay environment and psychological factors. 45 typically developing individuals participated in the study, which involved 1) the Kinect game, requiring participants to engage in unprompted volitional whole-body responses to emotional stimuli, and 2) psychosocial assessments such as the Broader Autism Phenotype Questionnaire (BAPQ) and the Brief Symptom Inventory (BSI). Principal component analysis revealed patterns of gameplay that were associated with psychological characteristics of individuals. Preference for emotional content in general was associated with fewer social difficulties. The present work offers preliminary support for utilizing Kinect video games to understand emotion orienting and social capabilities, and shows that implicit patterns of preference identified during gameplay may relate to psychological and psychiatric phenomena. While the issues are complex and further research is needed, this work may inform the development of novel approaches to diagnostic and therapeutic tools.

Atypical looking behaviors in neuropsychiatric conditions such as autism spectrum disorders (ASD) are not only a reflection of inherently abnormal neuropsychological processes, but also suggest that future access to observational learning opportunities may be limited. The work presented in this paper uses interactive eye tracking as a first step towards the development of automated tools that can help toddlers and young children with atypical visual attention learn to attend to social information in a more typical fashion. In our study, we designed an automated visual strategy training system that would redirect a viewers' attention to locations highly salient to the normative control group when the viewer drifted from those locations for a significant period of time. We evaluated our experimental technique on typically-developing adults, obtaining results that suggest that looking patterns can be altered to be more similar to those evidenced by a normative group of young children. Furthermore, these alterations appear be retained in post-training sessions when considering new presentations of videos participants had been trained upon, and, on more sensitive outcome measures based on integrated scanpath probabilities (heatmaps), seemed to generalize presentations not trained upon as well. The development of these techniques may provide a new model for modifying attentional biases not only in toddlers with ASD, but also in children affected by other neuropsychiatric conditions, and may thus lead to new therapeutic interventions as well as more efficacious methods for identifying the patterns associated with abnormal, attention-driven experience.

Many different eye-tracking calibration techniques have been developed [e.g. see Talmi and Liu 1999; Zhu and Ji 2007]. A community standard is a 9-point-sparse calibration that relies on sequential presentation of known scene targets. However, fixating different points has been described as tedious, dull and tiring for the eye [Bulling, Gellersen, Pfeuffer, Turner and Vidal 2013].

Head-mounted eye-tracking systems allow us to observe participants' gaze behaviors in largely unconstrained, real-world settings. We have developed novel, untethered, mobile, low-cost, lightweight, easily-assembled head-mounted eye-tracking devices, comprised entirely of off-the-shelf components, including untethered, point-of-view, sports cameras. In total, the parts we have used cost ~$153, and we suggest untested alternative components that reduce the cost of parts to ~$31. Our device can be easily assembled using hobbying skills and techniques. We have developed hardware, software, and methodological techniques to perform point-of-regard estimation, and to temporally align scene and eye videos in the face of variable frame rate, which plagues low-cost, lightweight, untethered cameras. We describe an innovative technique for synchronizing eye and scene videos using synchronized flashing lights. Our hardware, software, and calibration designs will be made publicly available, and we describe them in detail here, to facilitate replication of our system. We also describe novel smooth-pursuit-based calibration methodology, which affords rich sampling of calibration data while compensating for lack of information regarding the extent of visibility on participants' scene recordings. Validation experiments indicate accuracy within 0.752 degrees of visual angle on average.

Fixation identification algorithms facilitate data comprehension and provide analytical convenience in eye-tracking analysis. However, current fixation algorithms for eye-tracking analysis are heavily dependent on parameter choices, leading to instabilities in results and incompleteness in reporting. This work examines the nature of human scanning patterns during complex scene viewing. We show that standard implementations of the commonly used distance-dispersion algorithm for fixation identification are functionally equivalent to greedy spatiotemporal tiling. We show that modeling the number of fixations as a function of tiling size leads to a measure of fractal dimensionality through box counting. We apply this technique to examine scale-free gaze behaviors in toddlers and adults looking at images of faces and blocks, as well as large number of adults looking at movies or static images. The distributional aspects of the number of fixations may suggest a fractal structure to gaze patterns in free scanning and imply that the incompleteness of standard algorithms may be due to the scale-free behaviors of the underlying scanning distributions. We discuss the nature of this hypothesis, its limitations, and offer directions for future work.