Observation of classroom interactions can provide concrete feedback to teachers, but current methods rely on manual annotation, which is resource-intensive and hard to scale. This work explores AI-driven analysis of classroom recordings, focusing on multimodal instructional activity and discourse recognition as a foundation for actionable feedback. Using a densely annotated dataset of 164 hours of video and 68 lesson transcripts, we design parallel, modality-specific pipelines. For video, we evaluate zero-shot multimodal LLMs, fine-tuned vision–language models, and self-supervised video transformers on 24 activity labels. For transcripts, we fine-tune a transformer-based classifier with contextualized inputs and compare it against prompting-based LLMs on 19 discourse labels. To handle class imbalance and multi-label complexity, we apply per-label thresholding, context windows, and imbalance-aware loss functions. The results show that fine-tuned models consistently outperform prompting-based approaches, achieving macro-F1 scores of 0.577 for video and 0.460 for transcripts. These results demonstrate the feasibility of automated classroom analysis and establish a foundation for scalable teacher feedback systems.

It is commonly assumed that digital learning environments such as intelligent tutoring systems facilitate learning and positively impact achievement. This study explores how different groups of students exhibit distinct relationships between learning behaviors and academic achievement in an intelligent tutoring system for English as a foreign language. We examined whether these differences are linked to students’ prior knowledge, personality traits, and motivation. We collected behavioral trace data from 507 German seventh-grade students during the 2021/22 school year and applied machine learning models to predict English performance based on learning behaviors (best-performing model’s 2 = .41). To understand the impact of specific behaviors, we applied the explainable AI method SHAP and identified three student clusters with distinct learning behavior patterns. Subsequent analyses revealed that these clusters also varied in prior knowledge and motivation: one with high prior knowledge and average motivation, another with low prior knowledge and average motivation, and a third with both low prior knowledge and low motivation. Our findings suggest that learning behaviors are linked differently to academic success across students and are closely tied to their prior knowledge and motivation. This hints towards the importance of personalizing learning systems to support individual learning needs better.

Learners often experience aware and unaware mind wandering during educational tasks, both negatively impacting learning outcomes. Differentiating these types of task-unrelated thoughts is crucial, as they stem from different cognitive processes and warrant tailored support that addresses the specific nature of mind wandering. Automated detection of these episodes could help mitigate their adverse effects, for example, by developing adaptive, attention-aware learning environments. In this study (N = 87), we explored a novel multimodal approach, combining eye tracking, facial videos, and physiological wristbands (i.e., electrodermal activity and heart rate), to predict aware and unaware mind wandering during lecture video watching. In addition, to allow comparison to previous research, we also predicted an integrated mind-wandering category. Mind wandering was assessed using 15 two-stage thought probes to determine task-unrelated thoughts and the participants’ awareness of their mind wandering. Our findings indicate that a multimodal approach outperforms unimodal methods, utilizing the top 100 features from the fused data. Specifically, aware mind wandering was detected at 20% above chance (AUC-PR = 0.396), unaware mind wandering at 14% above chance (AUC-PR = 0.267), and the combined category at 40% above chance (AUC-PR = 0.637). Eye tracking and video features proved more predictive than physiological measures when used as standalone modalities. SHAP analysis, employed to explain the results, highlighted the significance of integrating features from all three modalities for effective detection, particularly emphasizing the role of video-based facial expressions in identifying unaware mind wandering. Going beyond the current state of the art, this study demonstrates the potential of leveraging multimodal data to enhance the precision of aware and unaware mind-wandering detection and differentiation, setting a foundation for advancing educational technologies that respond dynamically to learners’ cognitive states.

Successful learning depends on learners' ability to sustain attention, which is particularly challenging in online education due to limited teacher interaction. A potential indicator for attention is gaze synchrony, demonstrating predictive power for learning achievements in video-based learning in controlled experiments focusing on manipulating attention. This study (N=84) examines the relationship between gaze synchronization and self-reported attention of learners, using experience sampling, during realistic online video learning. Gaze synchrony was assessed through Kullback-Leibler Divergence of gaze density maps and MultiMatch algorithm scanpath comparisons. Results indicated significantly higher gaze synchronization in attentive participants for both measures and self-reported attention significantly predicted post-test scores. In contrast, synchrony measures did not correlate with learning outcomes. While supporting the hypothesis that attentive learners exhibit similar eye movements, the direct use of synchrony as an attention indicator poses challenges, requiring further research on the interplay of attention, gaze synchrony, and video content type.

Student engagement is a key component of learning and teaching, resulting in a plethora of automated methods to measure it. Whereas most of the literature explores student engagement analysis using computer-based learning often in the lab, we focus on using classroom instruction in authentic learning environments. We collected audiovisual recordings of secondary school classes over a one and a half month period, acquired continuous engagement labeling per student (N=15) in repeated sessions, and explored computer vision methods to classify engagement from facial videos. We learned deep embeddings for attentional and affective features by training Attention-Net for head pose estimation and Affect-Net for facial expression recognition using previously-collected large-scale datasets. We used these representations to train engagement classifiers on our data, in individual and multiple channel settings, considering temporal dependencies. The best performing engagement classifiers achieved student-independent AUCs of .620 and .720 for grades 8 and 12, respectively, with attention-based features outperforming affective features. Score-level fusion either improved the engagement classifiers or was on par with the best performing modality. We also investigated the effect of personalization and found that only 60 seconds of person-specific data, selected by margin uncertainty of the base classifier, yielded an average AUC improvement of .084.

There is a constantly growing number of initiatives asserting the relevance of programming already in primary education and offering respective interventions with the goal to foster interest in and positive attitudes toward programming. To evaluate to what extent this goal is achieved, assessing students' attitudes toward programming reliably is indispensable. However, there still is a need for validated instruments for assessing this in elementary school students. This seems particularly relevant as self-concept and attitudes toward a school subject were repeatedly observed to be significant predictors of learning motivation and achievement. The newly developed Self-Concept and Attitude toward Programming Assessment (SCAPA) is based on existing instruments for assessing students' self-concept and attitude toward mathematics. SCAPA measures aspects of students' self-concept and attitudes toward programming on seven scales: i) self-reported previous programming experience and understanding, ii) self-concept, iii) intrinsic value belief, iv) attainment value belief, v) utility value belief, vi) cost belief, and vii) compliance and persistence. We administered SCAPA to 197 elementary school students between seven and ten years of age in the context of an evaluation of a computational thinking intervention. Data were analyzed for reliability (i.e., internal consistency on item and scale level) and construct validity (by means of confirmatory factor analysis). Results indicated good reliability for all scales except for the self-reported previous programming experience and understanding scale. Overall, these results reflect SCAPA's suitability for assessing different aspects of elementary school students' self-concept and attitudes toward programming.

This paper addresses the problem of understanding joint attention in third-person social scene videos. Joint attention is the shared gaze behaviour of two or more individuals on an object or an area of interest and has a wide range of applications such as human-computer interaction, educational assessment, treatment of patients with attention disorders, and many more. Our method, Attention Flow, learns joint attention in an end-to-end fashion by using saliency-augmented attention maps and two novel convolutional attention mechanisms that determine to select relevant features and improve joint attention localization. We compare the effect of saliency maps and attention mechanisms and report quantitative and qualitative results on the detection and localization of joint attention in the VideoCoAtt dataset, which contains complex social scenes.

Although a great number of today’s learning applications run on devices with an interactive screen, the high-resolution interaction data which these devices provide have not been used for workload-adaptive systems yet. This paper aims at exploring the potential of using touch sensor data to predict user states in learning scenarios. For this purpose, we collected touch interaction patterns of children solving math tasks on a multi-touch device. 30 fourth-grade students from a primary school participated in the study. Based on these data, we investigate how machine learning methods can be applied to predict cognitive workload associated with tasks of varying difficulty. Our results show that interaction patterns from a touchscreen can be used to significantly improve automatic prediction of high levels of cognitive workload (average classification accuracy of 90.67% between the easiest and most difficult tasks). Analyzing an extensive set of features, we discuss which characteristics are most likely to be of high value for future implementations. We furthermore elaborate on design choices made for the used multiple choice interface and discuss critical factors that should be considered for future touch-based learning interfaces.