Observation of classroom interactions can provide concrete feedback to teachers, but current methods rely on manual annotation, which is resource-intensive and hard to scale. This work explores AI-driven analysis of classroom recordings, focusing on multimodal instructional activity and discourse recognition as a foundation for actionable feedback. Using a densely annotated dataset of 164 hours of video and 68 lesson transcripts, we design parallel, modality-specific pipelines. For video, we evaluate zero-shot multimodal LLMs, fine-tuned vision–language models, and self-supervised video transformers on 24 activity labels. For transcripts, we fine-tune a transformer-based classifier with contextualized inputs and compare it against prompting-based LLMs on 19 discourse labels. To handle class imbalance and multi-label complexity, we apply per-label thresholding, context windows, and imbalance-aware loss functions. The results show that fine-tuned models consistently outperform prompting-based approaches, achieving macro-F1 scores of 0.577 for video and 0.460 for transcripts. These results demonstrate the feasibility of automated classroom analysis and establish a foundation for scalable teacher feedback systems.

Learners often experience aware and unaware mind wandering during educational tasks, both negatively impacting learning outcomes. Differentiating these types of task-unrelated thoughts is crucial, as they stem from different cognitive processes and warrant tailored support that addresses the specific nature of mind wandering. Automated detection of these episodes could help mitigate their adverse effects, for example, by developing adaptive, attention-aware learning environments. In this study (N = 87), we explored a novel multimodal approach, combining eye tracking, facial videos, and physiological wristbands (i.e., electrodermal activity and heart rate), to predict aware and unaware mind wandering during lecture video watching. In addition, to allow comparison to previous research, we also predicted an integrated mind-wandering category. Mind wandering was assessed using 15 two-stage thought probes to determine task-unrelated thoughts and the participants’ awareness of their mind wandering. Our findings indicate that a multimodal approach outperforms unimodal methods, utilizing the top 100 features from the fused data. Specifically, aware mind wandering was detected at 20% above chance (AUC-PR = 0.396), unaware mind wandering at 14% above chance (AUC-PR = 0.267), and the combined category at 40% above chance (AUC-PR = 0.637). Eye tracking and video features proved more predictive than physiological measures when used as standalone modalities. SHAP analysis, employed to explain the results, highlighted the significance of integrating features from all three modalities for effective detection, particularly emphasizing the role of video-based facial expressions in identifying unaware mind wandering. Going beyond the current state of the art, this study demonstrates the potential of leveraging multimodal data to enhance the precision of aware and unaware mind-wandering detection and differentiation, setting a foundation for advancing educational technologies that respond dynamically to learners’ cognitive states.

Successful learning depends on learners' ability to sustain attention, which is particularly challenging in online education due to limited teacher interaction. A potential indicator for attention is gaze synchrony, demonstrating predictive power for learning achievements in video-based learning in controlled experiments focusing on manipulating attention. This study (N=84) examines the relationship between gaze synchronization and self-reported attention of learners, using experience sampling, during realistic online video learning. Gaze synchrony was assessed through Kullback-Leibler Divergence of gaze density maps and MultiMatch algorithm scanpath comparisons. Results indicated significantly higher gaze synchronization in attentive participants for both measures and self-reported attention significantly predicted post-test scores. In contrast, synchrony measures did not correlate with learning outcomes. While supporting the hypothesis that attentive learners exhibit similar eye movements, the direct use of synchrony as an attention indicator poses challenges, requiring further research on the interplay of attention, gaze synchrony, and video content type.

The present study examined reading interactions and multiple document comprehension of 108 university students who read five partly conflicting documents on a health-related issue. Documents were presented on a large touch interface that either enabled a simultaneous or imposed a sequential presentation of the documents. None of the three multiple document comprehension measures (number of intertextual connections in essays, score in source-content mapping task, and number of source names recalled) was affected by document presentation. However, in the sequential condition, the number of revisits to documents was positively related to memory for source names and source-content integration, but not to intertextual integration. Conversely, in the simultaneous condition, participants who grouped documents during reading showed greater intertextual and source-content integration, but no greater memory for source names than those who had not grouped documents. To conclude, reading interactions played a more important role in readers’ multiple document comprehension than document presentation.

Assessment of cognitive load is a major step towards adaptive interfaces. However, non-invasive assessment is rather subjective as well as task specific and generalizes poorly, mainly due to methodological limitations. Additionally, it heavily relies on performance data like game scores or test results. In this study, we present an eye-tracking approach that circumvents these shortcomings and allows for effective generalizing across participants and tasks. First, we established classifiers for predicting cognitive load individually for a typical working memory task (n-back), which we then applied to an emergency simulation game by considering the similar ones and weighting their predictions. Standardization steps helped achieve high levels of cross-task and cross-participant classification accuracy between 63.78 and 67.25 percent for the distinction between easy and hard levels of the emergency simulation game. These very promising results could pave the way for novel adaptive computer-human interaction across domains and particularly for gaming and learning environments.

Student engagement is a key component of learning and teaching, resulting in a plethora of automated methods to measure it. Whereas most of the literature explores student engagement analysis using computer-based learning often in the lab, we focus on using classroom instruction in authentic learning environments. We collected audiovisual recordings of secondary school classes over a one and a half month period, acquired continuous engagement labeling per student (N=15) in repeated sessions, and explored computer vision methods to classify engagement from facial videos. We learned deep embeddings for attentional and affective features by training Attention-Net for head pose estimation and Affect-Net for facial expression recognition using previously-collected large-scale datasets. We used these representations to train engagement classifiers on our data, in individual and multiple channel settings, considering temporal dependencies. The best performing engagement classifiers achieved student-independent AUCs of .620 and .720 for grades 8 and 12, respectively, with attention-based features outperforming affective features. Score-level fusion either improved the engagement classifiers or was on par with the best performing modality. We also investigated the effect of personalization and found that only 60 seconds of person-specific data, selected by margin uncertainty of the base classifier, yielded an average AUC improvement of .084.

Functional near-infrared spectroscopy (fNIRS) provides reliable results for determining cognitive load based on averaged cortical blood flow during multiple repetitions of short cognitive tasks. At the same time, it remains unclear how to use this technique for assessing cognitive load during prolonged single-trial activity. In this study, we used a computer-based emergency simulation game for inducing different levels of cognitive load. We propose a novel approach to measure cognitive load using specific time slots, determined based on simulation log-data interpreted in light of Barrouillet's time-based resource-sharing model. To validate this approach, we compared cortical activity in dorsolateral prefrontal cortex (DLPFC) and left inferior frontal gyrus (IFG) regions measured at four specific time slots during a simulation. We found significant associations between cognitive load and neuronal activity within the DLPFC depending on the chosen time slot, whereas no such dependencies were found for the IFG. These results illustrate how knowledge of task structure could be used advantageously for the identification of cognitive load. Although requiring further investigation in terms of reliability and generalizability, the presented approach can be considered promising evidence that fNIRS might be suitable for more general reliable assessments of cognitive load during prolonged single-trial activities and for real-time adaptations in simulation-based learning environments.

Abstract We investigated whether a passive brain–computer interface that was trained to distinguish low and high mental workload in the electroencephalogram (EEG) can be used to identify (1) texts of different readability difficulties and (2) texts read at different presentation speeds. For twelve subjects we calibrated a subject-dependent, but task-independent predictive model classifying mental workload. We then recorded EEG data from each subject, while twelve texts in blocks of three were presented to them word by word. Half of the texts were easy, and the other half were difficult texts according to classic reading formulas. From each text category three texts were read at a self-adjusted comfortable presentation speed and the other three at an increased speed. For each subject we applied the predictive model to EEG data of each word of the twelve texts. We found that the resulting predictive values for mental workload were higher for difficult texts than for easy texts. Predictive values from texts presented at an increased speed were also higher than for those presented at a normal self-adjusted speed. The results suggest that the task-independent predictive model can be used on single-subject level to build a highly predictive user model of the reader over time. Such a model could be employed in a system which continuously monitors brain activity related to mental workload and adapts to specific reader’s abilities and characteristics by adjusting the difficulty of text materials and the way it is presented to the reader in real time. A neuroadaptive system like this could foster efficient reading and text-based learning by keeping readers’ mental workload levels at an individually optimal level.

This paper addresses the problem of understanding joint attention in third-person social scene videos. Joint attention is the shared gaze behaviour of two or more individuals on an object or an area of interest and has a wide range of applications such as human-computer interaction, educational assessment, treatment of patients with attention disorders, and many more. Our method, Attention Flow, learns joint attention in an end-to-end fashion by using saliency-augmented attention maps and two novel convolutional attention mechanisms that determine to select relevant features and improve joint attention localization. We compare the effect of saliency maps and attention mechanisms and report quantitative and qualitative results on the detection and localization of joint attention in the VideoCoAtt dataset, which contains complex social scenes.

State of the art brain-computer interfaces (BCIs) largely focus on detecting single, specific, often experimentally induced or manipulated aspects of the user state. In a less controlled, more naturalistic environment, a larger variety of mental processes may be active and possibly interacting. When moving BCI applications from the lab to real-life applications, these additional unaccounted mental processes could interfere with user state decoding, thus decreasing system efficacy and decreasing real-world applicability. Here, we assess the impact of affective valence on classification of working memory load, by re-analyzing a dataset that used an affective N-back task with picture stimuli. Our analyses showed that classification of working memory load under affective valence can lead to good classification accuracies (> 70 percent), which can be further improved via data integration over time. However, positive as well as negative affective valence resulted in decreased classification accuracies, when compared to the neutral affective context. Furthermore, classifiers failed to generalize across affective contexts, highlighting the need for user state models that can account for different contexts or new, context-independent, EEG features.

Experimental research investigating the processes or influencing factors of diagnostic reasoning or diagnostic success is predominantly conducted with case descriptions spanning no more than one A4 page. In this paper, we argue for a more authentic task setting in the form of multiple documents case descriptions, and make suggestions how to design them to suit different research questions. We further review methods used in previous studies, such as think-aloud protocols and written justifications of diagnoses, and discuss how they can be used in order to assess the cognitive processes underlying diagnostic reasoning in more detail. Additionally, based on findings from the field of multiple documents comprehension, we outline how participants' gaze behavior on and their interaction with the documents might also be used to assess processes of information comparison and corroboration during reading as part of participants' diagnostic reasoning process.

The reliable estimation of cognitive load is an integral step towards real-time adaptivity of learning or gaming environments. We introduce a novel and robust machine learning method for cognitive load assessment based on behavioral and physiological measures in a combined within- and cross-participant approach. 47 participants completed different scenarios of a commercially available emergency personnel simulation game realizing several levels of difficulty based on cognitive load. Using interaction metrics, pupil dilation, eye-fixation behavior, and heart rate data, we trained individual, participant-specific forests of extremely randomized trees differentiating between low and high cognitive load. We achieved an average classification accuracy of 72%. We then apply these participant-specific classifiers in a novel way, using similarity between participants, normalization, and relative importance of individual features to successfully achieve the same level of classification accuracy in cross-participant classification. These results indicate that a combination of behavioral and physiological indicators allows for reliable prediction of cognitive load in an emergency simulation game, opening up new avenues for adaptivity and interaction.

Real-time evaluation of a person's cognitive load can be desirable in many situations. It can be employed to automatically assess or adjust the difficulty of a task, as a safety measure, or in psychological research. Eye-related measures, such as the pupil diameter or blink rate, provide a non-intrusive way to assess the cognitive load of a subject and have therefore been used in a variety of applications. Usually, workload classifiers trained on these measures are highly subject-dependent and transfer poorly to other subjects. We present a novel method to generalize from a set of trained classifiers to new and unknown subjects. We use normalized features and a similarity function to match a new subject with similar subjects, for which classifiers have been previously trained. These classifiers are then used in a weighted voting system to detect workload for an unknown subject. For real-time workload classification, our methods performs at 70.4% accuracy. Higher accuracy of 76.8% can be achieved in an offline classification setting.

This paper presents a novel approach for automatic prediction of risk of ADHD in schoolchildren based on touch interaction data. We performed a study with 129 fourth-grade students solving math problems on a multiple-choice interface to obtain a large dataset of touch trajectories. Using Support Vector Machines, we analyzed the predictive power of such data for ADHD scales. For regression of overall ADHD scores, we achieve a mean squared error of 0.0962 on a four-point scale (R² = 0.5667). Classification accuracy for increased ADHD risk (upper vs. lower third of collected scores) is 91.1%.

Virtual reality technologies, along with motion based input devices allow for the design of innovative interfaces between learners and digital knowledge resources. These interfaces might facilitate knowledge work in educational and scientific contexts. Compared to 2D interfaces, immersive 3D environments provide greater flexibility regarding the interface design, however, so far no general, theory-driven and validated design principles are available. Seeing that complex learning environments can foster the development of various cognitive abilities, like multiperspective reasoning skills (MPRS), such design principles are highly desirable. Using multiperspective hypermedia environments (MHEs) as a testbed, the presented project aims to identify and evaluate design principles, derived from cognitive science. We will create and study interactive, immersive 3D-interface to MHEs using virtual reality technology. To evaluate the developed system, we will contrast the acquisition of MPRS in 2D and 3D learning environments. We expect that the developed design principles will be directly applicable for enhancing the accessibility of other knowledge environments.

This demo presents the multidisciplinary development of the enrichment program “Listening to Math: Kids compose with LEGO”, that aims at fostering the math and music skills of gifted primary school students. In this program LEGO bricks are used as tangible representation of music notes. The so called LEGO-Table, a tabletop computer, translates patterns made out of bricks into piano melodies. We will present the finished program, address our experiences made during the project, and we will also give some insights on the implementation and evaluation of the program. Functions of the LEGO-Table will be demonstrated interactively on a tablet.

Although a great number of today’s learning applications run on devices with an interactive screen, the high-resolution interaction data which these devices provide have not been used for workload-adaptive systems yet. This paper aims at exploring the potential of using touch sensor data to predict user states in learning scenarios. For this purpose, we collected touch interaction patterns of children solving math tasks on a multi-touch device. 30 fourth-grade students from a primary school participated in the study. Based on these data, we investigate how machine learning methods can be applied to predict cognitive workload associated with tasks of varying difficulty. Our results show that interaction patterns from a touchscreen can be used to significantly improve automatic prediction of high levels of cognitive workload (average classification accuracy of 90.67% between the easiest and most difficult tasks). Analyzing an extensive set of features, we discuss which characteristics are most likely to be of high value for future implementations. We furthermore elaborate on design choices made for the used multiple choice interface and discuss critical factors that should be considered for future touch-based learning interfaces.

We present a multi-touch tabletop application that utilizes LEGO bricks as physical representations for musical notes to create a novel digital learning environment for musical composition principles. Our application makes use of the physical qualities that graspable objects provide and combines them with dynamic visualizations on a multi-touch screen. While the system can be used to collaboratively create melodies and harmonies, our concept specifically aims at introducing and teaching composition principles. In this paper, we illustrate the concept and the iterative process that lead to the final application design. Furthermore, we discuss findings from four composing workshops which were conducted with school children with varying levels of musical expertise.

Previous research indicates that web users rely to a great extent on the ranking provided by the search engine and predominantly access the first few web pages presented. In case that the information sources presented in the top of the search engine results page (SERP) are of rather low trustworthiness, this might lead to a biased or incomplete view of the topic—especially when dealing with controversial issues. Study 1, thus, systematically investigated whether participants who were asked to search for an unfamiliar and controversial medical issue accessed fewer trustworthy information sources and consequently included less information from trustworthy pages in their argumentation when the search results were ranked from least to most trustworthy on a Google-like SERP than when they were ranked from most to least trustworthy. Results from Study 1 confirmed these assumptions. Furthermore, Study 2 showed that when the same materials were presented in a grid interface, the impact of the position of the search results on their selection was substantially reduced. Irrespective of whether the most trustworthy search results were presented in the top or the bottom row of the grid interface, users predominantly selected the most trustworthy search results from the SERP and included the same amount of information from trustworthy pages in their argumentation.

According to Cognitive Load Theory the type and amount of workload (WL) during learning is crucial for successful learning and should be held within an optimal range of learners' memory capacity. Therefore, we aim at developing electroencephalogram (EEG) based learning environments adapting to learners individual WL online. To achieve this goal efficient classification methods are necessary. Support Vector Machines (SVMs) can accurately classify WL using within-task classification, but within-task classification is not feasible in complex learning environments. Therefore, the present study examined cross-task classification accuracies for SVMs trained on EEG-signals, recorded while participants (N= 21) had to solve three working memory tasks. While within-task classification accuracies were high for WM tasks (average: 95% - 97 %), cross-task classification performances were not significant over chance level. Since cross-task classification is a necessary step towards developing generalized classifiers, we will discuss the benefits and drawbacks as well as possible enhancements in the course of this paper to use it as an effective approach for learning environments.

In the above article, published online in Behaviour & Information Technology, http://dx.doi.org/10.1080/0144929X.2011.599040, Table 1 appeared incorrectly. The table should have appeared as follows...

The present study examined how both the interface of search engines and Internet-specific epistemic beliefs influence novices' source evaluations during Web search on a medical topic. A standard Google-like list interface was compared to a tabular interface that presented search results grouped according to objective, subjective, or commercial information in order to provide users with affordances for source evaluations. Results revealed that university students using the tabular interface paid less visual attention to commercial search results and selected objective search results more often and commercial ones less often than students using the list interface. Furthermore, the epistemic belief that the Web contains (among other types of information) correct knowledge was related to an increased selection of objective search results and to longer fixations on non-selected search results. Moreover, epistemic beliefs moderated the effects of the search interface, such that students with strong beliefs that the Web contains correct knowledge showed a more focused information selection and better search outcomes in terms of their argumentative summaries when using the tabular interface than when using the list interface. In contrast, these effects were not found for students with doubts about the Web containing correct knowledge.

This study examined to what extent users spontaneously evaluate the trustworthiness of Web search results presented by a search engine. For this purpose, a methodological paradigm was used in which the trustworthiness order of search results was experimentally manipulated by presenting search results on a search engine results page (SERP) either in a descending or ascending trustworthiness order. Moreover, a standard list format was compared to a grid format in order to examine the impact of the search results interface on Web users' evaluation processes. In an experiment addressing a controversial medical topic, 80 participants were assigned to one of four conditions with trustworthiness order (descending vs. ascending) and search results interface (list vs. grid) varied as between-subjects factors. In order to investigate participants' evaluation processes their eye movements and mouse clicks were captured during Web search. Results revealed that a list interface caused more homogenous and more linear viewing sequences on SERPs than a grid interface. Furthermore, when using a list interface most attention was given to the search results on top of the list. In contrast, with a grid interface nearly all search results on a SERP were attended to equivalently long. Consequently, in the ascending trustworthiness order participants using a list interface attended significantly longer to the least trustworthy search results and selected the most trustworthy search results significantly less often than participants using a grid interface. Thus, the presentation of Web search results by means of a grid interface seems to support users in their selection of trustworthy information sources.