Automated detectors are routinely used in learning analytics for high-stakes, high-risk interventions. Such interventions depend on detectors with a low rate of false positives (i.e., predicting the construct is present when it is not present) in order to avoid giving an intervention where it is not needed, especially when such interventions can be costly or even harmful. This in turn suggests that such a detector needs to have high precision at the cut-off used by the detector for decision-making. However, high precision is difficult to achieve for the common case where the base rate of the target construct is low. In this paper, we demonstrate the difficulty of achieving high precision for low base rates, and demonstrate how other metrics (such as F1, Kappa, Specificity, and AUC ROC) are insufficient for this specific use case and situation, despite their merits and advantages for other use cases and situations.

This study investigates the use of webcam-based eye tracking to model attention and comprehension in both neurotypical and neurodivergent learners. Leveraging the WebGazer, a previously used online data collection tool, we collected gaze and interaction data (N=354) during online reading tasks to explore task unrelated thought (TUT) and comprehension in an ecologically valid setting. Our findings challenge the "one size fits all" approach to learner modeling by demonstrating distinct differences in indicators of both constructs between neurotypical and neurodivergent learners. We compared general models trained on the entire population with tailored models specific to neurodivergent and neurotypical groups. Results indicate that diagnosis-specific models provide more accurate predictions (AUROC's .59-.70 vs. .57 for the general model), and through SHAPley analysis, we note that the strongest indicators of each construct vary as the training population is refined, highlighting the limitations of generalized approaches. This work supports the scalability of webcam-based cognitive modeling and underscores the potential for personalized learning analytics and modeling to better support diverse learning needs.

In this study, we focus on different strategies drivers use in terms of interleaving between driving and non-driving related tasks (NDRT) while taking back control from automated driving. We conducted two driving simulator experiments to examine how different cognitive demands of texting, priorities, and takeover time budgets affect drivers’ takeover strategies. We also evaluated how different takeover strategies affect takeover performance. We found that the choice of takeover strategy was influenced by the priority and takeover time budget but not by the cognitive demand of the NDRT. The takeover strategy did not have any effect on takeover quality or NDRT engagement but influenced takeover timing.

Peer feedback can be a powerful tool as it presents learning opportunities for both the learner receiving feedback as well as the learner providing feedback. Despite its utility, it can be difficult to implement effectively, particularly for younger learners, who are often novices at providing feedback. It can be difficult for students to learn what constitutes “good” feedback – particularly in open-ended problem-solving contexts. To address this gap, we investigate both classical natural language processing techniques and large language models, specifically ChatGPT, as potential approaches to devise an automated detector of feedback quality (including both student progress towards goals and next steps needed). Our findings indicate that the classical detectors are highly accurate and, through feature analysis, we elucidate the pivotal elements influencing its decision process. We find that ChatGPT is less accurate than classical NLP but illustrate the potential of ChatGPT in evaluating feedback, by generating explanations for ratings, along with scores. We discuss how the detector can be used for automated feedback evaluation and to better scaffold peer feedback for younger learners.

Previous learning analytics efforts have attempted to leverage the link between students’ gaze behaviors and learning experiences to build effective real-time interventions. Historically, however, these technologies have not been scalable due to the high cost of eye-tracking devices. Further, such efforts have been almost exclusively focused on neurotypical students, despite recent work that suggests a “one size fits many” approach can disadvantage neurodivergent students. Here we attempt to address these limitations by examining the validity and applicability of using scalable, webcam-based eye tracking as a basis for adaptively responding to neurodivergent students in an educational setting. Forty-three neurodivergent students read a text and answered questions about their in-situ thought patterns while a webcam-based eye tracker assessed their gaze locations. Results indicate that eye-tracking measures were sensitive to: 1) moments when students experienced difficulty disengaging from their own thoughts and 2) students’ familiarity with the text. Our findings highlight the fact that a free, open-source, webcam-based eye-tracker can be used to assess differences in reading patterns and online thought patterns. We discuss the implications and possible applications of these results, including the idea that webcam-based eye tracking may be a viable solution for designing real-time interventions for neurodivergent student populations.

Despite a recent shift towards online learning, recommendations for multimedia design principles in programming-based instruction remain unclear. Specifically, how can we teach people to code, a text-heavy medium, properly in online instruction? This question is especially important since the text-based format of screencasts may interact with psychological mechanisms known to affect cognitive processing and learning. We investigate this question, and find that previous results from other domains do not necessarily hold in the programming education. We also explore how design changes in textual aids affect learners' performance in programming-based multimedia learning. Our results suggest that the redundancy effect does not significantly hinder learning, which conflicts with previous findings, and that the spatial contiguity effect occurs even between textual components. This work contributes to an evidence-based understanding of how to design more effective multimedia learning environments for programming-based instruction.

We explore the transfer of control from an automated vehicle to the driver. Based on data from N=19 participants who participated in a driving simulator experiment, we find evidence that the transfer of control often does not take place in one step. In other words, when the automated system requests the transfer of control back to the driver, the driver often does not simply stop the non-driving task. Rather, the transfer unfolds as a process of interleaving the non-driving and driving tasks. We also find that the process is moderated by the length of time available for the transfer of control: interleaving is more likely when more time is available. Our interface designs for automated vehicles must take these results into account so as to allow drivers to safely take back control from automation.

We zone out roughly 20-40% of the time during reading – a rate that is concerning given the negative relationship between mind-wandering and comprehension. We tested if Eye-Mind Reader – an intelligent interface that targeted mind-wandering as it occurred – could mitigate its negative impact on reading comprehension. When an eye-gaze-based classifier indicated that a reader was mind-wandering, those in a MW-Intervention condition were asked to self-explain the concept they were reading about. If the self-explanation quality was deemed subpar by an automated scoring mechanism, readers were asked to re-read parts of the text in order to correct their comprehension deficits and improve their self-explanation. Each participant in the MW-Intervention condition was paired with a Yoked-Control counterpart who received the exact same interventions regardless of whether they were mind-wandering. Results indicate that re-reading improved self-explanation quality for the MW-Intervention group, but not the control group. The two conditions performed equally well on textbase (i.e. fact-based) and inference-level comprehension questions immediately after reading. However, after a week-long delay, the MW-Intervention condition significantly outperformed the yoked-control condition on both comprehension assessments (ds = .352 and .307). Our findings suggest that real-time interventions during critical periods of mind-wandering can promote long-term retention and comprehension.

Individual and organizational computer security rests on how people interpret and use the security information they are presented. One challenge is determining whether a given URL is safe or not. This paper explores the visual behaviors that users employ to gauge URL safety. We conducted a user study on 20 participants wherein participants classified URLs as safe or unsafe while wearing an eye tracker that recorded eye gaze (where they look) and pupil dilation (a proxy for cognitive effort). Among other things, our findings suggest that: users have a cap on the amount of cognitive resources they are willing to expend on vetting a URL; they tend to believe that the presence of www in the domain name indicates that the URL is safe; and they do not carefully parse the URL beyond what they perceive as the domain name.

This study examines the extent to which instructions to self-explain vs. other-explain a text lead readers to produce different forms of explanations. Natural language processing was used to examine the content and characteristics of the explanations produced as a function of instruction condition. Undergraduate students (n = 146) typed either self-explanations or other-explanations while reading a science text. The linguistic properties of these explanations were calculated using three automated text analysis tools. Machine learning classifiers in combination with the features were used to predict instruction condition (i.e., self- or other-explanation). The best machine learning model performed at rates above chance (kappa = .247; accuracy = 63%). Follow-up analyses indicated that students in the self-explanation condition generated explanations that were more cohesive and that contained words that were more related to social order (e.g., ethics). Overall, the results suggest that natural language processing techniques can be used to detect subtle differences in students' processing of complex texts.

Online lectures are an increasingly popular tool for learning, yet research on instructor visibility during an online lecture, and students' environmental settings, has not been well-explored. The current study addresses this gap in the literature by experimentally manipulating online display format and social learning settings to understand their influence on student learning and mind-wandering experiences. Results suggest that instructor visibility within an online lecture does not impact students' MW or retention performance. However, we found some evidence that students' social setting during viewing has an impact on MW (p = .05). Specifically, students who watched the lecture in a classroom with others reported significantly more MW than students who watched the lecture alone. Finally, social setting also moderated the negative relationship between MW and material retention. Our results demonstrate that learning experiences during online lectures can vary based on where, and with whom, the lectures are watched.

Current learning technologies have no direct way to assess students' mental effort: are they in deep thought, struggling to overcome an impasse, or are they zoned out? To address this challenge, we propose the use of EEG-based cognitive load detectors during learning. Despite its potential, EEG has not yet been utilized as a way to optimize instructional strategies. We take an initial step towards this goal by assessing how experimentally manipulated (easy and difficult) sections of an intelligent tutoring system (ITS) influenced EEG-based estimates of students' cognitive load. We found a main effect of task difficulty on EEG-based cognitive load estimates, which were also correlated with learning performance. Our results show that EEG can be a viable source of data to model learners' mental states across a 90-minute session.

Attention is critical to learning. Hence, advanced learning technologies should benefit from mechanisms to monitor and respond to learners' attentional states. We study the feasibility of integrating commercial off-the-shelf (COTS) eye trackers to monitor attention during interactions with a learning technology called GuruTutor. We tested our implementation on 135 students in a noisy computer-enabled high school classroom and were able to collect a median 95% valid eye gaze data in 85% of the sessions where gaze data was successfully recorded. Machine learning methods were employed to develop automated detectors of mind wandering (MW) -- a phenomenon involving a shift in attention from task-related to task-unrelated thoughts that is negatively correlated with performance. Our student-independent, gaze-based models could detect MW with an accuracy (F1 of MW = 0.59) significantly greater than chance (F1 of MW = 0.24). Predicted rates of mind wandering were negatively related to posttest performance, providing evidence for the predictive validity of the detector. We discuss next steps towards developing gaze-based, attention-aware, learning technologies that can be deployed in noisy, real-world environments.

Writing training systems have been developed to provide students with instruction and deliberate practice on their writing. Although generally successful in providing accurate scores, a common criticism of these systems is their lack of personalization and adaptive instruction. In particular, these systems tend to place the strongest emphasis on delivering accurate scores, and therefore, tend to overlook additional indices that may contribute to students' success, such as their affective states during writing practice. This study takes an initial step toward addressing this gap by building a predictive model of students' affect using information that can potentially be collected by computer systems. We used individual difference measures, text indices, and keystroke analyses to predict engagement and boredom in 132 writing sessions. The results suggest that these three categories of indices were successful in modeling students' affective states during writing. Taken together, indices related to students' academic abilities, text properties, and keystroke logs were able classify high and low engagement and boredom in writing sessions with accuracies between 76.5% and 77.3%. These results suggest that information readily available in writing training systems can inform affect detectors and ultimately improve student models within intelligent tutoring systems.