Emotions play a central role in shaping learning within digital environments. Although their effects may depend on how students’ emotional experiences manifest into concrete behaviors, the links between these dimensions remain underexplored. This study investigates the most common behaviors during episodes of boredom, confusion, frustration, and engaged concentration in an educational game, as well as associations with situational interest, self-efficacy, prior knowledge, and learning gains, using interaction logs and sensor-free affect detectors. Results show that boredom is linked to off-task roaming, both consistently associated with lower motivation and learning. In contrast, behaviors during engaged concentration, frustration, and especially confusion vary widely, shaped by motivational traits and prior knowledge and offering diverse associations with learning. Concrete regulatory responses in these states—such as systematizing findings with in-game tools, skimming domain content to resolve doubts, or testing hypotheses—are positively associated with learning and motivation, reflecting students’ ability to regulate emotions and address cognitive challenges. However, less constructive responses, such as aimless wandering, were tied to lower knowledge and motivation, underscoring the need for additional support. These findings extend existing affective theory by underscoring the importance of considering the behavioral dimension when analyzing students’ emotions in digital learning environments.

The advancements in automatic language processing being ushered in by Large Language Models suggest enormous potential for better personalization during student learning. However, this potential can be best exploited if we know that LLMs are equally capable of interacting with students who speak or write in a range of different dialects. This case study uses systematically manipulated student essays, previously evaluated by human raters, to examine how ChatGPT responds to and addresses specific dialect differences. Results point to important concerns about the potential biases and limitations of both LLMs and humans when evaluating and providing feedback to students who use minoritized dialects. Addressing these concerns is critical for the field of learning analytics, as it seeks to ensure equity and asset-based approaches to learning analytics.

For more than a decade, a handful of theoretical models have shaped a substantial amount of the research related to students’ emotional experiences during learning. This research has been productive, but articulating the underlying implicit assumptions in existing theories and their implications in our empirical interpretations can help to better investigate the reciprocal relationships between learning and emotion, and subsequently, to develop better interventions. This paper expands upon the existing theoretical frameworks, increasing the types of questions we ask about affect dynamics. We do so within the context of Crystal Island, a virtual world that allows middle school students to investigate microbiology questions. Specifically, we use this data to examine and revise the assumptions that are implicit in these models and the methods we use to investigate them.

Student affect is widely recognized as a major influence on learning gains and engagement, which has led to the development of many automated affect detectors. However, in order to respond effectively to student affect, we must know how students interpret it. This study proposes a novel automated detector that models when students attribute their epistemic emotion to task difficulty. The goal is to use detectors like this one to better understand how to respond to students' affective states (in this case, boredom, confusion, frustration and nervousness). We then discuss the implications of this novel detector for real-time support in game-based learning environments.

Measuring algorithmic bias in machine learning has historically focused on statistical inequalities pertaining to specific groups. However, the most common metrics (i.e., those focused on individual- or group-conditioned error rates) are not currently well-suited to educational settings because they assume that each individual observation is independent from the others. This is not statistically appropriate when studying certain common educational outcomes, because such metrics cannot account for the relationship between students in classrooms or multiple observations per student across an academic year. In this paper, we present novel adaptations of algorithmic bias measurements for regression for both independent and nested data structures. Using hierarchical linear models, we rigorously measure algorithmic bias in a machine learning model of the relationship between student engagement in an intelligent tutoring system and year-end standardized test scores. We conclude that classroom-level influences had a small but significant effect on models. Examining significance with hierarchical linear models helps determine which inequalities in educational settings might be explained by small sample sizes rather than systematic differences.

Numerous studies aim to enhance learning in digital environments through emotionally-sensitive interventions. The D’Mello and Graesser (2012) model of affect dynamics hypothesizes that when a learner encounters confusion, the degree to which it is prolonged (and transitions into frustration) or resolved, significantly affects their learning outcomes in digital environments. However, studies yield inconclusive results regarding relations between confusion, frustration, and learning. More research is needed to explore how confusion and frustration manifest during learning and its relation to outcomes. We go beyond past work looking at the rate, duration, and transitions of confusion and frustration by treating these affective states as non-linear dynamical systems consisting of expressive and behavioral components. We examined the frequency and recurrence of facial expressions associated with basic emotions (as automatically labeled by AffDex, a standard tool for analyzing emotions with video data) during confused and frustrated states (as automatically labeled with BROMP-based detectors applied to students’ interaction data). We compare these co-occurring patterns to learning outcomes (pre-tests, post-tests, and learning gains) within a digital learning environment, Betty’s Brain. Results showed that the frequency and recurrence rate of basic emotions expressed during confusion and frustration are complex and remain incompletely understood. Specifically, we show that confusion and frustration have different relationships with learning outcomes, depending on which basic emotion expressions they co-occur with. Implications of this study open avenues for better understanding these emotions as complex and non-linear dynamical systems, in the long-term enabling personalized feedback and emotional support within digital learning environments that enhance learning outcomes.

Peer feedback can be a powerful tool as it presents learning opportunities for both the learner receiving feedback as well as the learner providing feedback. Despite its utility, it can be difficult to implement effectively, particularly for younger learners, who are often novices at providing feedback. It can be difficult for students to learn what constitutes “good” feedback – particularly in open-ended problem-solving contexts. To address this gap, we investigate both classical natural language processing techniques and large language models, specifically ChatGPT, as potential approaches to devise an automated detector of feedback quality (including both student progress towards goals and next steps needed). Our findings indicate that the classical detectors are highly accurate and, through feature analysis, we elucidate the pivotal elements influencing its decision process. We find that ChatGPT is less accurate than classical NLP but illustrate the potential of ChatGPT in evaluating feedback, by generating explanations for ratings, along with scores. We discuss how the detector can be used for automated feedback evaluation and to better scaffold peer feedback for younger learners.

Abstract Automated, data‐driven decision making is increasingly common in a variety of application domains. In educational software, for example, machine learning has been applied to tasks like selecting the next exercise for students to complete. Machine learning methods, however, are not always equally effective for all groups of students. Current approaches to designing fair algorithms tend to focus on statistical measures concerning a small subset of legally protected categories like race or gender. Focusing solely on legally protected categories, however, can limit our understanding of bias and unfairness by ignoring the complexities of identity. We propose an alternative approach to categorization, grounded in sociological techniques of measuring identity. By soliciting survey data and interviews from the population being studied, we can build context‐specific categories from the bottom up. The emergent categories can then be combined with extant algorithmic fairness strategies to discover which identity groups are not well‐served, and thus where algorithms should be improved or avoided altogether. We focus on educational applications but present arguments that this approach should be adopted more broadly for issues of algorithmic fairness across a variety of applications.

Affect dynamics, the study of how affect develops and manifests over time, has become a popular area of research in affective computing for learning. In this article, we first provide a detailed analysis of prior affect dynamics studies, elaborating both their findings and the contextual and methodological differences between these studies. We then address methodological concerns that have not been previously addressed in the literature, discussing how various edge cases should be treated. Next, we present mathematical evidence that several past studies applied the transition metric (L) incorrectly - leading to invalid conclusions of statistical significance - and provide a corrected method. Using this corrected analysis method, we reanalyze ten past affect datasets collected in diverse contexts and synthesize the results, determining that the findings do not match the most popular theoretical model of affect dynamics. Instead, our results highlight the need to focus on cultural factors in future affect dynamics research.

Students in computerized learning environments often direct their own learning processes, which requires metacognitive awareness of what should be learned next. We investigated a novel method of measuring verbalized metacognition by applying natural language processing (NLP) to transcripts of interviews conducted in a classroom with 99 middle school students who were using a computerized learning environment. We iteratively adapted the NLP method for the linguistic characteristics of these interviews, then applied it to study three research questions regarding the relationships between verbalized metacognition and measures of 1) learning, 2) confusion, and 3) metacognitive problem-solving strategies. Verbalized metacognition was not directly related to learning, but was related to confusion and metacognitive problem-solving strategies. Results also suggested that interviews themselves may improve learning by encouraging metacognition. We discuss implications for designing computerized environments that support self-regulated learning through metacognition.

Confusion has been shown to be prevalent during complex learning and has mixed effects on learning. Whether confusion facilitates or hampers learning may depend on whether it is resolved or not. Confusion resolution, behind which is the resolution of cognitive disequilibrium, requires learners to possess some skills, but it is unclear what these skills are. One possibility may be metacognitive strategies (MS), strategies for regulating cognition. This study examined the relationship between confusion and actions related to MS in Betty's Brain, a computer-based learning environment. The results revealed that MS behavior differed during and outside confusion. However, confusion resolution was not related to MS behavior, and MS did not moderate the effect of confusion on learning.

We investigated the affective states (both individual and shared emotions) of students using a collaborative and educational game for English called Ibigkas! Our goal was two-fold: (1) To determine the incidence and persistence of affective states exhibited by the students when working individually and in groups, and (2) to adapt the Baker Rodrigo Ocumpaugh Monitoring Protocol for collaborative learning situations. Our findings for this study are as follows: (1) in single-player mode, students exhibited greater engaged concentration, pride, and frustration and less excitement, delight, and confusion compared to the multiplayer mode; (2) that individual emotions can be distinct from group emotions; (3) that negative emotions like frustration and blame/guilt were only felt at the individual level and were not observed as shared by all the members of the group; (4) affective states tended to persist more within an individualized game setting compared to the collaborative game setting where there was a greater number of opportunities to experience a wider range of emotions, hence the low chance of persistence; (5) students within an individualized setting spent more time solving the game rounds, had fewer incorrect answers, even as they experienced more frustration, and finally, (6) students within a collaborative setting had fewer errors when they had a higher incidence of excitement and had more errors when they appeared to be concentrating due to the presence of the “gaming the system” behavior.

Education research has explored the role of students' affective states in learning, but some evidence suggests that existing models may not fully capture the meaning or frequency of how students transition between different states. In this study we examine the patterns of educationally-relevant affective states within the context of Betty's Brain, an open-ended, computer-based learning system used to teach complex scientific processes. We examine three types of affective transitions based on similarity with the theorized D'Mello and Graesser model, transition between two affective states, and the sustained instances of certain states. We correlate of the frequency of these patterns with learning outcomes and our findings suggest that boredom is a powerful indicator of students' knowledge, but not necessarily indicative of learning. We discuss our findings within the context of both research and theory on affect dynamics and the implications for pedagogical and system design.

Previous studies have demonstrated strong links between students' linguistic knowledge, their affective language patterns and their success in math. Other studies have shown that demographic and click-stream variables in online learning environments are important predictors of math success. This study builds on this research in two ways. First, it combines linguistics and click-stream variables along with demographic information to increase prediction rates for math success. Second, it examines how random variance, as found in repeated participant data, can explain math success beyond linguistic, demographic, and click-stream variables. The findings indicate that linguistic, demographic, and click-stream factors explained about 14% of the variance in math scores. These variables mixed with random factors explained about 44% of the variance.

The relationship between learners' cognitive and affective states has become a topic of increased interest, especially because it is an important component of self-regulated learning (SRL) processes. This paper studies sixth grade students' SRL processes as they work in Betty's Brain, an agent-based open-ended learning environment (OELE). In this environment, students learn science topics by building causal models. Our analyses combine observational data on student affect to log files of students' interactions within the OELE. Preliminary analyses show that two relatively infrequent affective states, boredom and delight, show especially marked differences among high and low performing students. Further analysis shows that many of these differences occur after receiving feedback from the virtual agents in the Betty's Brain environment. We discuss the implications of these differences and how they can be used to construct adaptive personalized scaffolds.

This paper examines the effect of different linguistic features (as identified through Natural Language Processing tools) on affective measures of student engagement using a discovery with models approach. We build on previous literature, using automated detectors that identify when a middle-school student using an online mathematics tutor is experiencing boredom, confusion, frustration, or engaged concentration, to identify which problems are most engaging (or not) at scale. We then apply previously validated NLP tools to determine the degree to which engagement findings may be related to the linguistic properties of word problems, contributing to a growing literature on the effects of language on mathematics learning.

Advances in the learning analytics community have created opportunities to deliver early warnings that alert teachers and instructors when a student is at risk of not meeting academic goals [6], [71]. Alert systems have also been developed for school district leaders [33] and for academic advisors in higher education [39], but other professionals in the K-12 system, namely guidance counselors, have not been widely served by these systems. In this study, we use college enrollment models created for the ASSISTments learning system [55] to develop reports that target the needs of these professionals, who often work directly with students, but usually not in classroom settings. These reports are designed to facilitate guidance counselors' efforts to help students to set long term academic and career goals. As such, they provide the calculated likelihood that a student will attend college (the ASSISTments College Prediction Model or ACPM), alongside student engagement and learning measures. Using design principles from risk communication research and student feedback theories to inform a co-design process, we developed reports that can inform guidance counselor efforts to support student achievement.

Affect detection is a key component in intelligent educational interfaces that respond to students’ affective states. We use computer vision and machine-learning techniques to detect students’ affect from facial expressions (primary channel) and gross body movements (secondary channel) during interactions with an educational physics game. We collected data in the real-world environment of a school computer lab with up to 30 students simultaneously playing the game while moving around, gesturing, and talking to each other. The results were cross-validated at the student level to ensure generalization to new students. Classification accuracies, quantified as area under the receiver operating characteristic curve (AUC), were above chance (AUC of 0.5) for all the affective states observed, namely, boredom (AUC = .610), confusion (AUC = .649), delight (AUC = .867), engagement (AUC = .679), frustration (AUC = .631), and for off-task behavior (AUC = .816). Furthermore, the detectors showed temporal generalizability in that there was less than a 2% decrease in accuracy when tested on data collected from different times of the day and from different days. There was also some evidence of generalizability across ethnicity (as perceived by human coders) and gender, although with a higher degree of variability attributable to differences in affect base rates across subpopulations. In summary, our results demonstrate the feasibility of generalizable video-based detectors of naturalistic affect in a real-world setting, suggesting that the time is ripe for affect-sensitive interventions in educational games and other intelligent interfaces.

Affect detection is a key component in developing intelligent educational interfaces that are capable of responding to the affective needs of students. In this paper, computer vision and machine learning techniques were used to detect students' affect as they used an educational game designed to teach fundamental principles of Newtonian physics. Data were collected in the real-world environment of a school computer lab, which provides unique challenges for detection of affect from facial expressions (primary channel) and gross body movements (secondary channel) - up to thirty students at a time participated in the class, moving around, gesturing, and talking to each other. Results were cross validated at the student level to ensure generalization to new students. Classification was successful at levels above chance for off-task behavior (area under receiver operating characteristic curve or (AUC = .816) and each affective state including boredom (AUC =.610), confusion (.649), delight (.867), engagement (.679), and frustration (.631) as well as a five-way overall classification of affect (.655), despite the noisy nature of the data. Implications and prospects for affect-sensitive interfaces for educational software in classroom environments are discussed.

Choosing a college major is a major life decision. Interests stemming from students' ability and self-efficacy contribute to eventual college major choice. In this paper, we consider the role played by student learning, affect and engagement during middle school, using data from an educational software system used as part of regular schooling. We use predictive analytics to leverage automated assessments of student learning and engagement, investigating which of these factors are related to a chosen college major. For example, we already know that students who game the system in middle school mathematics are less likely to major in science or technology, but what majors are they more likely to select? Using data from 356 college students who used the ASSISTments system during their middle school years, we find significant differences in student knowledge, performance, and off-task and gaming behaviors between students who eventually choose different college majors.