The recent downward trend in reading proficiency for U.S. students underscores the urgent need for tools that can identify and support students at risk of falling behind. This study examined the extent to which diagnostic assessments of foundational reading skills can predict subsequent changes in performance on a state-level reading assessment, the Minnesota Comprehensive Assessments (MCA). Using ReadBasix diagnostic scores across six sub-skills, we trained a multi-class classification model to predict shifts in students’ MCA proficiency levels. Our results show that diagnostic measures of foundational skills, such as morphology and reading efficiency, can provide meaningful insight into students’ future reading trajectories. In addition, there appears to be a threshold for these key reading skills that needs to be reached before significant improvements in reading efficiency can occur. These findings highlight the importance of assessing and monitoring specific skill development, rather than relying solely on broad outcome measures, to guide instructional decisions. By linking sub-skill diagnostics to state assessment outcomes, this work suggests a path forward for more targeted, data-driven interventions aimed at reversing national trends in declining reading proficiency.

Socio-cultural context influences how learners engage with, experience, and make sense of learning. As such, Learning Analytics systems must intentionally account for this context in their design and interpretation. While many researchers in the field are already addressing these issues, sometimes implicitly or under different frameworks, we argue for a more unified focus on socio-cultural context as a guiding construct. Drawing on scholarship from education, sociology, anthropology, and psychology, we define socio-cultural context in terms of power, norms, and identity. Through two case studies of our own work on collaboration and engagement analytics, we illustrate how overlooking socio-cultural context can lead to incomplete systems that do not account for the variety of ways students might show up in learning spaces. We build on existing work and propose socio-cultural context as an organizing lens for future research. Doing so enables interdisciplinary collaboration and design that ultimately ensures analytics better serves learners.

Early exposure to ethical reasoning and diversity, equity, and inclusion (DEI) concepts is critical for preparing socially responsible computer scientists. To support this goal, we designed and implemented a required non-programming first-year undergraduate computer science course that is taken concurrently with an introductory programming course. This discussion-based course integrates DEI and ethics as foundational themes. The new course adopts a breadth-first structure delivered over a 10-week quarter, with 5 of the 18 sessions focused on DEI and ethics. Over three consecutive academic years, we collected pre- and post-course survey data from 451 students enrolled in the new course. Results show an %statistically significant increase in student's recognition of the importance of DEI and ethics and a substantial increase in understanding how these dimensions intersect with technical practice. In this experience report, we describe the course design, instructional methods, and survey instruments; present key findings; and reflect on lessons learned. This work contributes a model for embedding DEI and ethics into the early undergraduate computing curriculum.

There has been increasing interest in data enclaves in recent years, both in education and other fields. Data enclaves make it possible to conduct analysis on large-scale and higher-risk data sets, while protecting the privacy of the individuals whose data is included in the data sets, thus mitigating risks around data disclosure. In this article, we provide a post-mortem on the MORF (MOoc Replication Framework) 2.1 infrastructure, a data enclave expected to sunset and be replaced in the upcoming years, reviewing the core factors that reduced its usefulness for the community. We discuss challenges to researchers in terms of usability, including challenges involving learning to use core technologies, working with data that cannot be directly viewed, debugging, and working with restricted outputs. Our post-mortem discusses possibilities for ways that future infrastructures could get past these challenges.

This study investigates the use of webcam-based eye tracking to model attention and comprehension in both neurotypical and neurodivergent learners. Leveraging the WebGazer, a previously used online data collection tool, we collected gaze and interaction data (N=354) during online reading tasks to explore task unrelated thought (TUT) and comprehension in an ecologically valid setting. Our findings challenge the "one size fits all" approach to learner modeling by demonstrating distinct differences in indicators of both constructs between neurotypical and neurodivergent learners. We compared general models trained on the entire population with tailored models specific to neurodivergent and neurotypical groups. Results indicate that diagnosis-specific models provide more accurate predictions (AUROC's .59-.70 vs. .57 for the general model), and through SHAPley analysis, we note that the strongest indicators of each construct vary as the training population is refined, highlighting the limitations of generalized approaches. This work supports the scalability of webcam-based cognitive modeling and underscores the potential for personalized learning analytics and modeling to better support diverse learning needs.

For more than a decade, a handful of theoretical models have shaped a substantial amount of the research related to students’ emotional experiences during learning. This research has been productive, but articulating the underlying implicit assumptions in existing theories and their implications in our empirical interpretations can help to better investigate the reciprocal relationships between learning and emotion, and subsequently, to develop better interventions. This paper expands upon the existing theoretical frameworks, increasing the types of questions we ask about affect dynamics. We do so within the context of Crystal Island, a virtual world that allows middle school students to investigate microbiology questions. Specifically, we use this data to examine and revise the assumptions that are implicit in these models and the methods we use to investigate them.

Student affect is widely recognized as a major influence on learning gains and engagement, which has led to the development of many automated affect detectors. However, in order to respond effectively to student affect, we must know how students interpret it. This study proposes a novel automated detector that models when students attribute their epistemic emotion to task difficulty. The goal is to use detectors like this one to better understand how to respond to students' affective states (in this case, boredom, confusion, frustration and nervousness). We then discuss the implications of this novel detector for real-time support in game-based learning environments.

Peer feedback can be a powerful tool as it presents learning opportunities for both the learner receiving feedback as well as the learner providing feedback. Despite its utility, it can be difficult to implement effectively, particularly for younger learners, who are often novices at providing feedback. It can be difficult for students to learn what constitutes “good” feedback – particularly in open-ended problem-solving contexts. To address this gap, we investigate both classical natural language processing techniques and large language models, specifically ChatGPT, as potential approaches to devise an automated detector of feedback quality (including both student progress towards goals and next steps needed). Our findings indicate that the classical detectors are highly accurate and, through feature analysis, we elucidate the pivotal elements influencing its decision process. We find that ChatGPT is less accurate than classical NLP but illustrate the potential of ChatGPT in evaluating feedback, by generating explanations for ratings, along with scores. We discuss how the detector can be used for automated feedback evaluation and to better scaffold peer feedback for younger learners.

Generative AI has the potential to scale a number of educational practices, previously limited by resources. One such instructional approach is mastery learning, a pedagogy emphasizing proficiency before progression that is highly resource (teacher time, materials) intensive. The rise of computer-based instruction offered partial solutions, tailoring student progression and automating some facets of the mastery learning process. This work in progress considers the application of large language models for content generation tailored to mastery learning. We present a paired framework for analyzing and evaluating the generated content relative to rubrics designed by the teacher. Recognizing the potential of large language models, we critically assess the potential of improving mastery-based instruction. We close our discussion by considering the applications and limitations of this approach.

Previous learning analytics efforts have attempted to leverage the link between students’ gaze behaviors and learning experiences to build effective real-time interventions. Historically, however, these technologies have not been scalable due to the high cost of eye-tracking devices. Further, such efforts have been almost exclusively focused on neurotypical students, despite recent work that suggests a “one size fits many” approach can disadvantage neurodivergent students. Here we attempt to address these limitations by examining the validity and applicability of using scalable, webcam-based eye tracking as a basis for adaptively responding to neurodivergent students in an educational setting. Forty-three neurodivergent students read a text and answered questions about their in-situ thought patterns while a webcam-based eye tracker assessed their gaze locations. Results indicate that eye-tracking measures were sensitive to: 1) moments when students experienced difficulty disengaging from their own thoughts and 2) students’ familiarity with the text. Our findings highlight the fact that a free, open-source, webcam-based eye-tracker can be used to assess differences in reading patterns and online thought patterns. We discuss the implications and possible applications of these results, including the idea that webcam-based eye tracking may be a viable solution for designing real-time interventions for neurodivergent student populations.

Massive Open Online Courses (MOOCs) have increased the accessibility of quality educational content to a broader audience across a global network. They provide access for students to material that would be difficult to obtain locally, and an abundance of data for educational researchers. Despite the international reach of MOOCs, however, the majority of MOOC research does not account for demographic differences relating to the learners' country of origin or cultural background, which have been shown to have implications on the robustness of predictive models and interventions. This paper presents an exploration into the role of nation-level metrics of culture, happiness, wealth, and size on the generalizability of completion prediction models across countries. The findings indicate that various dimensions of culture are predictive of cross-country model generalizability. Specifically, learners from indulgent, collectivist, uncertainty-accepting, or short-term oriented, countries produce more generalizable predictive models of learner completion.

Eye tracking has been a research tool for decades, providing insights into interactions, usability, and, more recently, gaze-enabled interfaces. Recent work has utilized consumer-grade and webcam-based eye tracking, but is limited by the need to repeatedly calibrate the tracker, which becomes cumbersome for use outside the lab. To address this limitation, we developed an unsupervised algorithm that maps gaze vectors from a webcam to fixation features used for user modeling, bypassing the need for screen-based gaze coordinates, which require a calibration process. We evaluated our approach using three datasets (N=377) encompassing different UIs (computerized reading, an Intelligent Tutoring System), environments (laboratory or the classroom), and a traditional gaze tracker used for comparison. Our research shows that webcam-based gaze features correlate moderately with eye-tracker-based features and can model user engagement and comprehension as accurately as the latter. We discuss applications for research and gaze-enabled user interfaces for long-term use in the wild.

Eye movements provide a window into cognitive processes, but much of the research harnessing this data has been confined to the laboratory. We address whether eye gaze can be passively, reliably, and privately recorded in real-world environments across extended timeframes using commercial-off-the-shelf (COTS) sensors. We recorded eye gaze data from a COTS tracker embedded in participants (N=20) work environments at pseudorandom intervals across a two-week period. We found that valid samples were recorded approximately 30% of the time despite calibrating the eye tracker only once and without placing any other restrictions on participants. The number of valid samples decreased over days with the degree of decrease dependent on contextual variables (i.e., frequency of video conferencing) and individual difference attributes (e.g., sleep quality and multitasking ability). Participants reported that sensors did not change or impact their work. Our findings suggest the potential for the collection of eye-gaze in authentic environments.

We designed and tested an attention-aware learning technology (AALT) that detects and responds to mind wandering (MW), a shift in attention from task-related to task-unrelated thoughts, that is negatively associated with learning. We leveraged an existing gaze-based mind wandering detector that uses commercial off the shelf eye tracking to inform real-time interventions during learning with an Intelligent Tutoring System in real-world classrooms. The intervention strategies, co-designed with students and teachers, consisted of using student names, reiterating content, and asking questions, with the aim to reengage wandering minds and improve learning. After several rounds of iterative refinement, we tested our AALT in two classroom studies with 287 high-school students. We found that interventions successfully reoriented attention, and compared to two control conditions, reduced mind wandering, and improved retention (measured via a delayed assessment) for students with low prior-knowledge who occasionally (but not excessively) mind wandered. We discuss implications for developing gaze-based AALTs for real-world contexts.

Students using online learning environments need to effectively self-regulate their learning. However, with an absence of teacher-provided structure, students often resort to less effective, passive learning strategies versus constructive ones. We consider the potential benefits of interventions that promote retrieval practice – retrieving learned content from memory – which is an effective strategy for learning and retention. The goal is to nudge students towards completing short, formative quizzes when they are likely to succeed on those assessments. Towards this goal, we developed a machine-learning model using data from 32,685 students who used an online mathematics platform over an entire school year to prospectively predict scores on three-item assessments (N = 210,020) from interaction patterns up to 9 minutes before the assessment as well as Item Response Theory (IRT) estimates of student ability and quiz difficulty. These models achieved a student-independent correlation of 0.55 between predicted and actual scores on the assessments and outperformed IRT-only predictions (r = 0.34). Model performance was largely independent of the length of the analyzed window preceding a quiz. We discuss potential for future applications of the models to trigger dynamic interventions that aim to encourage students to engage with formative assessments rather than more passive learning strategies.

We developed generalizable affect detectors using 133,966 instances of 18 affective states collected from 69,174 students who interacted with an online math learning platform called Algebra Nation over the entire school year. To enable scalability and generalizability, we used generic interaction features (e.g., viewing a video, taking a quiz), which do not require specialized sensors and are domain- and (to a certain extent) system-independent. We experimented with standard classifiers, recurrent neural networks, and genetically evolved neural networks for affect modeling. Prediction accuracies, quantified with Spearman's rho, were modest and ranged from .08 (for surprise) to .34 (for happiness) with a mean of .25. Our model trained on Algebra students generalized to a different set of Geometry students (n = 28,458) on the same platform. We discuss implications for scaling up affect detection for affect-sensitive online learning environments which aim to improve engagement and learning by detecting and responding to student affect.

It is widely acknowledged that the language we use reflects numerous psychological constructs, including our thoughts, feelings, and desires. Can the so called "noncognitive" traits with known links to success, such as growth mindset, leadership ability, and intrinsic motivation, be similarly revealed through language? We investigated this question by analyzing students' 150-word open-ended descriptions of their own extracurricular activities or work experiences included in their college applications. We used the Common Application-National Student Clearinghouse data set, a six-year longitudinal dataset that includes college application data and graduation outcomes for 278,201 U.S. high-school students. We first developed a coding scheme from a stratified sample of 4,000 essays and used it to code seven traits: growth mindset, perseverance, goal orientation, leadership, psychological connection (intrinsic motivation), self-transcendent (prosocial) purpose, and team orientation, along with earned accolades. Then, we used standard classifiers with bag-of-n-grams as features and deep learning techniques (recurrent neural networks) with word embeddings to automate the coding. The models demonstrated convergent validity with the human coding with AUCs ranging from .770 to .925 and correlations ranging from .418 to .734. There was also evidence of discriminant validity in the pattern of inter-correlations (rs between -.206 to .306) for both human- and model-coded traits. Finally, the models demonstrated incremental predictive validity in predicting six-year graduation outcomes net of sociodemographics, intelligence, academic achievement, and institutional graduation rates. We conclude that language provides a lens into noncognitive traits important for college success, which can be captured with automated methods.

We leverage a unique national dataset of 41,359 college applications to prospectively predict 4-year bachelor's graduation in a generalizable manner. Our features include sociodemographics, institutional graduation rates, academic achievement, standardized test scores, engagement in extracurricular activities, work experiences, and ratings by teachers and high-school guidance counselors. A random forest classifier successfully predicted 4-year graduation for 71.4% of the students (base rate = 44%) using all 166 of the aforementioned features and a split-half validation method. A stochastic hill-climbing feature selection procedure effectively maintained the same classification accuracy, but with a minimal set of 37 features, consisting of an approximately equal representation of sociodemographics, cognitive, and noncognitive factors. We advocate against using these results for admissions decisions, instead contemplating how they might be used to provide parents and educators with actionable information to guide students towards college success.

There has been a recent surge in the need for computer science educational resources. Often, this need is addressed with computer-based learning tools. However, many of these tools target coding skills and software design rather than teaching foundational topics. This work proposes an interactive tool for teaching process management using the common abstraction of the Dining Philosophers Problem. We demonstrate the effectiveness and entertainment value of this tool with feedback from undergraduate students enrolled in computer science courses. We believe this type of educational tool more adequately addresses the upcoming need for additional teaching resources in computer science, especially when used in conjunction with a traditional teaching environment.

Attention is critical to learning. Hence, advanced learning technologies should benefit from mechanisms to monitor and respond to learners' attentional states. We study the feasibility of integrating commercial off-the-shelf (COTS) eye trackers to monitor attention during interactions with a learning technology called GuruTutor. We tested our implementation on 135 students in a noisy computer-enabled high school classroom and were able to collect a median 95% valid eye gaze data in 85% of the sessions where gaze data was successfully recorded. Machine learning methods were employed to develop automated detectors of mind wandering (MW) -- a phenomenon involving a shift in attention from task-related to task-unrelated thoughts that is negatively correlated with performance. Our student-independent, gaze-based models could detect MW with an accuracy (F1 of MW = 0.59) significantly greater than chance (F1 of MW = 0.24). Predicted rates of mind wandering were negatively related to posttest performance, providing evidence for the predictive validity of the detector. We discuss next steps towards developing gaze-based, attention-aware, learning technologies that can be deployed in noisy, real-world environments.