Persistence after failure is critical for learning—but when students make mistakes in intelligent tutoring systems, they often choose not to try again. How can digital platforms encourage students to persist at these moments? We conducted a randomized controlled trial in an intelligent tutoring system for math and science, involving 164,532 students (Grades 8-12) who completed 17 million practice problems. We tested two scalable interventions: a brief persuasive prompt encouraging students to try again, and a visual default nudge that highlighted the retry option. Both interventions increased persistence after failure, and when combined, their effects were additive—suggesting they operate through distinct psychological mechanisms. The nudge had a much larger immediate effect, but the prompt showed proportionally greater spillover to untreated problems. These findings advance theories of persuasive design, demonstrating that implicit, interface-level nudges and explicit motivational prompts can be combined to avoid redundancy while amplifying impact.

Most learners worldwide are multilingual, yet implementing multilingual education remains challenging in practice. EdTech offers an opportunity to bridge this gap and expand access for linguistically diverse learners. We conducted a quasi-experiment in Uganda with 2,931 participants enrolled in a non-formal radio- and mobile-based engineering course, where learners self-selected instruction in Leb Lango (a local language), English, or a Hybrid option combining both languages. The Leb Lango version of the course was used disproportionately by learners from rural areas, those with less formal education, and those with lower prior knowledge, broadening participation among disadvantaged learners. Moreover, the availability of Leb Lango instruction was associated with higher active participation, even among learners who registered for English instruction. Although Leb Lango learners began with lower performance, they demonstrated faster learning gains and achieved comparable final examination outcomes to English and Hybrid learners. These results suggest that providing local language options to learners is an effective way to make EdTech more accessible.

As AI tutors become increasingly capable of delivering rich, personalized feedback at scale, a key challenge remains: novice learners often struggle to process detailed explanations on their own. Structured reflection, grounded in decades of self-explanation research, is a theoretically compelling solution. By helping learners parse feedback and prompting them to actively interpret it, reflection activities are designed to reduce cognitive overload and deepen understanding. But does adding reflection to already rich AI-generated feedback actually help, or does it simply add friction? We tested this in a randomized experiment comparing Python practice with AI-generated, personalized feedback to ''reflective practice,'' which paired identical feedback with structured self-explanation prompts. Contrary to our predictions, reflection never improved performance on any measure. Instead, it proved to be a temporal bottleneck: it doubled time spent on feedback and reduced practice volume by 40%, without making each learning opportunity more effective. Learners who cycled through more practice-and-feedback iterations outperformed reflective learners at the end of the session and maintained a small, nonsignificant advantage on transfer. Notably, reflection did not provide the scaffolding benefit we predicted for novices—and when individual differences did emerge, they favored higher-volume practice for more knowledgeable learners. Both practice conditions also substantially outperformed a high-quality video baseline (d = 0.66–0.93), replicating benefits of active practice with feedback over passive instruction. These findings provide initial evidence that when feedback is already elaborated and personalized, self-explanation activities may add little value, particularly when their time-related costs are considered. As AI-generated feedback reaches learners at scale, these findings underscore the necessity of empirically validating pedagogical scaffolds—even those with strong theoretical support—before deploying them broadly.

Given the cumulative nature of computer science, success in introductory computing (CS1) courses requires students to not only learn the material but also develop effective self-regulated learning (SRL) habits. While theories of SRL emphasize planning, performance, and self-reflection as essential phases of effective learning, there is limited evidence on how to help learners put these phases into practice. In this context, Mastery-Based Tests (MBT), which allow students to retake assessments after receiving feedback, have shown promise for improving learning outcomes. However, prior work in computer science is largely observational and does not directly test MBT's impact on SRL behaviors. This paper presents a pilot study (N = 6) exploring this relationship in CS1. Using a between-subjects design, we observed that learners who first completed an MBT achieved higher post-test scores, demonstrated higher metacognitive accuracy, and self-reported more productive SRL behaviors. These patterns suggest that MBTs warrant further investigation as a viable scaffold for fostering self-regulation in CS1.

The Doer Effect states that completing more active learning activities, like practice questions, is more strongly related to positive learning outcomes than passive learning activities, like reading, watching, or listening to course materials. Although broad, most evidence has emerged from practice with tutoring systems in Western, Industrialized, Rich, Educated, and Democratic (WEIRD) populations in North America and Europe. Does the Doer Effect generalize beyond WEIRD populations, where learners may practice in remote locales through different technologies? Through learning analytics, we provide evidence from N = 234 Ugandan students answering multiple-choice questions via phones and listening to lectures via community radio. Our findings support the hypothesis that active learning is more associated with learning outcomes than passive learning. We find this relationship is weaker for learners with higher prior educational attainment. Our findings motivate further study of the Doer Effect in diverse populations. We offer considerations for future research in designing and evaluating contextually relevant active and passive learning opportunities including leveraging familiar technology, increasing the number of practice opportunities, and aligning multiple data sources.

Expanding access to education in rural African communities remains difficult, largely due to limited internet connectivity. Mobile learning courses delivered via radio and offline mobile phones offer a promising, scalable solution. However, it is challenging to track student engagement in these environments due to the absence of tools that monitor students' interactions with the radio. In this study, we investigate the potential of ''Prize Codes'' -- codes read aloud during broadcasts that students enter via text message -- to serve as a real-time measure of student engagement with mobile-learning broadcasts. Using data from a 2024 implementation of Yiya AirScience, a mobile engineering course in Uganda, we evaluate the validity of Prize Codes as an engagement metric. Specifically, we test whether Prize Code measures (1) demonstrate reliability, with students who enter correct codes in one lesson being more likely to do so in subsequent lessons; (2) demonstrate convergent validity with existing measures of engagement; and (3) demonstrate predictive validity, predicting learning outcomes in the course. Our findings suggest that Prize Codes are a reliable and valid measure of engagement. Prize-Code accuracy demonstrates strong internal consistency (alpha = .97) and moderate test-retest reliability (ICC = .44). The measure aligns closely with synchronous participation (87% agreement, Cohen's kappa = .50), indicating it captures similar engagement patterns. Importantly, students who consistently enter correct Prize Codes perform significantly better on assessments, with Prize Code engagement predicting final exam scores above and beyond other engagement metrics. After establishing the measure's validity, we use it to (1) characterize patterns of engagement with Yiya broadcasts, (2) investigate early engagement with the broadcasts as a predictor of course persistence, and (3) replicate findings about the benefits of learning by doing. This study suggests that Prize Codes can be a feasible, scalable approach for tracking real-time engagement in resource-limited mobile learning settings at scale.

Gaming the system, a behavior in which learners exploit a system's properties to make progress while avoiding learning, has frequently been shown to be associated with lower learning. However, when we applied a previously validated gaming detector across conditions in experiments with an algebra tutor, the detected gaming was not associated with reduced learning, challenging its validity in our study context. Our exploratory data analysis suggested that varying contextual factors across and within conditions contributed to this lack of association. We present a new approach, latent variable-based gaming detection (LV-GD), that controls for contextual factors and more robustly estimates student-level latent gaming tendencies. In LV-GD, a student is estimated as having a high gaming tendency if the student is detected to game more than the expected level of the population given the context. LV-GD applies a statistical model on top of an existing action-level gaming detector developed based on a typical human labeling process, without additional labeling effort. Across three datasets, we find that LV-GD consistently outperformed the original detector in validity measured by association between gaming and learning as well as reliability. LV-GD also afforded high practical utility: it more accurately revealed intervention effects on gaming, revealed a correlation between gaming and perceived competence in math and helped understand productive detected gaming behaviors. Our approach is not only useful for others wanting a cost-effective way to adapt a gaming detector to their context but is also generally applicable in creating robust behavioral measures.

Analytics of student learning data are increasingly important for continuous redesign and improvement of tutoring systems and courses. There is still a lack of general guidance on converting analytics into better system design, and on combining multiple methods to maximally improve a tutor. We present a multi-method approach to data-driven redesign of tutoring systems and its empirical evaluation. Our approach systematically combines existing and new learning analytics and instructional design methods. In particular, our methods involve identifying difficult skills and creating focused tasks for learning these difficult skills effectively following content redesign strategies derived from analytics. In our past work, we applied this approach to redesigning an algebraic modeling unit and found initial evidence of its effectiveness. In the current work, we extended this approach and applied it to redesigning two other tutor units in addition to a second iteration of redesigning the previously redesigned unit. We conducted a one-month classroom experiment with 129 high school students. Compared to the original tutor, the redesigned tutor led to significantly higher learning outcomes, with time mainly allocated to focused tasks rather than original full tasks. Moreover, it reduced over- and under-practice, yielded a more effective practice experience, and selected skills progressing from easier to harder to a greater degree. Our work provides empirical evidence of the effectiveness and generality of a multi-method approach to data-driven instructional redesign.