As AI tutoring systems are tested for use in K-12 classrooms, understanding their effects on learning and how interaction modality shapes student engagement is essential. Across three studies, students could ask for help from a generative AI tutor chatbot which responded in a text, audio, or video format during classroom math lessons. In the text modality, students experienced a typical text-only chatbot experience. In audio and video modalities, text was still present but each message was read aloud by an AI-generated voice and, in the case of video, accompanied by a human-like AI-generated avatar. In Study 1, we collected qualitative feedback on each modality. In Study 2, students were randomly assigned to one modality as they worked through solving math problems as part of normal course work. In Study 3, students were assigned a default modality but had the agency to switch modality. Results indicated a clear preference for text. Although initial engagement was higher in the audio condition, students frequently switched to text when given a choice. Student feedback and behavior also showed an aversion to the video modality. When modality was assigned, performance was also lowest in the video condition. However, when students could choose their modality, these differences disappeared. Instead, students who exercised agency over the modality engaged in longer conversations with the tutor, and the increased engagement fully mediated the effect of agency on accuracy. These findings suggest that giving students agency over modality can support sustained interactions with an AI tutor leading to higher performance.

We investigate how students learn to apply context-specific math strategies by analyzing data from MATHia (a widely used Intelligent Tutoring System) collected from a large set of schools. In particular, we focus on a set of lessons designed to teach ratios and proportions, where students learn multiple strategies individually and then are presented with lessons in which they are presented with options to make a choice between strategies. Our results demonstrate that a majority of students may not learn conditional reasoning to select optimal strategies. To understand this more deeply, we use knowledge tracing models and also explain and interpret neural representations of strategies learned using BERT from step-level interactions between students and the ITS. Finally, we study the effectiveness of MATHia’s adaptive supports that attempt to guide students to the optimal strategy, and compare insights from our data to those produced by state-of-the-art generative AI models. Our results demonstrate that LLMs may produce results that seem to reflect ideal, expected outcomes in strategy learning, but the generation may not accurately reflect the complexities of real student learning.

Student motivation is essential for learning, especially in digital environments where students must regulate their own engagement. Reward systems are commonly used to encourage engagement, but these features can go unused, either because students do not find the rewards engaging or because they are not well understood. In the present study, we tested whether increased visibility of a reward system within a mathematics learning platform for middle-school students would drive increased engagement and whether this was associated with improved performance within the platform. Using log data from 28,749 students across the United States, we analyzed how the purchase and application of non-instructional rewards related to lesson completion, passing the first attempt, and passing after initial failure. In a series of path models, greater visibility significantly increased both purchasing and application of rewards which in turn increased completion rates of lessons. The relationship between applying rewards and passing the lesson varied by reward type. Opting to use rewards contingent on success predicted higher pass rates whereas reward use contingent on completion predicted lower pass rates. Overall, the findings suggested that engagement with a reward system can promote persistence, and the types of rewards learners choose can act as signals for different states (e.g., struggling). However, all of this is contingent on learners' understanding and perceived value of the system.

We developed the A.I. Math Personalization Tool (AMPT) to enhance cultural relevance in math word problems by giving students agency over the content. AMPT leverages generative AI to directly engage students as co-authors of math word problems. Through scaffolded conversations, the AI allows students to provide the context for a problem. Then, the AI integrates that context with the pedagogical standards of a target learning domain. We measured the attitudes of students towards mathematics before and after interaction with AMPT. After a single 30-min session co-authoring math word problems with AMPT, students’ sense of belonging in mathematics significantly increased, while other attitudes remained unchanged. AMPT provided students with the opportunity to express themselves and see their interests reflected in the math domain. After experiencing this level of agency over math content, their sense of belonging in mathematics increased. The results of this study demonstrate the potential for generative AI to enhance student choice, motivation, and, ultimately, achievement in mathematics.

Learning engineering applies data and learning science principles to better understand outcomes and support improvement research. One important approach is A/B testing-common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE), and the International Consortium for Innovation and Collaboration in Learning Engineering (IEEE ICICLE). Several systems supporting A/B testing in educational applications have arisen recently, including UpGrade, E-TRIALS, and Terracotta. A/B testing can help improve educational platforms, yet there are challenging issues unique to conducting such work in these contexts. In response, a number of digital learning platforms have opened their systems to learning-improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore how A/B testing is conducted in educational contexts, how digital learning platforms are accelerating education research, and how empirical approaches can be used to drive powerful gains in student learning. It will also discuss opportunities for funding to conduct platform-enabled learning engineering.

Learning engineering adds tools and processes to learning platforms to support improvement research. One kind of tool is A/B testing-common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE), and the International Consortium for Innovation and Collaboration in Learning Engineering (IEEE ICICLE). Recently, several A/B testing systems have arisen that focus on conducting research in educational environments, including UpGrade, Terracotta, and E-TRIALS. A/B testing can help improve educational platforms, yet there are challenging issues unique to conducting such work in these contexts. In response, a number of digital learning platforms have opened their systems to learning-improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore challenges of A/B testing in educational contexts, how learning platforms are accelerating education research, and how empirical approaches can be used to drive powerful gains in student learning. It will also discuss opportunities for funding to conduct platform-enabled learning research.

Adapting to a student's problem solving strategy can lead to improved engagement and motivation. In this work, we develop an AI-based approach to analyze math learning strategies at scale. Specifically, we use a state-of-the-art AI model, namely, BERT to learn structure within strategies observed in large datasets. In particular, we consider the MATHia ITS and define strategies as sequences of steps that a student follows in solving the problem. We apply BERT pre-training to learn semantic representations of strategies from a workspace in MATHia that allows for different strategies. Further, we fine-tune these embeddings to train them on downstream tasks such as identifying a strategy and understanding drift in strategy. Our preliminary results are encouraging and demonstrate that BERT can uncover hidden structure in strategies and therefore is a promising direction to analyze large-scale math learning data.