As AI tutoring systems are tested for use in K-12 classrooms, understanding their effects on learning and how interaction modality shapes student engagement is essential. Across three studies, students could ask for help from a generative AI tutor chatbot which responded in a text, audio, or video format during classroom math lessons. In the text modality, students experienced a typical text-only chatbot experience. In audio and video modalities, text was still present but each message was read aloud by an AI-generated voice and, in the case of video, accompanied by a human-like AI-generated avatar. In Study 1, we collected qualitative feedback on each modality. In Study 2, students were randomly assigned to one modality as they worked through solving math problems as part of normal course work. In Study 3, students were assigned a default modality but had the agency to switch modality. Results indicated a clear preference for text. Although initial engagement was higher in the audio condition, students frequently switched to text when given a choice. Student feedback and behavior also showed an aversion to the video modality. When modality was assigned, performance was also lowest in the video condition. However, when students could choose their modality, these differences disappeared. Instead, students who exercised agency over the modality engaged in longer conversations with the tutor, and the increased engagement fully mediated the effect of agency on accuracy. These findings suggest that giving students agency over modality can support sustained interactions with an AI tutor leading to higher performance.

We investigate how students learn to apply context-specific math strategies by analyzing data from MATHia (a widely used Intelligent Tutoring System) collected from a large set of schools. In particular, we focus on a set of lessons designed to teach ratios and proportions, where students learn multiple strategies individually and then are presented with lessons in which they are presented with options to make a choice between strategies. Our results demonstrate that a majority of students may not learn conditional reasoning to select optimal strategies. To understand this more deeply, we use knowledge tracing models and also explain and interpret neural representations of strategies learned using BERT from step-level interactions between students and the ITS. Finally, we study the effectiveness of MATHia’s adaptive supports that attempt to guide students to the optimal strategy, and compare insights from our data to those produced by state-of-the-art generative AI models. Our results demonstrate that LLMs may produce results that seem to reflect ideal, expected outcomes in strategy learning, but the generation may not accurately reflect the complexities of real student learning.

We developed the A.I. Math Personalization Tool (AMPT) to enhance cultural relevance in math word problems by giving students agency over the content. AMPT leverages generative AI to directly engage students as co-authors of math word problems. Through scaffolded conversations, the AI allows students to provide the context for a problem. Then, the AI integrates that context with the pedagogical standards of a target learning domain. We measured the attitudes of students towards mathematics before and after interaction with AMPT. After a single 30-min session co-authoring math word problems with AMPT, students’ sense of belonging in mathematics significantly increased, while other attitudes remained unchanged. AMPT provided students with the opportunity to express themselves and see their interests reflected in the math domain. After experiencing this level of agency over math content, their sense of belonging in mathematics increased. The results of this study demonstrate the potential for generative AI to enhance student choice, motivation, and, ultimately, achievement in mathematics.

Learning engineering applies data and learning science principles to better understand outcomes and support improvement research. One important approach is A/B testing-common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE), and the International Consortium for Innovation and Collaboration in Learning Engineering (IEEE ICICLE). Several systems supporting A/B testing in educational applications have arisen recently, including UpGrade, E-TRIALS, and Terracotta. A/B testing can help improve educational platforms, yet there are challenging issues unique to conducting such work in these contexts. In response, a number of digital learning platforms have opened their systems to learning-improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore how A/B testing is conducted in educational contexts, how digital learning platforms are accelerating education research, and how empirical approaches can be used to drive powerful gains in student learning. It will also discuss opportunities for funding to conduct platform-enabled learning engineering.

Measuring algorithmic bias in machine learning has historically focused on statistical inequalities pertaining to specific groups. However, the most common metrics (i.e., those focused on individual- or group-conditioned error rates) are not currently well-suited to educational settings because they assume that each individual observation is independent from the others. This is not statistically appropriate when studying certain common educational outcomes, because such metrics cannot account for the relationship between students in classrooms or multiple observations per student across an academic year. In this paper, we present novel adaptations of algorithmic bias measurements for regression for both independent and nested data structures. Using hierarchical linear models, we rigorously measure algorithmic bias in a machine learning model of the relationship between student engagement in an intelligent tutoring system and year-end standardized test scores. We conclude that classroom-level influences had a small but significant effect on models. Examining significance with hierarchical linear models helps determine which inequalities in educational settings might be explained by small sample sizes rather than systematic differences.

Learning engineering adds tools and processes to learning platforms to support improvement research. One kind of tool is A/B testing-common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE), and the International Consortium for Innovation and Collaboration in Learning Engineering (IEEE ICICLE). Recently, several A/B testing systems have arisen that focus on conducting research in educational environments, including UpGrade, Terracotta, and E-TRIALS. A/B testing can help improve educational platforms, yet there are challenging issues unique to conducting such work in these contexts. In response, a number of digital learning platforms have opened their systems to learning-improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore challenges of A/B testing in educational contexts, how learning platforms are accelerating education research, and how empirical approaches can be used to drive powerful gains in student learning. It will also discuss opportunities for funding to conduct platform-enabled learning research.

There is an increasing opportunity for AI-supported teacher-student orchestration. Preliminary evidence in small studies suggests that AI that better supports such coordination could lead to substantial student learning gains but much is unknown about how such orchestration might work at scale. In this work we focus on an existing tool, widely available to teachers as part of a commonly used math platform, to gain insights into how teachers perceive this tool and how they use a feature which allows them to mark when and why they help particular students as those students work on the software. Our teacher survey reveals that many teachers do use the tool's suggestion to inform which students to support, and our quantitative analysis of log files shows that when marking a student as helped, teachers most often report providing encouragement. Providing encouragement is far more frequent than providing direct math instruction, though when students are highlighted as being likely to fail the current section, teachers more frequently mark that they provided direct math help. Our work helps showcase the existing use and potential new directions for AI-supported teacher-student orchestration at scale.

Adapting to a student's problem solving strategy can lead to improved engagement and motivation. In this work, we develop an AI-based approach to analyze math learning strategies at scale. Specifically, we use a state-of-the-art AI model, namely, BERT to learn structure within strategies observed in large datasets. In particular, we consider the MATHia ITS and define strategies as sequences of steps that a student follows in solving the problem. We apply BERT pre-training to learn semantic representations of strategies from a workspace in MATHia that allows for different strategies. Further, we fine-tune these embeddings to train them on downstream tasks such as identifying a strategy and understanding drift in strategy. Our preliminary results are encouraging and demonstrate that BERT can uncover hidden structure in strategies and therefore is a promising direction to analyze large-scale math learning data.

Adaptive educational software is likely to better support broader and more diverse sets of learners by considering more comprehensive views (or models) of such learners. For example, recent work proposed making inferences about “non-math” factors like reading comprehension while students used adaptive software for mathematics to better support and adapt to learners. We build on this proposed approach to more comprehensive learning modeling by providing an empirical basis for making inferences about students’ reading ability from their performance on activities in adaptive software for mathematics. We lay out an approach to predicting middle school students’ reading ability using their performance on activities within Carnegie Learning’s MATHia, a widely used intelligent tutoring system for mathematics. We focus on how performance in an early, introductory activity as an especially powerful place to consider instruction-embedded assessment of non-math factors like reading comprehension to guide adaptation based on factors like reading ability. We close by discussing opportunities to extend this work by focusing on particular knowledge components or skills tracked by MATHia that may provide important “levers” for driving adaptation based on students’ reading ability while they learn and practice mathematics.

Abstract Automated, data‐driven decision making is increasingly common in a variety of application domains. In educational software, for example, machine learning has been applied to tasks like selecting the next exercise for students to complete. Machine learning methods, however, are not always equally effective for all groups of students. Current approaches to designing fair algorithms tend to focus on statistical measures concerning a small subset of legally protected categories like race or gender. Focusing solely on legally protected categories, however, can limit our understanding of bias and unfairness by ignoring the complexities of identity. We propose an alternative approach to categorization, grounded in sociological techniques of measuring identity. By soliciting survey data and interviews from the population being studied, we can build context‐specific categories from the bottom up. The emergent categories can then be combined with extant algorithmic fairness strategies to discover which identity groups are not well‐served, and thus where algorithms should be improved or avoided altogether. We focus on educational applications but present arguments that this approach should be adopted more broadly for issues of algorithmic fairness across a variety of applications.

Learning engineering adds tools and processes to learning platforms to support improvement research. One kind of tool is A/B testing, which is common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE). A number of A/B testing systems focused on educational applications have arisen recently, including UpGrade and E-TRIALS. A/B testing can be part of the puzzle of how to improve educational platforms, and yet challenging issues in education go beyond the generic paradigm. For example, the importance of teachers and instructors to learning means that students are not only connecting with software as individuals, but also as part of a shared classroom experience. Further, learning in topics like mathematics can be highly dependent on prior learning, and thus A or B may not be better overall, but only in interaction with prior knowledge. In response, a set of learning platforms is opening their systems to improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore how A/B testing in educational contexts is different, how learning platforms are opening up new possibilities, and how these empirical approaches can be used to drive powerful gains in student learning. It will also discuss forthcoming opportunities for funding to conduct platform-enabled learning research.

The emerging discipline of Learning Engineering is focused on putting into place tools and processes that use the science of learning as a basis for improving educational outcomes. An important part of Learning Engineering focuses on improving the effectiveness of educational software. In many software domains, A/B testing has become a prominent technique to achieve the software's goals. Many large companies (Amazon, Google, Facebook, etc.) run thousands of AB tests and present at the Annual Conference on Digital Experimentation (CODE), but that venue is too broad to address AB testing issues specific to EdTech platforms. We see a need to address issues with running large-scale A/B tests within the educational context, where the use of A/B testing lags other industries. This workshop will explore ways in which A/B testing in educational contexts differs from other domains and proposals to overcome current challenges so that this approach can become a more useful tool in the learning engineer's toolbox.

No abstract available.

If we wish to embed assessment for accountability within instruction, we need to better understand the relative contribution of different types of learner data to statistical models that predict scores on assessments used for accountability purposes. The present work scales up and extends predictive models of math test scores from existing literature and specifies six categories of models that incorporate information about student prior knowledge, socio-demographics, and performance within the MATHia intelligent tutoring system. Linear regression and random forest models are learned within each category and generalized over a sample of 23,000+ learners in Grades 6, 7, and 8 over three academic years in Miami-Dade County Public Schools. After briefly exploring hierarchical models of this data, we discuss a variety of technical and practical applications, limitations, and open questions related to this work, especially concerning to the potential use of instructional platforms like MATHia as a replacement for time-consuming standardized tests.

Nearly every adaptive learning system aims to present students with materials personalized to their level of understanding (Enyedy, 2014). Typically, such adaptation follows some form of mastery learning (Bloom, 1968), in which students are asked to master one topic before proceeding to the next topic. Mastery learning programs have a long history of success (Guskey and Gates, 1986; Kulik, Kulik & Bangert-Drowns, 1990) and have been shown to be superior to alternative instructional approaches.

Learners often think math is unrelated to their own interests. Instructional software has the potential to provide personalized instruction that responds to individuals' interests. Carnegie Learning's MATHia™ software for middle school mathematics asks learners to specify domains of their interest (e.g., sports & fitness, arts & music), as well as names of friends/classmates, and uses this information to both choose and personalize word problems for individual learners. Our analysis of MATHia's relatively coarse-grained personalization contrasts with more finegrained analysis in previous research on word problems in the Cognitive Tutor (e.g., finding effects on performance in parts of problems that depend on more difficult skills), and we explore associations of aggregate preference "honoring" with learner performance. To do so, we define a notion of "strong" learner interest area preferences and find that honoring such preferences has a small negative association with performance. However, learners that both merely express preferences (either interest area preferences or setting names of friends/classmates), and those that express strong preferences, tend to perform in ways that are associated with better learning compared to learners that do not express such preferences. We consider several explanations of these findings and suggest important topics for future research.