As AI tutoring systems are tested for use in K-12 classrooms, understanding their effects on learning and how interaction modality shapes student engagement is essential. Across three studies, students could ask for help from a generative AI tutor chatbot which responded in a text, audio, or video format during classroom math lessons. In the text modality, students experienced a typical text-only chatbot experience. In audio and video modalities, text was still present but each message was read aloud by an AI-generated voice and, in the case of video, accompanied by a human-like AI-generated avatar. In Study 1, we collected qualitative feedback on each modality. In Study 2, students were randomly assigned to one modality as they worked through solving math problems as part of normal course work. In Study 3, students were assigned a default modality but had the agency to switch modality. Results indicated a clear preference for text. Although initial engagement was higher in the audio condition, students frequently switched to text when given a choice. Student feedback and behavior also showed an aversion to the video modality. When modality was assigned, performance was also lowest in the video condition. However, when students could choose their modality, these differences disappeared. Instead, students who exercised agency over the modality engaged in longer conversations with the tutor, and the increased engagement fully mediated the effect of agency on accuracy. These findings suggest that giving students agency over modality can support sustained interactions with an AI tutor leading to higher performance.

We developed the A.I. Math Personalization Tool (AMPT) to enhance cultural relevance in math word problems by giving students agency over the content. AMPT leverages generative AI to directly engage students as co-authors of math word problems. Through scaffolded conversations, the AI allows students to provide the context for a problem. Then, the AI integrates that context with the pedagogical standards of a target learning domain. We measured the attitudes of students towards mathematics before and after interaction with AMPT. After a single 30-min session co-authoring math word problems with AMPT, students’ sense of belonging in mathematics significantly increased, while other attitudes remained unchanged. AMPT provided students with the opportunity to express themselves and see their interests reflected in the math domain. After experiencing this level of agency over math content, their sense of belonging in mathematics increased. The results of this study demonstrate the potential for generative AI to enhance student choice, motivation, and, ultimately, achievement in mathematics.

Educational stakeholders are often particularly interested in sparse, delayed student outcomes, like end-of-year statewide exams. The rare occurrence of such assessments makes it harder to identify students likely to fail such assessments, as well as making it slow for researchers and educators to be able to assess the effectiveness of particular educational tools. Prior work has primarily focused on using logs from students full usage (e.g. year-long) of an educational product to predict outcomes, or considered predictive accuracy using a few minutes to predict outcomes after a short (e.g. 1 hour) session. In contrast, we investigate machine learning predictors using students' logs during their first few hours of usage can provide useful predictive insight into those students' end-of-school year external assessment. We do this on three diverse datasets: from students in Uganda using a literacy game product, and from students in the US using two mathematics intelligent tutoring systems. We consider various measures of the accuracy of the resulting predictors, including its ability to identify students at different parts along the assessment performance distribution. Our findings suggest that short-term log usage data, from 2-5 hours, can be used to provide valuable signal about students' long-term external performance.

Measuring algorithmic bias in machine learning has historically focused on statistical inequalities pertaining to specific groups. However, the most common metrics (i.e., those focused on individual- or group-conditioned error rates) are not currently well-suited to educational settings because they assume that each individual observation is independent from the others. This is not statistically appropriate when studying certain common educational outcomes, because such metrics cannot account for the relationship between students in classrooms or multiple observations per student across an academic year. In this paper, we present novel adaptations of algorithmic bias measurements for regression for both independent and nested data structures. Using hierarchical linear models, we rigorously measure algorithmic bias in a machine learning model of the relationship between student engagement in an intelligent tutoring system and year-end standardized test scores. We conclude that classroom-level influences had a small but significant effect on models. Examining significance with hierarchical linear models helps determine which inequalities in educational settings might be explained by small sample sizes rather than systematic differences.

Adaptive educational software is likely to better support broader and more diverse sets of learners by considering more comprehensive views (or models) of such learners. For example, recent work proposed making inferences about “non-math” factors like reading comprehension while students used adaptive software for mathematics to better support and adapt to learners. We build on this proposed approach to more comprehensive learning modeling by providing an empirical basis for making inferences about students’ reading ability from their performance on activities in adaptive software for mathematics. We lay out an approach to predicting middle school students’ reading ability using their performance on activities within Carnegie Learning’s MATHia, a widely used intelligent tutoring system for mathematics. We focus on how performance in an early, introductory activity as an especially powerful place to consider instruction-embedded assessment of non-math factors like reading comprehension to guide adaptation based on factors like reading ability. We close by discussing opportunities to extend this work by focusing on particular knowledge components or skills tracked by MATHia that may provide important “levers” for driving adaptation based on students’ reading ability while they learn and practice mathematics.