Measures of disengagement provide insights into unproductive use of learning opportunities. Although measures of active disengagement, such as gaming the system and mind-wandering, are well studied, loss of practice time due to outright task avoidance remains relatively understudied. The current study addresses this gap by extending existing within-task measures (idle time) with two new session-level measures (delayed start and early stop) to capture loss of practice time due to task avoidance. We characterize the combined lost time as coasted time and the associated behavior as coasting behavior. Using ASSISTments logs (N = 1,425), we find that students dedicate only 40% of available classwork time to math practice and coast through the remaining 60%. Of the coasted time, 36% resulted from delayed starts, 2% from mid-practice idling, and 62% from stopping early. Delayed start and early stop showed moderate temporal stability (G = 0.73 and 0.71, respectively), suggesting that coasting is a consistent behavioral pattern. Even after excluding early stops attributable to assignment completion (i.e., early stop = 0), coasted time remained substantial at 32%. While we observe significant differences in coasting by gender and IEP status, we do not observe them by other demographic factors or school locale. Critically, students who continued working beyond the first assignment completion (''extra effort'') performed significantly better on standardized tests. For research, coasting offers a new lens on opportunity loss by combining session-level disengagement with within-task disengagement. For practitioners, our results highlight the need for platform affordances that support sustained engagement and more productive use of available practice time.

Math anxiety has been found to negatively correlate with math achievement, affecting students’ choices to take fewer math classes and avoid math educational opportunities. Educational technology tools can ameliorate some of the negative effects of math anxiety. We examined students’ math anxiety, effort in an educational technology platform, and their relationship with students’ math achievement. Multilevel latent class analysis was used to identify student profiles of math anxiety. Regression analysis was used to examine how students of different profiles interacted with MathSpring, an adaptive intelligent tutor that provides affective supports to students during math problem-solving. The student's math achievement was measured by a standardized test. Our analysis indicated heterogeneity in math anxiety, and students could fall into one of three groups: Highly Anxious, Performance Anxious, and Calm. Highly Anxious students tended to give up more often when solving questions in MathSpring and had the lowest math achievement outcomes. For these students, using hints to solve problems in MathSpring was significantly associated with increased math outcomes. These findings have implications for the field's understanding of how students of different math anxiety profiles can demonstrate varying efforts in math educational technology platforms, and different math learning outcomes.

Students' effort and emotions are important contributors to math learning. In a recent study evaluating the efficacy of MathSpring, a scalable web-based intelligent tutoring system that provides students with personalized math problems and affective support, system usage data were collected for 804 U.S. 10-12-year-olds. To understand the patterns in students' interactions with MathSpring and how patterns vary across students, hierarchical cluster heatmap analysis was performed. Guided by the patterns from the heatmap, latent profile analysis was conducted to identify student subgroups. Both analyses indicated that there were two groups of students: ''Confident Solvers'' who solved problems on their first attempt and reported high confidence, and ''Struggling Solvers'' who gave up, did not read problems, and reported higher frustration. These analyses provide insight into students' behaviors in online learning environments that can be scaled to students nationwide. Developers can use these insights to design systems that respond to students' usage and inform teachers about students' effort and emotions.

Self-regulated learning (SRL), or students actively managing their own learning, involves motivational beliefs, affect, effort, and detecting and evaluating mistakes. Students need feedback to engage in self-directed learning; scalable educational technology tools that offer formative assessment can allow students to evaluate their progress and adjust their strategies for effective math problem-solving without the need for teachers to provide feedback, fostering SRL at scale. This qualitative study examined whether a widely-used formative assessment math technology platform, ASSISTments, improved U.S. 11--14-year-old students' SRL. This study occurred in the context of a nationwide impact study examining the effects of ASSISTments with a professional learning community for teachers on students' math achievement. We used interview data from 33 teachers across two cohorts collected as part of the impact study to understand teachers' perspectives on their students' SRL development. Findings revealed that most teachers felt that using ASSISTments improved their students' accountability over their work, effort while solving math problems, motivation for math, and positive beliefs about the importance of making mistakes. This study leveraged teacher perspectives to provide insights into potential effects of interventions that may otherwise not be captured, highlighting a potential mechanism through which students' SRL can be improved nationwide.

AI technology-based educational programs have been shown to support students' learning. However, to address infrastructure, operational, sustainability, and training barriers when scaling interventions to schools, districts, institutions, and states, cost effectiveness should be considered as part of design, development, and implementation. We consider the cost of the ASSISTments intervention during a large-scale implementation as a part of an efficacy study. ASSISTments is an AI-enhanced educational tool that allows teachers to assign math work and provides students with supports for learning. The intervention was implemented in U.S. 7th grade (age 12--13 years) math classes with 4000+ students, and the long-term effects on math learning were measured one year later at the end of 8th grade (age 13--14 years). To estimate the implementation cost and contextualize the detected effects, we conducted a cost analysis by: (a) identifying the ''ingredients'' or components required, (b) determining the costs of these components, and (c) calculating the total program cost and the average cost per participant. The estimated cost is about 46.23 per student for an average long-term effect size of 0.10, placing ASSISTments at the lower end of cost relative to other interventions. Cost-effectiveness should be treated as an essential design and evaluation parameter from the earliest stages of development to ensure that AI-enabled technologies can reach students at scale in a fiscally responsible manner.

research-article Share on Student Effort and Progress Learning Analytics Data Inform Teachers' SEL Discussions in Math Class Authors: Natalie Brezack WestEd, USA WestEd, USA 0000-0001-5236-9721View Profile , Wynnie Chan WestEd, USA WestEd, USA 0009-0001-9851-192XView Profile , Mingyu Feng WestEd, USA WestEd, USA 0000-0001-9635-1611View Profile Authors Info & Claims LAK '24: Proceedings of the 14th Learning Analytics and Knowledge ConferenceMarch 2024Pages 338–348https://doi.org/10.1145/3636555.3636888Published:18 March 2024Publication History 0citation41DownloadsMetricsTotal Citations0Total Downloads41Last 12 Months41Last 6 weeks22 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Publisher SiteGet Access

Many districts in the U.S. are investing in education technologies to improve student learning. Yet, when technologies with established promise of evidence are deployed at scale, they frequently encounter challenges that compromise their efficacy. MathSpring is a technology-based math learning platform that offers personalized content, remedial tutoring, and affective support for students and reports for teachers. In a pilot study involving monthly usage with a researcher in the room, MathSpring showed promise of improving student learning. Following the pilot study, our team examined the efficacy of the intervention in a randomized controlled trial (RCT) with 64 fifth- and sixth-grade math teachers (34 treatment, 30 control) in 47 schools. Despite extensive training and support for teachers, usage was lower than expected. This paper presents an overview of the intervention and explores the challenges teachers faced to implement MathSpring. We discuss the factors that influence wide classroom adoption of technologies like MathSpring in a post-pandemic educational landscape.

Technologies used by teachers and students generate vast amounts of data that can be analyzed to provide insights into improving teaching and learning. However, practitioners are left out of the process. We describe the development of an approach by which researchers and practitioners can work together to use data intensive research methods to launch improvement efforts within schools. This paper describes elements of the first year of a researcher-practitioner partnership, highlighting initial findings, challenges, and strategies for overcoming these challenges.

How students do homework has been under-researched relative to classroom learning because it is more difficult to collect data on students' homework behaviors. Presumably, such data would have implications for students' achievement. To understand how students do homework and how homework performance and behaviors relate to end-of-year standardized test scores, we analyzed the system logs from an online homework support platform used by more than 1,500 seventh-grade students in Maine.

This paper argues that improving learning reliably and at scale requires a specific orientation toward measurement, understood broadly. Drawing on examples from a partnership between SRI International and The Carnegie Foundation for the Advancement of Teaching, this paper describes measures of student behaviors that are being used by researchers and instructors to improve learning environments at more than 50 community colleges and four-year universities for thousands of students.

A basic question of instruction is how effective it is in promoting student learning. This paper presents a study determining the relative efficacy of different instructional content by applying an educational data mining technique, learning decomposition. We use logistic regression to determine how much learning caused by different methods of presenting same skill, relative to each other. We analyze more than 60,000 performance data across 181 items from more than 2,000 students. Our results show that items are not all as effective on promoting student learning. We also did preliminary study on validating our results by comparing them with rankings from human experts. Our study demonstrates an easier and quicker approach of evaluating the quality of ITS contents than experimental studies.

Secondary teachers across the country are being asked to use formative assessment data to inform their classroom instruction. At the same time, critics of No Child Left Behind are calling the bill "No Child Left Untestedö emphasizing the negative side of assessment, in that every hour spent assessing students is an hour lost from instruction. Or does it have to be? What if we better integrated assessment into the classroom, and we allowed students to learn during the test? Maybe we could even provide tutoring on the steps of solving problems. Our hypothesis is that we can achieve more accurate assessment by not only using data on whether students get test items right or wrong, but by also using data on the effort required for students to learn how to solve a test item. We provide evidence for this hypothesis using data collected with our E-ASSISTment system by more than 600 students over the course of the 2004-2005 school year. We also show that we can track student knowledge over time using modern longitudinal data analysis techniques. In a separate paper [9], we report on the ASSISTment system's architecture and scalability, while this paper is focused on how we can reliably assess student learning.