The science of learning has generated a wealth of research and theory on how people think and learn. One product of this research is what are known as 'principles of learning' - teaching and learning strategies that have garnered empirical support for their effectiveness at enhancing learning across a range of learners and domains in rigorous lab and classroom tests. For example, the principle of interleaved practice involves mixing problems that can be solved using different solutions -- to support discrimination learning - and spacing problems that can be solved using the same solution over time -- to support long-term memory retention [1]. In the present work, we describe an on-going cluster randomized trial where we attempt to test the efficacy of interleaved practice across multiple schools and geographic regions in a nationally representative U.S. sample. Despite strong evidence that interleaved practice leads to greater math learning than blocking problems [2], blocked practice predominates math curricula [3]. This finding follows a general trend that learning research is underutilized in educational contexts [4]. One reason for this disconnect is that learning takes place within a dynamic system that is highly contextual [5]. What works in one context may not work the same way in another. As a result, scaling learning research remains a major challenge. Although interleaving is seemingly straightforward for application, we describe several challenges we faced in our own work when attempting to scale interleaving in this large-scale study. To address these challenges we outline a process guided by a working implementation model for translating interleaving to concrete materials. We also describe early results of our first (of three) data collection years assessing the impact of a mostly interleaving intervention relative to a mostly blocked intervention on students' math achievement. Although our work is ongoing, the lessons learned even at this early stage can have important implications for scaling interleaved practice as well as learning principles more broadly.

Prior work analyzing tutoring sessions provided evidence that highly effective tutors, through their interaction with students and their experience, can perceptively recognize incorrect processes or “bugs” when students incorrectly answer problems. Researchers have studied these tutoring interactions examining instructional approaches to address incorrect processes and observed that the format of the feedback can influence learning outcomes. In this work, we recognize the incorrect answers caused by these buggy processes as Common Wrong Answers (CWAs). We examine the ability of teachers and instructional designers to identify CWAs proactively. As teachers and instructional designers deeply understand the common approaches and mistakes students make when solving mathematical problems, we examine the feasibility of proactively identifying CWAs and generating Common Wrong Answer Feedback (CWAFs) as a formative feedback intervention for addressing student learning needs. As such, we analyze CWAFs in three sets of analyses. We first report on the accuracy of the CWAs predicted by the teachers and instructional designers on the problems across two activities. We then measure the effectiveness of the CWAFs using an intent-to-treat analysis. Finally, we explore the existence of personalization effects of the CWAFs for the students working on the two mathematics activities.

Within the field of education technology, learning analytics has increased in popularity over the past decade. Researchers conduct experiments and develop software, building on each other’s work to create more intricate systems. In parallel, open science — which describes a set of practices to make research more open, transparent, and reproducible — has exploded in recent years, resulting in more open data, code, and materials for researchers to use. However, without prior knowledge of open science, many researchers do not make their datasets, code, and materials openly available, and those that are available are often difficult, if not impossible, to reproduce. The purpose of the current study was to take a close look at our field by examining previous papers within the proceedings of the International Conference on Learning Analytics and Knowledge, and document the rate of open science adoption (e.g., preregistration, open data), as well as how well available data and code could be reproduced. Specifically, we examined 133 research papers, allowing ourselves 15 minutes for each paper to identify open science practices and attempt to reproduce the results according to their provided specifications. Our results showed that less than half of the research adopted standard open science principles, with approximately 5% fully meeting some of the defined principles. Further, we were unable to reproduce any of the papers successfully in the given time period. We conclude by providing recommendations on how to improve the reproducibility of our research as a field moving forward.

As evidence grows supporting the importance of non-cognitive factors in learning, computer-assisted learning platforms increasingly incorporate non-academic interventions to influence student learning and learning related-behaviors. Non-cognitive interventions often attempt to influence students’ mindset, motivation, or metacognitive reflection to impact learning behaviors and outcomes. In the current paper, we analyze data from five experiments, involving seven treatment conditions embedded in mastery-based learning activities hosted on a computer-assisted learning platform focused on middle school mathematics. Each treatment condition embodied a specific non-cognitive theoretical perspective. Over seven school years, 20,472 students participated in the experiments. We estimated the effects of each treatment condition on students’ response time, hint usage, likelihood of mastering knowledge components, learning efficiency, and post-tests performance. Our analyses reveal a mix of both positive and negative treatment effects on student learning behaviors and performance. Few interventions impacted learning as assessed by the post-tests. These findings highlight the difficulty in positively influencing student learning behaviors and outcomes using non-cognitive interventions.

There have been numerous efforts documenting the effects of open science in existing papers; however, these efforts typically only consider the author's analyses and supplemental materials from the papers. While understanding the current rate of open science adoption is important, it is also vital that we explore the factors that may encourage such adoption. One such factor may be publishing organizations setting open science requirements for submitted articles: encouraging researchers to adopt more rigorous reporting and research practices. For example, within the education technology discipline, theACM Conference on Learning @ Scale ([email protected]) has been promoting open science practices since 2018 through a Call For Papers statement. The purpose of this study was to replicate previous papers within the proceedings of [email protected] and compare the degree of open science adoption and robust reproducibility practices to other conferences in education technology without a statement on open science. Specifically, we examined 93 papers and documented the open science practices used. We then attempted to reproduce the results with invitation from authors to bolster the chance of success. Finally, we compared the overall adoption rates to those from other conferences in education technology. Although the overall responses to the survey were low, our cursory review suggests that researchers at [email protected] might be more familiar with open science practices compared to the researchers who published in theInternational Conference on Artificial Intelligence in Education (AIED) and theInternational Conference on Educational Data Mining (EDM): 13 of 28 AIED and EDM responses were unfamiliar with preregistrations and 7 unfamiliar with preprints, while only 2 of 7 [email protected] responses were unfamiliar with preregistrations and 0 with preprints. The overall adoption of open science practices at [email protected] was much lower with only 1% of papers providing open data, 5% providing open materials, and no papers had a preregistration.

Across the past decade, open science has increased in momentum, making research more openly available and reproducible. In addition, learning at scale systems have been developed to collect and apply models, features and reports to better support students and teachers towards their goals. In this tutorial, we will provide an overview of open science practices and their benefits and mitigation within research. In the second part of this tutorial, we will use the Open Science Framework to make, collaborate, and share projects - demonstrating how to make materials, code, and data open. The final part of this tutorial will go over some mitigation strategies when releasing datasets and materials so other researchers may easily reproduce them. Participants in this tutorial learn what the practices of open science are, how to use them in their own research, and how to use the Open Science Framework.