Learning engineering applies data and learning science principles to better understand outcomes and support improvement research. One important approach is A/B testing-common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE), and the International Consortium for Innovation and Collaboration in Learning Engineering (IEEE ICICLE). Several systems supporting A/B testing in educational applications have arisen recently, including UpGrade, E-TRIALS, and Terracotta. A/B testing can help improve educational platforms, yet there are challenging issues unique to conducting such work in these contexts. In response, a number of digital learning platforms have opened their systems to learning-improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore how A/B testing is conducted in educational contexts, how digital learning platforms are accelerating education research, and how empirical approaches can be used to drive powerful gains in student learning. It will also discuss opportunities for funding to conduct platform-enabled learning engineering.

High-quality conversational datasets are crucial for the successful development of Intelligent Tutoring Systems (ITS) that utilize a Large Language Model (LLM) backend. Synthetic student-teacher dialogues, generated using advanced GPT-4 models, are a common strategy for creating these datasets. However, subjects like physics that entail complex calculations pose a challenge. While GPT-4 presents impressive language processing capabilities, its limitations in fundamental mathematical reasoning curtail its efficacy for such subjects. To tackle this limitation, we introduce in this paper an innovative stateful prompt design. Our design orchestrates a mock conversation where both student and tutorbot roles are simulated by GPT-4. Each student response triggers an internal monologue, or ‘code soliloquy’ in the GPT-tutorbot, which assesses whether its subsequent response would necessitate calculations. If a calculation is deemed necessary, it scripts the relevant Python code and uses the Python output to construct a response to the student. Our approach notably enhances the quality of synthetic conversation datasets, especially for subjects that are calculation-intensive. The preliminary Subject Matter Expert evaluations reveal that our Higgs model, a fine-tuned LLaMA model, effectively uses Python for computations, which significantly enhances the accuracy and computational reliability of Higgs’ responses.

Learning engineering adds tools and processes to learning platforms to support improvement research. One kind of tool is A/B testing-common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE), and the International Consortium for Innovation and Collaboration in Learning Engineering (IEEE ICICLE). Recently, several A/B testing systems have arisen that focus on conducting research in educational environments, including UpGrade, Terracotta, and E-TRIALS. A/B testing can help improve educational platforms, yet there are challenging issues unique to conducting such work in these contexts. In response, a number of digital learning platforms have opened their systems to learning-improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore challenges of A/B testing in educational contexts, how learning platforms are accelerating education research, and how empirical approaches can be used to drive powerful gains in student learning. It will also discuss opportunities for funding to conduct platform-enabled learning research.

Greater financial literacy is critically needed among young adults in the United States [10,35], but many financial literacy education courses have been less effective than hoped for by educators and researchers [7,15]. Additionally, many have not been designed around established curricula or learning science principles, rendering findings difficult for researchers to study empirically [6,34]. In order to better understand the psychosocial mechanisms that predict success in improving learner knowledge and behavior, online educational interventions at scale can be an effective path forward. We conducted interviews with subject matter experts and young adult students to explore the highest priority learning objectives for a brief course curriculum to improve the financial literacy of US young adults. We then leveraged our findings from this study and content from our open-source textbooks to develop the first of several brief online learning interventions for deployment on the large-scale OpenStax Kinetic research infrastructure [2]. In this work-in-progress paper, we discuss the next steps in our research agenda, including course content development and deploying this intervention, as well as our broader plans for our future financial literacy education interventions and translating research into practice with our institutional collaborations.

OpenStax Kinetic is an innovative research infrastructure that aims to transform education and learning research in the digital age. With its access to large sample sizes, authentic learning environments, experimental control, scalability, security and privacy protection, Kinetic provides an unparalleled opportunity for researchers to study the complex interactions between different factors in digital learning environments. This versatile platform utilizes Qualtrics and can support various research designs, including correlational, longitudinal, and interventional studies. Kinetic's unique privacy-by-design implementation via secure enclaves ensures that researchers can analyze fully-identified data without compromising data security and privacy as well as affords greater analytical reproducibility. The findings from Kinetic can inform educational interventions and strategies to enhance student success in digital learning environments. Kinetic has the potential to significantly advance education and learning research by improving pedagogies, practices, and policies in education and learning sciences. In this demo of Kinetic, researchers will be able to interact with the test instance of the Kinetic system online and view the learner experience. All researchers will be able to engage in the experience of creating a study, releasing a study, and interacting with our implementation of secure enclaves for data analysis.

Learning engineering adds tools and processes to learning platforms to support improvement research. One kind of tool is A/B testing, which is common in large software companies and also represented academically at conferences like the Annual Conference on Digital Experimentation (CODE). A number of A/B testing systems focused on educational applications have arisen recently, including UpGrade and E-TRIALS. A/B testing can be part of the puzzle of how to improve educational platforms, and yet challenging issues in education go beyond the generic paradigm. For example, the importance of teachers and instructors to learning means that students are not only connecting with software as individuals, but also as part of a shared classroom experience. Further, learning in topics like mathematics can be highly dependent on prior learning, and thus A or B may not be better overall, but only in interaction with prior knowledge. In response, a set of learning platforms is opening their systems to improvement research by instructors and/or third-party researchers, with specific supports necessary for education-specific research designs. This workshop will explore how A/B testing in educational contexts is different, how learning platforms are opening up new possibilities, and how these empirical approaches can be used to drive powerful gains in student learning. It will also discuss forthcoming opportunities for funding to conduct platform-enabled learning research.