This conceptual paper develops responsible AI principles for educational technology and shows how a learning engineering process can make those principles operational rather than aspirational. It connects guidance on accountability, transparency, efficacy, human oversight, and ethical use to two development examples: a rule-based automatic question generation system and LLM-supported personalized feedback. The paper argues that learning engineering supplies the research grounding, documentation, evaluation, and iterative oversight needed to develop AI-enabled learning tools responsibly.

The doer effect is the learning science principle that doing formative practice while reading in a ''learn by doing'' approach is more effective for learning than reading alone. Research on the doer effect in higher education has found doing integrated practice to be an average of six times more effective for learning, and the relationship is causal. Replication research confirming these findings has contributed to the generalizability of this learn by doing method and validated the need to increase the availability of formative practice for students. In this paper, we investigate the doer effect in a novel way-using formative practice generated through artificial intelligence (AI). While research on automatically generated questions has established performance benchmarks and ensured their validity as practice items, could these generated items also engage the doer effect? This study uses a novel data set of student exam scores combined with microlevel reading and doing engagement data collected from two sections of a Cognitive Psychology course. These data are used to follow the established correlational doer effect analysis to examine if the results can be reproduced in this context. This study also contributes a rare opportunity to review natural learning contexts from a holistic data perspective; how course policies impact student engagement, how student engagement impacts exam scores, and how the learning impact of doing practice can be measured by the doer effect. Future implications for both teaching and learning with AI-generated practice and the scalability of the doer effect are discussed.

CoachMe is an AI-powered feature in the VitalSource Bookshelf platform that transforms reading into an interactive ''learn by doing'' experience. By delivering formative questions aligned with textbook content, CoachMe leverages the well-established doer effect-the learning science principle that doing practice while reading enhances learning more than reading alone [1][2]. Our accompanying paper, ''Scaling the Doer Effect: A Replication Analysis Using AI-Generated Questions,'' provides the first replication of the doer effect using AI-generated questions in a natural classroom context [3]. Conducted in two semesters of a Cognitive Psychology course, the study compared the impact of different course policies on student engagement, found higher exam scores when AI-generated practice questions were completed at higher rates, and replicated the doer effect analysis with findings consistent with the literature [3][2][4]. This demonstration allows attendees to experience CoachMe from a student's perspective. Participants will engage with AI-generated questions alongside an OER textbook title, receive immediate feedback, and explore how implementation strategies (e.g., completion-based assignment) affect learning outcomes. The demo also highlights the implementation approaches available to instructors. Attendees will walk away with a deeper understanding of how learning science principles, when paired with scalable AI, can be embedded directly into textbooks to drive measurable improvements in student learning.

Formative practice embedded in textbooks has been shown to substantially enhance learning, yet manually authoring high-quality questions at scale is prohibitive. Recent advances in automatic question generation (AQG) have enabled large-scale production of formative practice questions. This study investigates whether a large language model (LLM) can improve selection of textbook sentences for fill-in-the-blank cloze questions by identifying those that lead to higher rates of negative student feedback. A larger LLM was employed to expedite prompt engineering for sentence classification by a smaller model. Over 1.3 million student-question sessions spanning 2,500 textbooks were analyzed using an explanatory logistic regression model. Questions derived from sentences the filter rejects were over three times as likely to receive a thumbs down rating by students, even after controlling for previously established causal factors. This finding indicates that an LLM-based filter can be integrated into existing rule-based AQG pipelines to remove flawed items and raise overall question quality in large-scale educational applications with minimal additional overhead.

Decades of research have focused on the feedback delivered to students after answering questions—when to deliver feedback and what kind of feedback is most beneficial for learning. While there is a well-established body of research on feedback, new advances in technology have led to new methods for developing feedback and large-scale usage provides new data for understanding how feedback impacts learners. This paper focuses on feedback that was developed using artificial intelligence for an automatic question generation system. The automatically generated questions were placed alongside text as a formative learning tool in an e-reader platform. Three types of feedback were randomized across the questions: outcome feedback, context feedback, and common answer feedback. In this study, we investigate the effect of different feedback types on student behavior. This analysis is significant to the expanding body of research on automatic question generation, as little research has been reported on automatically generated feedback specifically, as well as the additional insights that microlevel data can reveal on the relationship between feedback and student learning behaviors.

Faculty often suspect that students are not doing the assigned textbook reading-a suspicion supported by research. This can pose a problem for both student learning and outcomes, especially in cases where the textbook is intended to be the primary source of learning content that students are expected to use before class lectures, activities, discussions, etc. In a recent study, researchers gathered a variety of faculty strategies for promoting reading engagement and measured the resulting reading from digital platform data. This study provided a useful benchmark for continued research on this topic of student engagement with their textbooks. In this work-in-progress paper, we replicate the methods of this study to investigate how assigning formative practice in their etextbook reader as a strategy impacted student reading behaviors. Initial results from 13 courses at two universities reveal using formative practice as a strategy dramatically increases student reading compared to courses where it was not assigned, and this strategy yielded higher average reading than strategies reported in previous research.

The future of digital learning should be focused on methods proven to be effective by learning science and learning analytics. One such method is learning by doing—combining formative practice with expository content so students actively engage with their learning resource. This generates the doer effect: the principle that students who do practice while they read have higher outcomes than those who only read [9]. Research on the doer effect has shown it to be causal to learning [10], and these causal findings have previously been replicated in a single course [19]. This study extends the replication of the doer effect by analyzing 15.2 million data events from 18,546 students in seven courses at an online higher education institution, the most students and courses known to date. Furthermore, we analyze each course five ways by using different outcomes, accounting for prior knowledge, and doing both correlational and causal analyses. By performing the doer effect analyses five ways on seven courses, new insights are gained on how this method of learning analytics can contribute to our interpretation of this learning science principle. Practical implications of the doer effect for students are discussed, and future research goals are established.

Advances in artificial intelligence and automatic question generation (AQG) have made it possible to generate the volume of formative practice questions needed to engage students in learning by doing. These automatically generated (AG) questions can be integrated with textbook content in a courseware environment so that students can practice as they read. Scaling this learn by doing method is a valuable pursuit, as it is proven to cause better learning outcomes (i.e., the doer effect). However, it is also necessary to ensure these AG questions perform equally as well as human-authored (HA) questions. In previous studies, it was found that AG and HA questions were essentially equivalent with respect to student engagement, difficulty, and persistence. While these question performance metrics expanded existing AQG research, this paper further extends this research by evaluating question discrimination using student data from a university Neuroscience course. It is found that the AG questions also perform as well as HA questions with respect to discrimination.

Courseware is a comprehensive learning environment that engages students in a learning by doing approach while also giving instructors data-driven insights on their class, providing a scalable solution for many instructional models. However, courseware-and the volume of formative questions required to make it effective-is time-consuming and expensive to create. By using artificial intelligence for automatic question generation, we can reduce the time and cost of developing formative questions in courseware. However, it is critical that automatically generated (AG) questions have a level of quality on par with human-authored (HA) questions in order to be confident in their usage at scale. Therefore, our research question is: are student interactions with AG questions equivalent to HA questions with respect to engagement, difficulty, and persistence metrics? This paper evaluates data for AG and HA questions that students used as formative practice in their university Communication course. Analysis of AG and HA questions shows that our first generation of AG questions perform equally well as HA questions in multiple important respects.

Although content creators, instructors, and students alike see the value in providing interactive courseware to help students learn, they remain a costly solution to create. Platforms that allow the creation and delivery of interactive courseware are only fully utilized by those who can afford the labor cost of building hundreds of formative items just to start. Most of the time this means barriers are too high in either financial cost or extremely long development timelines. Our solution is called SmartStart, a process that works to reduce these barriers to creating courseware by automating basic steps that otherwise require significant manual labor. Automatic question generation (AQG) is one in a series of steps in the SmartStart process that work together to transform textbook content into a courseware learning environment.