Formative practice embedded in textbooks has been shown to substantially enhance learning, yet manually authoring high-quality questions at scale is prohibitive. Recent advances in automatic question generation (AQG) have enabled large-scale production of formative practice questions. This study investigates whether a large language model (LLM) can improve selection of textbook sentences for fill-in-the-blank cloze questions by identifying those that lead to higher rates of negative student feedback. A larger LLM was employed to expedite prompt engineering for sentence classification by a smaller model. Over 1.3 million student-question sessions spanning 2,500 textbooks were analyzed using an explanatory logistic regression model. Questions derived from sentences the filter rejects were over three times as likely to receive a thumbs down rating by students, even after controlling for previously established causal factors. This finding indicates that an LLM-based filter can be integrated into existing rule-based AQG pipelines to remove flawed items and raise overall question quality in large-scale educational applications with minimal additional overhead.

Decades of research have focused on the feedback delivered to students after answering questions—when to deliver feedback and what kind of feedback is most beneficial for learning. While there is a well-established body of research on feedback, new advances in technology have led to new methods for developing feedback and large-scale usage provides new data for understanding how feedback impacts learners. This paper focuses on feedback that was developed using artificial intelligence for an automatic question generation system. The automatically generated questions were placed alongside text as a formative learning tool in an e-reader platform. Three types of feedback were randomized across the questions: outcome feedback, context feedback, and common answer feedback. In this study, we investigate the effect of different feedback types on student behavior. This analysis is significant to the expanding body of research on automatic question generation, as little research has been reported on automatically generated feedback specifically, as well as the additional insights that microlevel data can reveal on the relationship between feedback and student learning behaviors.

The future of digital learning should be focused on methods proven to be effective by learning science and learning analytics. One such method is learning by doing—combining formative practice with expository content so students actively engage with their learning resource. This generates the doer effect: the principle that students who do practice while they read have higher outcomes than those who only read [9]. Research on the doer effect has shown it to be causal to learning [10], and these causal findings have previously been replicated in a single course [19]. This study extends the replication of the doer effect by analyzing 15.2 million data events from 18,546 students in seven courses at an online higher education institution, the most students and courses known to date. Furthermore, we analyze each course five ways by using different outcomes, accounting for prior knowledge, and doing both correlational and causal analyses. By performing the doer effect analyses five ways on seven courses, new insights are gained on how this method of learning analytics can contribute to our interpretation of this learning science principle. Practical implications of the doer effect for students are discussed, and future research goals are established.

Advances in artificial intelligence and automatic question generation (AQG) have made it possible to generate the volume of formative practice questions needed to engage students in learning by doing. These automatically generated (AG) questions can be integrated with textbook content in a courseware environment so that students can practice as they read. Scaling this learn by doing method is a valuable pursuit, as it is proven to cause better learning outcomes (i.e., the doer effect). However, it is also necessary to ensure these AG questions perform equally as well as human-authored (HA) questions. In previous studies, it was found that AG and HA questions were essentially equivalent with respect to student engagement, difficulty, and persistence. While these question performance metrics expanded existing AQG research, this paper further extends this research by evaluating question discrimination using student data from a university Neuroscience course. It is found that the AG questions also perform as well as HA questions with respect to discrimination.

Courseware is a comprehensive learning environment that engages students in a learning by doing approach while also giving instructors data-driven insights on their class, providing a scalable solution for many instructional models. However, courseware-and the volume of formative questions required to make it effective-is time-consuming and expensive to create. By using artificial intelligence for automatic question generation, we can reduce the time and cost of developing formative questions in courseware. However, it is critical that automatically generated (AG) questions have a level of quality on par with human-authored (HA) questions in order to be confident in their usage at scale. Therefore, our research question is: are student interactions with AG questions equivalent to HA questions with respect to engagement, difficulty, and persistence metrics? This paper evaluates data for AG and HA questions that students used as formative practice in their university Communication course. Analysis of AG and HA questions shows that our first generation of AG questions perform equally well as HA questions in multiple important respects.