Learning experience designers consistently balance the trade-off between open and close-ended activities. The growth and scalability of Computer Based Learning Platforms (CBLPs) have only magnified the importance of these design trade-offs. CBLPs often utilize close-ended activities (i.e. Multiple-Choice Questions [MCQs]) due to feasibility constraints associated with the use of open-ended activities. MCQs offer certain affordances, such as immediate grading and the use of distractors, setting them apart from open-ended activities. Our current study examines the effectiveness of Fill-In problems as an alternative to MCQs for middle school mathematics. We report on a randomized study conducted from 2017 to 2022, with a total of 6,768 students from middle schools across the US. We observe that, on average, Fill-In problems lead to better post-test performance than MCQs; albeit deeper explorations indicate differences between the two design paradigms to be more nuanced. We find evidence that students with higher math knowledge benefit more from Fill-In problems than those with lower math knowledge.

Notional Machines (NMs) are a pedagogical device used by teachers in order to help students understand certain concepts. While NMs have been cataloged, the effectiveness of NMs has been rarely evaluated. We build upon this research by exploring what makes certain NMs more effective in various computer science and mathematics courses. We interview professors and students to assess NMs used in the classroom. Notably we found that most students are able to employ the NMs introduced by their professors, and that introductory students prefer template-like NMs, whereas upper level students rely on more conceptual NMs.