Estimating the difficulty of multiple-choice questions would be great help for educators who must spend substantial time creating and piloting stimuli for their tests, and for learners who want to practice. Supervised approaches to difficulty estimation have yielded to date mixed results. In this contribution we leverage an aspect of generative large models which might be seen as a weakness when answering questions, namely their uncertainty. Specifically, we exploit model uncertainty towards exploring correlations between two different metrics of uncertainty, and the actual student response distribution. While we observe some present but weak correlations, we also discover that the models’ behaviour is different in the case of correct vs wrong answers, and that correlations differ substantially according to the different question types which are included in our fine-grained, previously unused dataset of 451 questions from a Biopsychology course. In discussing our findings, we also suggest potential avenues to further leverage model uncertainty as an additional proxy for item difficulty.

Multiple-choice questions (MCQs) are popular among learners, but are often criticized for emphasizing recognition over active recall, which makes them less effective for long-term memory retention. Well-designed competitive distractors that are similar to the correct answer in meaning (semantically related) or wordform (orthographically related) can prompt deeper cognitive processing and overcome this deficit. However, creating high-quality distractors traditionally requires a significant time investment and domain expertise. Recent advances in artificial intelligence enable the generation of distractors at scale, but are insufficient when it comes to creating distractors that specifically target learners' misconceptions about word forms. In this study, we present a scalable, data-driven method for automatically generating orthographically related distractors, based on common incorrect responses from open-answer retrieval practice, supplemented with rule-based generation of common misspellings where necessary. We apply this method to a large dataset of learners' errors in vocabulary learning, demonstrating that it is feasible to create distractors that align with learners' shared misconceptions in a real-world setting. This work contributes to making MCQs a more effective pedagogical tool.

Despite the importance of memories in everyday life and the progress made in understanding how they are encoded and retrieved, the neural processes by which declarative memories are maintained or forgotten remain elusive. Part of the problem is that it is empirically difficult to measure the rate at which memories fade, even between repeated presentations of the source of the memory. Without such a ground-truth measure, it is hard to identify the corresponding neural correlates. This study addresses this problem by comparing individual patterns of functional connectivity against behavioral differences in forgetting speed derived from computational phenotyping. Specifically, the individual-specific values of the speed of forgetting in long-term memory (LTM) were estimated for 33 participants using a formal model fit to accuracy and response time data from an adaptive paired-associate learning task. Individual speeds of forgetting were then used to examine participant-specific patterns of resting-state fMRI connectivity, using machine learning techniques to identify the most predictive and generalizable features. Our results show that individual speeds of forgetting are associated with resting-state connectivity within the default mode network (DMN) as well as between the DMN and cortical sensory areas. Cross-validation showed that individual speeds of forgetting were predicted with high accuracy (r = .77) from these connectivity patterns alone. These results support the view that DMN activity and the associated sensory regions are actively involved in maintaining memories and preventing their decline, a view that can be seen as evidence for the hypothesis that forgetting is a result of storage degradation, rather than of retrieval failure.

Memorizing vocabulary is a crucial aspect of learning a new language. While personalized learning- or intelligent tutoring systems can assist learners in memorizing vocabulary, the majority of such systems are limited to typing-based learning and do not allow for speech practice. Here, we aim to compare the efficiency of typing- and speech based vocabulary learning. Furthermore, we explore the possibilities of improving such speech-based learning using an adaptive algorithm based on a cognitive model of memory retrieval. We combined a response time-based algorithm for adaptive item scheduling that was originally developed for typing-based learning with automatic speech recognition technology and tested the system with 50 participants. We show that typing- and speech-based learning result in similar learning outcomes and that using a model-based, adaptive scheduling algorithm improves recall performance relative to traditional learning in both modalities, both immediately after learning and on follow-up tests. These results can inform the development of vocabulary learning applications that–unlike traditional systems–allow for speech-based input.

Abstract Adaptive learning systems offer a personalised digital environment that continually adjusts to the learner and the material, with the goal of maximising learning gains. Whenever such a system encounters a new learner, or when a returning learner starts studying new material, the system first has to determine the difficulty of the material for that specific learner. Failing to address this “cold-start” problem leads to suboptimal learning and potential disengagement from the system, as the system may present problems of an inappropriate difficulty or provide unhelpful feedback. In a simulation study conducted on a large educational data set from an adaptive fact learning system (about 100 million trials from almost 140 thousand learners), we predicted individual learning parameters from response data. Using these predicted parameters as starting estimates for the adaptive learning system yielded a more accurate model of learners’ memory performance than using default values. We found that predictions based on the difficulty of the fact (“what”) generally outperformed predictions based on the ability of the learner (“who”), though both contributed to better model estimates. This work extends a previous smaller-scale laboratory-based experiment in which using fact-specific predictions in a cold-start scenario improved learning outcomes. The current findings suggest that similar cold-start alleviation may be possible in real-world educational settings. The improved predictions can be harnessed to increase the efficiency of the learning system, mitigate the negative effects of a cold start, and potentially improve learning outcomes.

Memorising vocabulary is an important aspect of formal foreign language learning. Advances in cognitive psychology have led to the development of adaptive learning systems that make vocabulary learning more efficient. These computer-based systems measure learning performance in real time to create optimal study strategies for individual learners. While such adaptive learning systems have been successfully applied to written word learning, they have thus far seen little application in spoken word learning. Here we present a system for adaptive, speech-based word learning. We show that it is possible to improve the efficiency of speech-based learning systems by applying a modified adaptive model that was originally developed for typing-based word learning. This finding contributes to a better understanding of the memory processes involved in speech-based word learning. Furthermore, our work provides a basis for the development of language learning applications that use real-time pronunciation assessment software to score the accuracy of the learner’s pronunciations. Speech-based learning applications are educationally relevant because they focus on what may be the most important aspect of language learning: to practice speech.

In this paper we present a computational cognitive model of task interruption and resumption, focusing on the effects of the problem state bottleneck. Previous studies have shown that the disruptiveness of interruptions is for an important part determined by three factors: interruption duration, interrupting-task complexity, and moment of interruption. However, an integrated theory of these effects is still missing. Based on previous research into multitasking, we propose a first step towards such a theory in the form of a process model that attributes these effects to problem state requirements of both the interrupted and the interrupting task. Subsequently, we tested two predictions of this model in two experiments. The experiments confirmed that problem state requirements are an important predictor for the disruptiveness of interruptions. This suggests that interfaces should be designed to a) interrupt users at low-problem state moments and b) maintain the problem state for the user when interrupted.