Conventional methods of obtaining student feedback on course experiences face a fundamental tradeoff between feedback frequency and quality: as feedback requests become more frequent, participation often declines, and responses become less thoughtful over time. To obtain both timely and thoughtful feedback from students, Kim and Piech [27] recently proposed a simple, lightweight course feedback mechanism: surveying each student a small number of times per term during randomly selected weeks. Termed High-Resolution Course Feedback (HRCF), this method has been shown to elicit feedback that instructors find helpful without placing excessive survey burden on students.

Mixed-effects models fit to observational practice data are widely used in learning analytics to estimate student-level variation in initial knowledge and learning rate, and the resulting estimates increasingly inform substantive claims about learners. We examine whether such estimates can be read as properties of learners or whether they depend on choices about which observations the model is fit to. As a case study, we revisit the ''astonishing regularity'' reported by Koedinger et al. (2023): that students vary substantially in initial knowledge but much less in learning rate. The finding is based on fits of the individual Additive Factors Model (iAFM) to 27 educational datasets, and rests on a model-derived estimate of student-level learning-rate variation being small in absolute terms. We refit the same model on the same datasets under two specifications, each varying how much of each student's practice on a given skill is used in fitting. The estimate of student-level variation in initial knowledge stays approximately stable across both specifications. The estimate of student-level variation in learning rate does not: it inflates by a median of 118% under one specification and is several times larger under the other. The same model, fit to the same data, returns substantially different estimates of how much students vary in learning rate depending on which observations are included. When estimates from mixed-effects models on observational practice data are used to support substantive claims about learners, sensitivity to such choices deserves a central place in how those estimates are reported and read.

A cognitive model is a descriptive account or computational representation of human thinking about a given concept, skill, or domain. A cognitive model of learning, includes both a way of organizing knowledge within a subject area and an account of how humans develop accurate and complete knowledge of that subject area. Learning designers engage in a variety of practices to unpack knowledge from subject matter experts and novices to develop cognitive models of learning and use those models to guide the design of instruction or instructional technologies. Traditional approaches to eliciting and organizing knowledge, such as conducting a cognitive task analysis (CTA) [10] with experts and novices, are labor-intensive and require specific expertise that many learning designers do not have. However, learning data generated from learners' interaction with the courses, reveal how humans think about and develop knowledge. We propose a novel framework that uses learning data to discover and refine cognitive models of learning. The framework includes a Variational Autoencoder (VAE) module and a Gaussian Mixture Model (GMM) module. We provide one case study in a corporate setting to demonstrate the effectiveness of the proposed framework compared to other approaches.

Adaptive instruction for online education can increase learning gains and decrease the work required of learners, instructors, and course designers. Reinforcement Learning (RL) is a promising tool for developing instructional policies, as RL models can learn complex relationships between course activities, learner actions, and educational outcomes. This paper demonstrates the first RL model to schedule educational activities in real time for a large online course through active learning. Our model learns to assign a sequence of course activities while maximizing learning gains and minimizing the number of items assigned. Using a controlled experiment with over 1,000 learners, we investigate how this scheduling policy affects learning gains, dropout rates, and qualitative learner feedback. We show that our model produces better learning gains using fewer educational activities than a linear assignment condition, and produces similar learning gains to a self-directed condition using fewer educational activities and with lower dropout rates.

Online learning systems with open navigation allow learners to select the next learning activity in order to achieve desired mastery. To help learners make an informed choice regarding the next learning activity, we propose to represent and communicate the learner's knowledge state as the average success rate in the course for each skill, rather than as the probability of correctly answering the next question. We first show that we can accurately estimate the proposed knowledge state. We then show that the proposed attention-based model to estimate the knowledge state requires fewer parameters, provides actionable information to the learners, and achieves equivalent or better accuracy compared to RNN (Recurrent Neural Network) based models.

Open navigation online learning systems allow learners to choose the next learning activity. These systems can be instrumented to provide learners with feedback to help them choose the next learning activity. One type of feedback is providing an estimate of the learner's current skill proficiency. A learner can then choose to skip the remaining learning activities for that skill after achieving proficiency in that skill. In this paper, we investigate whether predicting proficiency and communicating it to learners can save time for learners within a course. We evaluate the accuracy of the BKT based proficiency pre- diction framework for learner's proficiency prediction which considers one attempt per question. We extend the proficiency prediction framework to include multiple attempts at individual questions and show that it is more accurate in proficiency prediction than BKT based proficiency prediction framework. We discuss the potential implications of attempt enhanced framework on the learners' behavior for open navigation on- line learning systems.

Deep learning based knowledge tracing approaches achieve high accuracy in mastery prediction with pattern extraction on a large learning behavior data set. However, when there is little training data available, these approaches either fail to extract the key patterns or result in over fitting. Ideally, we aim to provide a similar learning experience to both the first group of learners, who interact with a new course or a new activity with little learning behavior data to provide personalized guidance, and the learners who interact with the course later. We propose a novel architecture, Attentive Neural Turing Machine (ANTM), to solve the cold start knowledge tracing problem. The proposed ANTM comprises an attentive controller module and differential reading and writing processes with extra memory bank. Accuracy (ACC) and Area Under Curve (AUC) measures are used for model performance comparison. Results show the proposed approach can learn fast and generalize well to unseen data. It achieves around 95% ACC trained with only 3 learners, while conventional deep learning based approaches achieve only 65% ACC with over prediction issues.

Online learning systems that provide actionable and personalized guidance can help learners make better decisions during learning. Bayesian Knowledge Tracing (BKT) extensions and deep learning based approaches have demonstrated improved mastery prediction accuracy compared to the basic BKT model; however, neither set of models provides actionable guidance on learning activities beyond mastery prediction. We propose a novel framework for personalized knowledge tracing with attention mechanism. Our proposed framework incorporates auxiliary learner attributes into knowledge tracing and interprets mastery prediction with the learning attributes. The proposed approach can also provide personalized next best learning activity recommendations. We demonstrate that the accuracy of the proposed approach in mastery prediction is slightly higher compared to deep learning based approaches and that the proposed approach can provide personalized next best learning activity recommendation.

E-learning is becoming popular as it provides learners the flexibility, targeted resources across the internet, personalized guidance, and immediate feedback during learning. However, lack of social interaction, an indispensable component in developing some skills, has been a pain point in e-learning. We propose using Alexa, a voice-controlled Intelligent Personal Assistants (IPA), in e-learning to provide in-person practice to achieve some desired learning goals. With Alexa enabled learning experiences, learners are able to practice with other students (one role of Alexa) or receive immediate feedback from teachers (another role of Alexa) in an e-learning environment. We propose a configuration driven conversation engine, which can support instructional designers to create diverse in-person practice opportunities in e-learning. We demonstrate that learning designers can create an Alexa activity with a few configuration steps. We also share results on the effectiveness of an Alexa activity with formative assessment evaluation in real world applications.

Learning analytics systems have the potential to bring enormous value to online education. Unfortunately, many instructors and platforms do not adequately leverage learning analytics in their courses today. In this paper, we report on the value of these systems from the perspective of course instructors. We study these ideas through OARS, a modular and real-time learning analytics system that we deployed across more than ten online courses with tens of thousands of learners. We leverage this system as a starting point for semi-structured interviews with a diverse set of instructors. Our study suggests new design goals for learning analytics systems, the importance of real-time analytics to many instructors, and the value of flexibility in data selection and aggregation for an instructor when working with an analytics system.

Engagement and motivation are particularly important in optional learning environments, like educational games and massive open online courses. Providing some aspects of autonomy and choice to the student can yield significant benefits to learner motivation and persistence; yet there is also evidence that unsupported learners may not always automatically choose to allocate their learning time to pedagogical activities that are most known to be as associated with better learning outcomes. We investigated the impact of choice on student engagement and learning in a Massive Open Online Course (MOOC) on introductory statistics and probability. We compared conditions in which students are given free choice over the practice problems completed to conditions in which students receive a full set of practice activities or no practice activities before completing a post-test. In all cases students were free to navigate to other sections of the course at any time. In one of the two topic sections that included personalized practice activities we found that students performed better in the condition in which they were prompted to complete all practice activities. Though more students in this condition dropped out before reaching the post-test, many more students completed the full set of practice activities in this section than those who did in the free choice condition. These results are still quite preliminary but suggest that providing a default encouraged opt in procedure can encourage students to do more problems than they would otherwise, and that doing such additional problems can yield learning gains.

This workshop will explore community based repositories for educational data and analytic tools that are used to connect researchers and reduce the barriers to data sharing. Leading innovators in the field, as well as attendees, will identify and report on bottlenecks that remain toward our goal of a unified repository. We will discuss these as well as possible solutions. We will present LearnSphere, an NSF funded system that supports collaborating on and sharing a wide variety of educational data, learning analytics methods, and visualizations while maintaining confidentiality. We will then have hands-on sessions in which attendees have the opportunity to apply existing learning analytics workflows to their choice of educational datasets in the repository (using a simple drag-and-drop interface), add their own learning analytics workflows (requires very basic coding experience), or both. Leaders and attendees will then jointly discuss the unique benefits as well as the limitations of these solutions. Our goal is to create building blocks to allow researchers to integrate their data and analysis methods with others, in order to advance the future of learning science.