Knowledge tracing (KT) is a problem of modeling students’ knowledge states to predict their future performance by observing their historical learning interactions. The collection of educational data presents significant challenges, as students’ limited learning engagement restricts the generation of large-scale interaction data, while stringent privacy regulations further limit the availability of student learning sequences from online platforms. Hence, it is crucial to enhance the capabilities of deep learning-based KT (DLKT) models by constructing large-scale datasets through the integration of student interaction data across multiple subjects and sources. The success of ChatGPT demonstrates that the decoder-only Transformer architecture is highly effective in capturing complex information from large-scale sequential data. Against this background, we propose a novel decoder-only Transformer architecture-based model, named Unified DLKT ( UniKT ), to learn coherent and unified representations across a wide range of data sources. Specifically, we combine student learning sequences from six educational scenarios and utilize a multi-source encoding to learn unified representations of interactions from mixed data. UniKT is a stack of Transformer decoder layers for handling long-term dependencies among students’ historical interactions and future performance. We evaluate UniKT on six publicly available real-world educational datasets, and experimental results demonstrate that our method outperforms the majority of existing DLKT models in terms of AUC and accuracy. Furthermore, the empirical analysis shows the strong transferability and adaptability of UniKT in learning from multiple sources. To encourage reproducible research, we make our data and code publicly available at https://pykt.org/ .

Accurately judging student on-going performance is crucial for adaptive teaching. In this work, we focus on the task of automatically predicting students' levels of mastery of math questions from teacher-student classroom dialogue data in online one-on-one classes. As a step toward this direction, we introduce the Multi-turn Classroom Dialogue (MCD) dataset as a benchmark testing the capabilities of machine learning models in classroom conversation understanding of student performance judgment. Our dataset contains aligned multi-turn spoken language of 5000+ unique samples of solving grade-8 math questions collected from 500+ hours' worth of online one-on-one tutoring classes. In our experiments, we assess various state-of-the-art models on the MCD dataset, highlighting the importance of understanding multi-turn dialogues and handling noisy ASR transcriptions. Our findings demonstrate the dataset's utility in advancing research on automated student performance assessment. To encourage reproducible research, we make our data publicly available at https://github.com/ai4ed/MCD.

Knowledge tracing (KT) is a crucial technique to predict students’ future performance by observing their historical learning processes. Due to the powerful representation ability of deep neural networks, remarkable progress has been made by using deep learning techniques to solve the KT problem. The majority of existing approaches rely on the homogeneous question assumption that questions have equivalent contributions if they share the same set of knowledge components. Unfortunately, this assumption is inaccurate in real-world educational scenarios. Furthermore, it is very challenging to interpret the prediction results from the existing deep learning based KT models. Therefore, in this paper, we present QIKT, a question-centric interpretable KT model to address the above challenges. The proposed QIKT approach explicitly models students’ knowledge state variations at a fine-grained level with question-sensitive cognitive representations that are jointly learned from a question-centric knowledge acquisition module and a question-centric problem solving module. Meanwhile, the QIKT utilizes an item response theory based prediction layer to generate interpretable prediction results. The proposed QIKT model is evaluated on three public real-world educational datasets. The results demonstrate that our approach is superior on the KT prediction task, and it outperforms a wide range of deep learning based KT models in terms of prediction accuracy with better model interpretability. To encourage reproducible results, we have provided all the datasets and code at https://pykt.org/.

The development of online intelligent educational systems has revolutionized personalized learning, presenting an opportunity for the integration of knowledge tracing (KT). KT is an essential task that leverages students’ historical interactions to model their knowledge states, enabling accurate predictions of their future performance. The application of deep learning models on KT tasks, also known as deep learning based knowledge tracing (DLKT) models, has accelerated the process of KT tasks in recent years. Although DLKT models have achieved promising results, there is a huge challenge to avoid models generating wrong estimations that yield bad guidance in educational contexts. Hence, in this work, we propose a simulation framework to explore the feasibility of minimizing the evaluation cost while guaranteeing prediction performance. In the framework, we initially design a simple yet efficient DLKT model that learns the actual process of students’ knowledge acquisition. We then select the reliable predictions generated by the proposed model and assign the unreliable ones to teaching professionals based on confidence estimations. We present the results of a proof-of-concept experiment on three real-world publicly available datasets to demonstrate that our framework can obtain the balance of human cost and automatic evaluation accuracy, which can be flexibly deployed to real-world educational contexts in the future. To encourage reproducible research, we make our code publicly available at https://github.com/gwbnwnwh/human-in-the-loop.

Accurately judging students' ongoing performance is very crucial for real-world educational scenarios. In this work, we focus on the task of automatically predicting students' levels of mastery of math questions from teacher-student classroom dialogue data in the online learning environment. We propose a novel neural network armed with a multi-granularity attention mechanism to capture the personalized pedagogical instructions from the very noisy teacher-student dialogue transcriptions. We conduct experiments on a real-world educational dataset and the results demonstrate the superiority and availability of our model in terms of various evaluation metrics.

Knowledge tracing (KT) is a task that predicts students' future performance based on their historical learning interactions. With the rapid development of deep learning techniques, existing KT approaches follow a data-driven paradigm that uses massive problem-solving records to model students' learning processes. However, although the educational contexts contain various factors that may have an influence on student learning outcomes, existing public KT datasets mainly consist of anonymized ID-like features, which may hinder the research advances towards this field. Therefore, in this work, we present, \emph{XES3G5M}, a large-scale dataset with rich auxiliary information about questions and their associated knowledge components (KCs)\footnote{\label{ft:kc}A KC is a generalization of everyday terms like concept, principle, fact, or skill.}. The XES3G5M dataset is collected from a real-world online math learning platform, which contains 7,652 questions, and 865 KCs with 5,549,635 interactions from 18,066 students. To the best of our knowledge, the XES3G5M dataset not only has the largest number of KCs in math domain but contains the richest contextual information including tree structured KC relations, question types, textual contents and analysis and student response timestamps. Furthermore, we build a comprehensive benchmark on 19 state-of-the-art deep learning based knowledge tracing (DLKT) models. Extensive experiments demonstrate the effectiveness of leveraging the auxiliary information in our XES3G5M with DLKT models. We hope the proposed dataset can effectively facilitate the KT research work.

Knowledge tracing (KT) is the task of using students' historical learning interaction data to model their knowledge mastery over time so as to make predictions on their future interaction performance. Recently, remarkable progress has been made of using various deep learning techniques to solve the KT problem. However, the success behind deep learning based knowledge tracing (DLKT) approaches is still left somewhat unknown and proper measurement and analysis of these DLKT approaches remain a challenge.

Knowledge tracing (KT) is the problem of predicting students’ future performance based on their historical interactions with intelligent tutoring systems. Recent studies have applied multiple types of deep neural networks to solve the KT problem. However, there are two important factors in real-world educational data that are not well represented. First, most existing works augment input representations with the co-occurrence matrix of questions and knowledge components1 (KCs) but fail to explicitly integrate such intrinsic relations into the final response prediction task. Second, the individualized historical performance of students has not been well captured. In this paper, we proposed AT-DKT to improve the prediction performance of the original deep knowledge tracing model with two auxiliary learning tasks, i.e., question tagging (QT) prediction task and individualized prior knowledge (IK) prediction task. Specifically, the QT task helps learn better question representations by predicting whether questions contain specific KCs. The IK task captures students’ global historical performance by progressively predicting student-level prior knowledge that is hidden in students’ historical learning interactions. We conduct comprehensive experiments on three real-world educational datasets and compare the proposed approach to both deep sequential KT models and non-sequential models. Experimental results show that AT-DKT outperforms all sequential models with more than 0.9% improvements of AUC for all datasets, and is almost the second best compared to non-sequential models. Furthermore, we conduct both ablation studies and quantitative analysis to show the effectiveness of auxiliary tasks and the superior prediction outcomes of AT-DKT. To encourage reproducible research, we make our data and code publicly available at https://github.com/pykt-team/pykt-toolkit 2.

Online dialogic instructions are a set of pedagogical instructions used in real-world online educational contexts to motivate students, help understand learning materials, and build effective study habits. In spite of the popularity and advantages of online learning, the education technology and educational data mining communities still suffer from the lack of large-scale, high-quality, and well-annotated teaching instruction datasets to study computational approaches to automatically detect online dialogic instructions and further improve the online teaching effectiveness. Therefore, in this paper, we present a dataset of online dialogic instruction detection, DialogID, which contains 30,431 effective dialogic instructions. These teaching instructions are well annotated into 8 categories. Furthermore, we utilize the prevalent pre-trained language models (PLMs) and propose a simple yet effective adversarial training learning paradigm to improve the quality and generalization of dialogic instruction detection. Extensive experiments demonstrate that our approach outperforms a wide range of baseline methods. The data and our code are available for research purposes from: https://github.com/ai4ed/DialogID.

Knowledge tracing (KT) is the task of using students' historical learning interaction data to model their knowledge mastery over time so as to make predictions on their future interaction performance. Recently, remarkable progress has been made of using various deep learning techniques to solve the KT problem. However, the success behind deep learning based knowledge tracing (DLKT) approaches is still left somewhat unknown and proper measurement and analysis of these DLKT approaches remain a challenge. First, data preprocessing procedures in existing works are often private and custom, which limits experimental standardization. Furthermore, existing DLKT studies often differ in terms of the evaluation protocol and are far away real-world educational contexts. To address these problems, we introduce a comprehensive python based benchmark platform, \textsc{pyKT}, to guarantee valid comparisons across DLKT methods via thorough evaluations. The \textsc{pyKT} library consists of a standardized set of integrated data preprocessing procedures on 7 popular datasets across different domains, and 10 frequently compared DLKT model implementations for transparent experiments. Results from our fine-grained and rigorous empirical KT studies yield a set of observations and suggestions for effective DLKT, e.g., wrong evaluation setting may cause label leakage that generally leads to performance inflation; and the improvement of many DLKT approaches is minimal compared to the very first DLKT model proposed by Piech et al. \cite{piech2015deep}. We have open sourced \textsc{pyKT} and our experimental results at \url{https://pykt.org/}. We welcome contributions from other research groups and practitioners.