We investigate how students learn to apply context-specific math strategies by analyzing data from MATHia (a widely used Intelligent Tutoring System) collected from a large set of schools. In particular, we focus on a set of lessons designed to teach ratios and proportions, where students learn multiple strategies individually and then are presented with lessons in which they are presented with options to make a choice between strategies. Our results demonstrate that a majority of students may not learn conditional reasoning to select optimal strategies. To understand this more deeply, we use knowledge tracing models and also explain and interpret neural representations of strategies learned using BERT from step-level interactions between students and the ITS. Finally, we study the effectiveness of MATHia’s adaptive supports that attempt to guide students to the optimal strategy, and compare insights from our data to those produced by state-of-the-art generative AI models. Our results demonstrate that LLMs may produce results that seem to reflect ideal, expected outcomes in strategy learning, but the generation may not accurately reflect the complexities of real student learning.

Assessing students' responses, especially natural language responses, is a major challenge in education. In general, in education contexts, automatically evaluating what learners do or say is important as it enables personalized instruction, e.g., based on what the learner knows tailored tasks and feedback are given to the learner. Recently, deep learning techniques led to state-of-the-art methods in NLP such as transformer-based methods which resulted in significant performance improvements for many NLP tasks such as text classification and question answering. However, there is not much work exploring such methods for assessing students' free answers, particularly in the context of code comprehension, which brings additional challenges as the student explanations include code references as well. This paper explores the potential of applying automated assessments methods using transformers to code comprehension. We fine-tuned pre-trained transformer models, including BERT, RoBERTa, CodeBERT, and SciBERT, to see how well they can automatically judge students' responses to code comprehension tasks. Our results demonstrate that these models can significantly enhance the accuracy and reliability of automated assessments, offering insights into how the latest NLP techniques can be leveraged in computer science education to support personalized learning experiences.

Adapting to a student's problem solving strategy can lead to improved engagement and motivation. In this work, we develop an AI-based approach to analyze math learning strategies at scale. Specifically, we use a state-of-the-art AI model, namely, BERT to learn structure within strategies observed in large datasets. In particular, we consider the MATHia ITS and define strategies as sequences of steps that a student follows in solving the problem. We apply BERT pre-training to learn semantic representations of strategies from a workspace in MATHia that allows for different strategies. Further, we fine-tune these embeddings to train them on downstream tasks such as identifying a strategy and understanding drift in strategy. Our preliminary results are encouraging and demonstrate that BERT can uncover hidden structure in strategies and therefore is a promising direction to analyze large-scale math learning data.

We present in this paper the results of a randomized control trial experiment that compared the effectiveness of two instructional strategies that scaffold learners' code comprehension processes: eliciting Free Self-Explanation and a Socratic Method. Code comprehension, i.e., understanding source code, is a critical skill for both learners and professionals. Improving learners' code comprehension skills should result in improved learning which in turn should help with retention in intro-to-programming courses which are notorious for suffering from very high attrition rates due to the complexity of programming topics. To this end, the reported experiment is meant to explore the effectiveness of various strategies to elicit self-explanation as a way to improve comprehension and learning during complex code comprehension and learning activities in intro-to-programming courses. The experiment showed pre-/post-test learning gains of 30% (M = 0.30, SD = 0.47) for the Free Self-Explanation condition and learning gains of 59% (M = 0.59,SD = 0.39) for the Socratic method. Furthermore, we investigated the behavior of the two strategies as a function of students' prior knowledge which was measured using learners' pretest score. For the Free Self-Explanation condition, there was no significant difference in mean learning gains for low vs. high knowledge students. The magnitude of the difference in performance (mean difference= 0.02,95% CI: -0.34 to 0.39) was very small (eta squared = 0.006). Likewise, the Socratic method showed no significant difference in mean learning gains between low vs. high performing students. The magnitude of the performance difference (mean difference =-0.24,95% CI: -0.534 to 0.03) was large (eta squared = 0.10). These findings suggest that eliciting self-explanations can be used as an effective strategy and that guided self-explanations as in the Socratic method condition is more effective at inducing learning gains.

Automatically identifying Chinese characters that are similar in their glyph, pronunciations and meaning are important for building smart question generation tools in a computer-assisted language-learning environment. Previous research on the Chinese character similarity measurement focused on char acter glyph (e.g. structures, strokes and radicals) with heuristic algorithms whose parameter have preset values. This article presents a machine learning (regression) approach to measure the similarity between two Chinese characters, based on the information which not only includes the glyph, but also pronunciation (pinyin) and semantic meaning derived from HowNet. We evaluated various regression models using a testing set consisting of 2586 pairs of characters selected from elementary Chinese textbooks used. The study results showed that four regression models (M5, Support Vector Machine, Gaussian Process and Linear Regression) have similar results (0.617⩽Mean Absolute Error⩽0.641, 0.772⩽Root Mean Square Error⩽0.790). In addition, the study implied that the performance of the regression model could be influenced by the character frequency. Moreover, we evaluated the regression model in a well-known Chinese language learning resource, called 100 pairs of the most confusing Chinese characters. The experiment results indicated that this approach has potential in the recognition and generation of confusing Chinese character pairs.

Identifying dialogue acts and dialogue modes during tutorial interactions is an extremely crucial sub-step in understanding patterns of effective tutor-tutee interactions. In this work, we develop a novel joint inference method that labels each utterance in a tutoring dialogue session with a dialogue act and a specific mode from a set of pre-defined dialogue acts and modes, respectively. Specifically, we develop our joint model using Markov Logic Networks (MLNs), a framework that combines first-order logic with probabilities, and is thus capable of representing complex, uncertain knowledge. We define first-order formulas in our MLN that encode the inter-dependencies between dialogue modes and more fine-grained dialogue actions. We then use a joint inference to jointly label the modes as well as the dialogue acts in an utterance. We compare our system against a pipeline system based on SVMs on a real-world dataset with tutoring sessions of over 500 students. Our results show that the joint inference system is far more effective than the pipeline system in mode detection, and improves over the performance of the pipeline system by about 6 points in F1 score. The joint inference system also performs much better than the pipeline system in the context of labeling modes that highlight important pedagogical steps in tutoring.

We present in this paper an innovative solution to the challenge of building effective educational technologies that offer tailored instruction to each individual learner. The proposed solution in the form of a conversational intelligent tutoring system, called DeepTutor, has been developed as a web application that is accessible 24/7 through a browser from any device connected to the Internet. The success of several large scale experiments with high-school students using DeepTutor is a solid proof that conversational intelligent tutoring at scale over the web is possible.

We present an overview of the design of a conversational intelligent tutoring system, called DeepTutor, based on the framework of Learning Progressions. Learning Progressions capture students' successful paths towards mastery. The assumption of the proposed tutor is that by guiding instruction based on Learning Progressions, the system will be more effective (and efficient for that matter).

Today, many common tasks (e.g. booking flights, ordering food) can be done by filling out web forms. Automatic processing of Web forms to support interactive speech input is useful for numerous reasons, including ease of use for mobile device users and accessibility for people with visual or print disabilities. In this paper, we propose an automated method to process web forms and convert them into dialog flows for spoken interaction. First we identify relevant information for each form element (including element type, label, values and help messages) and key relationships between form elements (including ordering and dependencies). We then generate two types of dialog flow for each Web form. Experimental results show that the method generates efficient and informative dialog flows for web tasks, a key step for building virtual assistants. An Android application has been realized as a use case of the generated dialog flows.

Research towards automated feedback can build on the work in other areas. In this paper we explore question generation techniques. Most research in question generation has focused on generating content specific questions that help students comprehend a set of documents that they must read. However, this approach is not so useful in writing activities, as students would generally understand the document that they themselves wrote. The aim of our project is to build a system which automatically generates feedback questions for academic writing support, particularly for critical review support. This paper presents our question generation system which relies on both syntax-based and template-based approaches, and uses Wikipedia as background knowledge.

We report preliminary data of an initial laboratory study examining the effectiveness of self-regulated learning (SRL) training versus no training on learners' ability to deploy SRL processes and learn about the circulatory system with MetaTutor. MetaTutor is an intelligent tutoring system (ITS) designed to train and foster learners' SRL processes while learning about several complex human body systems. We used a mixed methodology approach and include the results of a subset of the participants (N=30) whose product and process data we have analyzed. Overall, the results indicate that the SRL training group significantly outperformed the control group.

We present in this paper an approach to assessing student paraphrases in the intelligent tutoring system iSTART. The approach is based on measuring the semantic similarity between a student paraphrase and a reference text, called the textbase. The semantic similarity is estimated using knowledge-based word relatedness measures. The relatedness measures rely on knowledge encoded in Word-Net, a lexical database of English. We also experiment with weighting words based on their importance. The word importance information was derived from an analysis of word distributions in 2,225,726 documents from Wikipedia. Performance is reported for 12 different models which resulted from combining 3 different relatedness measures, 2 word sense disambiguation methods, and 2 word-weighting schemes. Furthermore, comparisons are made to other approaches such as Latent Semantic Analysis and the Entailer.

Spams are no longer limited to emails and Web-pages. The increasing penetration of spam in the form of comments in blogs and social networks has started becoming a nuisance and potential threat. In this work, we explore the challenges posed by this type of spam in the blogosphere with substantial generalization regarding other social media. Thus, we investigate the characteristics of comment spam in blogs based on their content. The framework uses some of the previously explored methods developed to effectively extract the features of the blog spam and also introduces a novel method of active learning from the raw data without requiring training instances. This makes the approach more flexible and realistic for such applications. We also incorporate the concept of co-training for supervised learning to get accurate results. The preliminary evaluation of the proposed framework shows promising results.

This paper evaluates four of the most commonly used, freely available, state-of-the-art parsers on a standard benchmark as well as with respect to a set of data relevant for measuring text cohesion, as one example of a learning technology application that requires fast and accurate syntactic parsing. We outline advantages and disadvantages of existing technologies and make recommendations. Our performance report uses traditional measures based on a gold standard as well as novel dimensions for parsing evaluation. To our knowledge, this is the first attempt to evaluate parsers across genres and grade levels for the implementation in learning technology using both gold standard and directed evaluation methods.

In this paper we study a graph-based approach to the task of textual entailment between a text and hypothesis. The approach takes into account the full lexico-syntactic context of both the text and hypothesis and relies heavily on the concept of subsumption. It starts with mapping the text and hypothesis into graph structures where nodes represent concepts and edges represent lexico-syntactic relations among concepts. Based on a subsumption score between the text-graph and hypothesis-graph an entailment decision is then made. The impact of synonymy on entailment is quantified and discussed. An important advantage of our solution is the ability to customize it so that high-confidence results are obtained.

This paper presents an open-domain textual Question-Answering system that uses several feedback loops to enhance its performance. These feedback loops combine in a new way statistical results with syntactic, semantic or pragmatic information derived from texts and lexical databases. The paper presents the contribution of each feedback loop to the overall performance of 76% human-assessed precise answers.

WordNet is a rich source of world knowledge from which formal axioms can be derived. In this paper we present a method for transforming the WordNet glosses into logic forms and further into axioms. The transformation of WordNet glosses into logic forms is useful for theorem proving and other applications. The paper demonstrates the utility of the WordNet axioms in a question answering system to rank and extract answers.

This paper presents few extensions to the logic form representation and a method for transforming WordNet glosses into logic forms using a set of high-precision rules combined with a set of high recall heuristics. An almost 3% increase in POS tagging accuracy is achieved over state-of-the art results at the expense of user intervention on only 7.52% of words. We apply a nearest neighbor solution to parser switching that leads to 6.43% increase in exact sentence accuracy for glosses. Logic Forms are derived with an accuracy of 89.46%.

This paper presents the architecture, operation and results obtained with the LASSO Question Answering system developed in the Natural Language Processing Laboratory at SMU. To find answers, the system relies on a combination of syntactic and semantic techniques. The search for the answer is based on a novel form of indexing called paragraph indexing. A score of 55.5% for short answers and 64.5% for long answers was achieved at the TREC-8 competition.