With the growing use of Large Language Model (LLM)-based Question-Answering (QA) systems in education, it is critical to evaluate their performance across individual pipeline components. In this work, we introduce EduMod-LLM, a modular function-calling LLM pipeline, and present a comprehensive evaluation along three key axes: function calling strategies, retrieval methods, and generative language models. Our framework enables fine-grained analysis by isolating and assessing each component. We benchmark function-calling performance across LLMs, compare our novel structure-aware retrieval method to vector-based and LLM-scoring baselines, and evaluate various LLMs for response synthesis. This modular approach reveals specific failure modes and performance patterns, supporting the development of interpretable and effective educational QA systems. Our findings demonstrate the value of modular function calling in improving system transparency and pedagogical alignment.

Recent advancements in large language models (LLMs) have paved the way for AI educational assistants in academic settings. However, AI assistants often respond differently than TAs, providing extensive explanations that may overwhelm students or inadvertently reveal more than intended. This study identifies the main differences between TA and LLM responses to students by using a four-class utterance classification system to compare the utterance distributions found in TA replies and in responses generated by Edison, a state-of-the-art AI educational assistant. Using this classification, striking distributional differences are observed: Edison produces far more Advance utterances, whereas TAs use many more React and Social Convention utterances. This research examines how differences in these distributions relate to response quality in student–TA interactions. Through prompt engineering, we align Edison's utterance distribution with TA patterns, producing responses that are more concise, directly address student questions, and avoid unnecessary elaboration.

While Large Language Models (LLMs) have emerged as promising methods for automated student question-answering, guaranteeing consistent instructional effectiveness of the response remains a key challenge. Therefore, there is a need for fine-grained analysis of State-Of-The-Art (SOTA) LLM-powered educational assistants.

Large Language Models (LLMs) can be used to automate many aspects of the educational field. In this paper, we look into the benefits of automating responses to student questions in course discussion forums using our Retrieval-Augmented Generation (RAG)-based LLM pipeline (Edison). Our research questions are:

This paper focuses on using Large Language Models to support teaching assistants in answering questions on large student forums such as Piazza and EdSTEM. Since student questions on these forums are often closely tied to specific aspects of the institution, instructor, and course delivery, general-purpose LLMs do not directly do well on this task. We introduce RetLLM-E, a method that combines text-retrieval and prompting approaches to enable LLMs to provide precise and high-quality answers to student questions. When presented with a student question, our system initiates a two-step process. First, it retrieves relevant context from (i) a dataset of student questions addressed by course instructors (Q&A Retrieval) and (ii) relevant segments of course materials (Document Retrieval). RetLLM-E then prompts LLM using the retrieved text and an engineered prompt structure to yield an answer optimized for the student question. We present a set of quantitative and human evaluation experiments, comparing our method to ground truth answers to questions in a test set of actual student questions. Our results demonstrate that our approach provides higher-quality responses to course-related questions than an LLM operating without context or relying solely on retrieval-based context. RetLLM-E can easily be adopted in different courses, providing instructors and students with context-aware automatic responses.

Today's rapidly evolving educational landscape prioritizes active student engagement. Classrooms at scale face particular challenges in fostering meaningful interactions between students and course content. In this study, we introduce EIT (Earnest Insight Toolkit), a tool designed to assess students' engagement within interactive lecture participation exercises-particularly in the context of large-scale hybrid classrooms. We use EIT to conduct a comprehensive assessment of student responses to interactive lecture poll questions. Our objective with EIT is to equip educators with valuable means of identifying at-risk students for enhancing intervention and support strategies and measuring student engagement with course content.

Recent advancements in instruction-tuned large language models offer new potential for enhancing students' experiences in large-scale classes. Deploying LLMs as student-facing assistants, however, presents challenges. Key issues include integrating class-specific content into responses and applying effective pedagogical techniques. This study addresses these challenges through retrieval and prompting techniques, focusing on mitigating hallucinations in LLM-generated responses, a crucial concern in education. Furthermore, practical deployment brings further challenges related to student data privacy and computational constraints. This research strives to enhance the quality and relevance of LLM responses while addressing practical deployment issues, with an emphasis on creating a versatile system for diverse domains and teaching styles.