This paper focuses on using Large Language Models to support teaching assistants in answering questions on large student forums such as Piazza and EdSTEM. Since student questions on these forums are often closely tied to specific aspects of the institution, instructor, and course delivery, general-purpose LLMs do not directly do well on this task. We introduce RetLLM-E, a method that combines text-retrieval and prompting approaches to enable LLMs to provide precise and high-quality answers to student questions. When presented with a student question, our system initiates a two-step process. First, it retrieves relevant context from (i) a dataset of student questions addressed by course instructors (Q&A Retrieval) and (ii) relevant segments of course materials (Document Retrieval). RetLLM-E then prompts LLM using the retrieved text and an engineered prompt structure to yield an answer optimized for the student question. We present a set of quantitative and human evaluation experiments, comparing our method to ground truth answers to questions in a test set of actual student questions. Our results demonstrate that our approach provides higher-quality responses to course-related questions than an LLM operating without context or relying solely on retrieval-based context. RetLLM-E can easily be adopted in different courses, providing instructors and students with context-aware automatic responses.

Recent advancements in instruction-tuned large language models offer new potential for enhancing students' experiences in large-scale classes. Deploying LLMs as student-facing assistants, however, presents challenges. Key issues include integrating class-specific content into responses and applying effective pedagogical techniques. This study addresses these challenges through retrieval and prompting techniques, focusing on mitigating hallucinations in LLM-generated responses, a crucial concern in education. Furthermore, practical deployment brings further challenges related to student data privacy and computational constraints. This research strives to enhance the quality and relevance of LLM responses while addressing practical deployment issues, with an emphasis on creating a versatile system for diverse domains and teaching styles.