This study investigates the consistency and fairness of grading in higher education by analyzing the impact of individual graders on class averages, identifying grading inconsistencies through similarity assessments of essays, and examining the correlation between the use of detailed grading rubrics and the frequency of student re-grade requests. Utilizing a comprehensive dataset of student essays, assigned grades, and grading rubrics, the research employs statistical analyses and natural language processing techniques to address these objectives at scale. Using the central limit theorem, we can identify significant differences in overall scoring between graders. Additionally, the study reveals instances where textually similar essays received markedly different grades, highlighting potential inconsistencies in grading practices. Furthermore, our analysis reveals that graders who apply a larger number of rubric items in their grading tend to receive higher number of student regrade requests (r>0.87). This indicates that while rubric-based grading is intended to enhance transparency, its extensive use may leave room for subjective interpretation, leading to more frequent grade challenges. These insights contribute to the ongoing discourse on improving grading practices and offer recommendations for enhancing transparency and equity in academic assessments at scale.

Cheating detection in large classes with online, take-home exams is an extremely difficult problem. While some cheating can be identified through statistical analysis of all student responses, this analysis can easily be fooled by "smart cheaters'' actively attempting to hide evidence of their unauthorized collaboration. We demonstrate the effectiveness of watermarks combined with creative question design to provide evidence of cheating. We provide results from an initial deployment of our answer watermarking method and do a case study into how "smart cheaters'' attempt to cover their tracks, demonstrating the need for more advanced methods of catching cheating in online, take-home exams.

Interactive teaching methods often lead to higher levels of student engagement with course material. Yet, as class sizes increase, the demand on teaching staff becomes unsustainable. Our solution, Socratic Mind, employs Large Language Models to provide scalable, interactive oral assessments by functioning as a virtual instructor. This paper discusses the outcomes and user feedback from the preliminary implementation of our system in a large classroom environment with 600 students.

Examinator v3.0 detects cheating in online take-home exams by comparing answers and the timestamps they were entered. A web interface enables efficient manual inspection. Use of the tool reveals that certain question types substantially enhance cheating detection, demonstrating the potential of automated algorithmic detection at scale. Examinator v3.0 has analyzed 915,831 pairs of exam submissions across three courses over two semesters at a top U.S. institution, identifying 46 instances of cheating.

As enrollment increases, teaching assistants (TAs) need help prioritizing responding to students' posts in on-line forums. In a graduate-level Artificial Intelligence forum, three instructors rank the urgency of posts which is compared to the ratings of 13 course TAs; correlation with the instructors' scores have an r=0.55, with a TA inter-rater reliability of 35%. However, when TAs used a codebook containing seven dimensions created by the instructors to define urgency levels, correlation increased to r=0.73 and reliability to 53%. The instructor rankings are also compared to cognitive presence ratings from the Community of Inquiry framework. Cognitive presence correlates with urgency with r=-0.68. These results suggest that recommendation agents that prioritize posts based on cognitive presence or the urgency codebook may be beneficial for TAs.

Embedding a post recommendation system in online course forums improves students' ability to find relevant course content. Yet, there is limited research quantifying how these interventions impact forum interactions and students' class performance. We randomly divide an on-line class in a Masters degree-seeking program into control and experimental sections. Midway through the class, for the experimental section, we introduce an information retrieval system that suggests relevant posts to students while they compose their new posts. The average number of initial posts in the discussion forum dropped by 55% for the experimental group compared to 21% in the control group. In post-hoc analysis, we classify students as having an active or passive (i.e., mostly lurking) forum participation style. Posts per student per assignment by passive participants in the experimental group decreased by 15%, while passive participants' posts in the control group increased by 200%. The number of answers given by instructors in the experimental group decreased twice as much as the control group after intervention, though the difference was not statistically significant. The reduction of posts was not associated with a decrease in academic performance (grades) for the experimental group. This line of research convinced the popular forum Piazza to implement a post recommendation system similar to that used in these experiments.

This paper presents the progress made towards developing an equitable predictive model for admission success to an online Master's program with a large pool of applicants. The overarching goal of this project is to help the future development of a systematic evaluation tool for programs with large applications. In the first phase of the project, we collected and processed data on 9,044 applications and have trained a predictive model using applicants' profile information such as demographic data, academic background, and test scores. In an ongoing phase, we seek to expand the applicants' database by incorporating the information in the letters of recommendation (LORs) and statements of purpose (SOPs) that are essential components of the application package for graduate programs and are extensively used to make decisions on granting admission. In this study, we assess various aspects of the LORs and SOPs using natural language processing to extract a comprehensive list of text features that are used to develop a classifier. We implement machine-learning algorithms such as Gradient Boosting to predict admission status and to identify the text features with the highest weight on the applicants' success. This work provides an understanding of the level of significance of a variety of text features that eventually helps the development of a comprehensive predictive model.