Learning analytics researchers often analyze qualitative student data such as coded annotations or interview transcripts to understand learning processes. With the rise of generative AI, fully automated and human–AI workflows have emerged as promising methods for analysis. However, methodological standards to guide such workflows remain limited. In this study, we propose that reasoning traces generated by large language model (LLM) agents, especially within multi-agent systems, constitute a novel and rich form of process data to enhance interpretive practices in qualitative coding. We apply cosine similarity to LLM reasoning traces to systematically detect, quantify, and interpret disagreements among agents, reframing disagreement as a meaningful analytic signal. Analyzing nearly 10,000 instances of agent pairs coding human tutoring dialog segments, we show that LLM agents’ semantic reasoning similarity robustly differentiates consensus from disagreement and correlates with human coding reliability. Qualitative analysis guided by this metric reveals nuanced instructional sub-functions within codes and opportunities for conceptual codebook refinement. By integrating quantitative similarity metrics with qualitative review, our method bears potential to improve and accelerate the process of establishing inter-rater reliability during coding by surfacing interpretive ambiguity, especially when LLMs collaborate with humans. We discuss how reasoning-trace disagreements represent a valuable new class of analytic signals advancing methodological rigor and interpretive depth in educational research.

Computer science pedagogy has overwhelmingly favored problem-solving practice over methods of engagement like worked-out example study especially in advanced classes. This is due to the belief that while these alternative methods may improve student conceptual learning, they may leave them less able to perform on authentic problem-solving tasks from a lack of hands-on practice. In this paper, we perform a direct comparison of this trade-off in a synchronous collaborative programming project by adjusting the boundary between problem-solving and collaborative reflection based on a worked-out example while keeping the total time on task constant. We find that the more time students spent on worked example study, the more was the observed improvement in the pre- to post-test scores with no significant difference in performance on a subsequent problem-solving task. These results, therefore, challenge the dominant place of problem-solving practice in the advanced curricular context and inform the design of collaborative programming projects at scale.

This lightning talk presents an ongoing effort investigating the use of a collaborative programming paradigm originating in industry called Mob Programming, for effective collaborative learning in the classroom. In industry, Mob Programming involves the participation of a group of developers solving one problem at the same time and place. Developers take turns cycling through a structured process for collaboration having been assigned pre-defined roles responsible for brainstorming ideas, deciding on a path forward and implementing the consensus decision respectively. Pedagogically, there are several compelling reasons to motivate the adoption of Mob Programming in learning settings . First, the collaboration is well-structured meaning that the interaction between even a large group of students will not descend into chaos. Second, students are assigned to roles, which allows for the differentiation of responsibilities and development of skills in different aspects of solving the problem. Third, the rotation of assigned roles allows students to learn and exhibit multiple competencies as well as appreciate bringing different perspectives to bear on solving the problem. In order to investigate whether this promise is borne out in practice, the paradigm is currently being investigated in the context of a Cloud Computing course offered online to undergraduate and graduate students at a large American university and its satellite campuses. Since this effort is still underway, faculty who implement or are interested in implementing collaborative learning for this classrooms are invited to attend and provide feedback or consider joining the effort to investigate this paradigm for use in learning settings.

This work presents a new paradigm for collaborative project-based computer science education called Online Mob Programming (OMP). OMP is adapted from the industrial practice of Mob Programming, where groups of developers work on the same problem, at the same time, in the same place. OMP was designed and implemented as a technique where a group of 4-6 students collaborate online through a structured process for solving programming tasks. In OMP, students rotate through clearly defined roles to collectively contribute towards a solution to a programming challenge. These roles require students to brainstorm potential solutions, decide on a path forward, and implement the correct course of action respectively. OMP was investigated in the context of a 6-week free online course on Cloud Computing. During the course, students participated in four intelligent conversational agent-coordinated OMP sessions. By instrumenting the online development environment, all student code revisions and chat logs were collected in addition to qualitative data from questionnaires. Analyses show evidence of success in terms of students following the structure of OMP and further investigations into differences in mob behavior based on the size, and problem outcome provide pedagogically valuable insights and a path toward building OMP into the computer science education curriculum.

We present a novel approach to automated question generation that improves upon prior work both from a technology perspective and from an assessment perspective. Our system is aimed at engaging language learners by generating multiple-choice questions which utilize specific inference steps over multiple sentences, namely coreference resolution and paraphrase detection. The system also generates correct answers and semantically-motivated phrase-level distractors as answer choices. Evaluation by human annotators indicates that our approach requires a larger number of inference steps, which necessitate deeper semantic understanding of texts than a traditional single-sentence approach.