Understanding how students reason about code is essential for providing tailored scaffolding in computer science (CS) education. Prior work has used think-aloud protocols with the Structure of the Observed Learning Outcomes (SOLO) taxonomy to examine students' code comprehension and programming levels. However, analyzing such data is labor-intensive and requires expert judgment. Recent advances in large language models (LLMs) offer a promising avenue for scaling this analysis, though their reliability for fine-grained coding remains uncertain. To address this gap, our study investigates the extent to which GPT-5 and 4o can classify SOLO levels and identify code-comprehension strategies from think-aloud transcripts of 27 high-school students working on block-based and text-based tasks. Results show modest alignment with human ratings for SOLO, with one-shot prompting improving agreement over zero-shot, though distinctions between adjacent lower levels (e.g., Prestructural 1 vs. 2) remained difficult. Strategy detection demonstrated stronger performance, achieving accuracies of 75–77% (block) and 62–67% (text), particularly for surface-visible strategies such as 'walkthroughs', 'control-structure identification', and 'pattern recognition', but weaker for less frequent, abstract, meta-cognitive strategies such as 'strategizing' (planning an approach) or 'thoroughness' (systematically checking work). These findings highlight both the potential and the limitations of using GPT-5 and 4o to analyze think-aloud data. While this work represents an initial step, with plans to examine more models, our preliminary results indicate that a human-in-the-loop approach is essential to ensure reliability and interpretive depth. Future work will extend this evaluation to other LLMs to better understand their role in supporting instructional decision-making.

Understanding how students comprehend code is essential for designing effective instructional support in computer science (CS). While prior studies have often relied on written responses, few have examined students' reasoning processes through think-aloud data. In this study, we analyzed the verbal reasoning of 27 high school students as they completed block-based and text-based code comprehension tasks targeting loops and conditional statements. Using an adapted SOLO taxonomy framework, we found that most students were classified at lower levels, with performance declining as they transitioned from block-based to text-based code. Students' strategy use, informed by prior work on code comprehension, showed that walkthroughs and identifying program structures were the most common approaches. Text-based tasks more often led students to use pattern-recognition strategies, such as interpreting operators or identifying numerical patterns, whereas block-based tasks occasionally prompted them to articulate broader problem-solving approaches. Overall, these findings demonstrate the value of applying the SOLO taxonomy to evaluate students' programming levels and highlight how programming modality impacts both the depth of understanding and the strategies students employ during code comprehension.

In this poster we highlight the strategies of K-12 in-service teachers who are learning to program in Java for the first time in an online graduate certificate program. Six teachers reported having limited and fundamental prior CS experience before learning to program in Java through a three-step Use-Modify-Create process. We implemented a model of metacognitive activity questionnaire to retrospectively collect a wide range of strategies reported by teachers as procedural steps which occurred before, during and after programming. Last, we interpret how these teachers' strategies were deployed as a higher-ordered process in the context of programming through a lens of abstraction and metacognitive regulation skills of planning, monitoring and evaluating the program solution.

Misconceptions in programming can significantly hinder students' conceptual development and their ability to apply core knowledge across contexts. Particularly in object-oriented programming (OOP), students often exhibit persistent conceptual misconceptions surrounding key concepts, such as classes and objects, which hinder their performance in advanced topics and courses, including data structures and algorithms. Traditional methods of identifying misconceptions, including interviews and detailed analysis of open-ended responses, are insightful but impractical for large classrooms. While coding platforms are widely used, they mainly detect syntax and output errors without revealing students' conceptual understanding. Furthermore, these systems often lack psychometric validation and fail to incorporate distractors grounded in authentic student thinking and misconceptions. This dissertation work addresses these gaps by developing and evaluating a three-tier diagnostic tool designed to identify misconceptions specifically in classes and objects using Kane's Validity Framework. The tool integrates answer, reason, and confidence tiers to capture both students' conceptual understanding and their level of certainty. Advances in large language models (LLMs) are leveraged to automate the generation of items with plausible distractors, enabling scalable and psychometrically sound item development. This work aims to contribute to the field by providing instructors with a research-informed tool for diagnosing misconceptions about classes and objects in large-scale CS2 courses that are psychometrically validated for their use. By combining structured diagnostic assessment design with recent advances in language models, the three-tier diagnostic tool will be a scalable, evidence-based approach to improving the teaching and learning of foundational OOP programming concepts.

Background and Context: While ample research has examined undergraduate students’ participation in Computer Science I (CS1), far less attention has been paid to Computer Science II (CS2) outcomes. Inspired by self-efficacy, Object-Oriented Programming (OOP) and CS2, we expand our understanding of the traditional computer programming curriculum sequence in CS curricula guidelines (CS1, CS2, and data structures and algorithms).

The expansion of K-12 computer science education has led to an increasing effort in preparing in-service teachers to learn and teach CS through participation in micro credentials, professional development, and online certificate programs. In this study, we recruited five novice K-12 in-service teachers enrolled in an online CS certificate program and highlighted their strategies used in programming a Java assignment through a three-phase process of planning, monitoring and evaluation.

Novice programmers often face challenges in designing computational artifacts and fixing code errors, which can lead to task abandonment and over-reliance on external support. While research has explored effective meta-cognitive strategies to scaffold novice programmers' learning, it is essential to first understand and assess students' conceptual, procedural, and strategic/conditional programming knowledge at scale. To address this issue, we propose a three-model framework that leverages Large Language Models (LLMs) to simulate, classify, and correct student responses to programming questions based on the SOLO Taxonomy. The SOLO Taxonomy provides a structured approach for categorizing student understanding into four levels: Pre-structural, Uni-structural, Multi-structural, and Relational. Our results showed that GPT-4o achieved high accuracy in generating and classifying responses for the Relational category, with moderate accuracy in the Uni-structural and Pre-structural categories, but struggled with the Multi-structural category. The model successfully corrected responses to the Relational level. Although further refinement is needed, these findings suggest that LLMs hold significant potential for supporting computer science education by assessing programming knowledge and guiding students toward deeper cognitive engagement.