The widespread adoption of Generative AI (GenAI) tools by students across different educational levels highlights the need for them to develop robust GenAI literacy, including a working understanding of these systems' fundamental concepts, their limitations, and implications for responsible use. However, misconceptions about GenAI, such as perceiving these systems as mere search engines or database lookup systems, are commonly observed among students, while the availability of teaching resources remains fragmented, and learning objectives lack alignment. This Working Group aims to design pedagogical resources for computing science instructors, enabling them to develop students' GenAI literacy. To achieve this, the Working Group will first identify a concise set of GenAI literacy learning objectives informed by instructor experience, research literature, and community input, and subsequently design pedagogical resources aligned with these objectives.

Peer code reviews involve students conducting a code review of a classmate's submission to a programming assignment. While peer code reviews have an established history of being used and studied in computing education, they have primarily been documented in introductory computing courses. This experience report describes how we implemented peer code reviews in an upper-division software engineering course that focuses on making modifications to large, existing code bases (i.e., brownfield development). We discuss the perceived learning benefits, perceived challenges, and agreement between peer and course staff reviews. Overall, students enjoyed being able to see different approaches to the programming task they had just submitted, but expressed concerns about feeling qualified to make effective peer code reviews due to their limited software engineering experience and the difficulty of assessing code design. We also find that students are capable of evaluating the functional correctness of their peer's submission, but struggle to give accurate assessments of their peer's design and code style. We conclude with recommendations specifically for instructors who wish to use peer code reviews in upper-division software engineering courses, such as using a structured template to scaffold the peer code reviews and allowing multiple opportunities to provide code reviews to improve students' self-efficacy.

Low-quality code can cost a company significant time and effort. As a result, code quality has been consistently studied in computing education research, especially in the context of CS1 students. However, less research has examined students' code quality while working on existing code bases (i.e., in tasks they are expected to do in industry). In this paper, we identify 1) common code quality defects introduced by upper-division students while contributing to an existing code base, 2) the severity, tool support, and language independence of those defects, and 3) programming experiences that may be associated with students' frequencies of defects, such as internship experience and use of Python (which was the language used in the programming tasks). In an upper division software engineering course, 48 students worked individually to 1) modify an existing feature and 2) implement a new feature in an open-source code base. Using an existing framework of code quality defects by Řechtáčková et al., we conducted a manual code review of all student submissions and found that students created defects related to Poor Design, Poor Documentation, Poor Formatting, and Unused Code at a high frequency. Students also seemed to copy-paste code from other files, which introduced defects related to Unused Code and Poor Design to their submission. Though our regression analysis did not reveal statistically significant predictors, students with prior internships, on average, introduced more code quality defects than those without any internship experience.

Program comprehension (PC) literature typically focuses on industry professionals comprehending large code bases or novice programmers comprehending short programs. As a result, limited work has aimed to understand how intermediate programmers comprehend large code bases, especially with the goal of supporting learners' incremental development of program comprehension expertise. Through the lens of the Block Model-a theory to support research on and teaching of PC-we aim to uncover 1) the comprehension process that intermediate programmers follow (i.e., top-down, bottom-up, etc.), and 2) common mappings between comprehension techniques used by intermediate programmers and comprehension blocks in the Block Model. We present a diary study of students' “process journals” inwhich they described their PC process while modifying the open-source idlelib code base. Our results showed that students typically followed a top-down and Text-first approach to understand a feature in the idlelib code base. Our findings also reveal how students used various program comprehension techniques (such as code navigation, using the IDE-based debugger, making experimental code changes, etc.) in terms of the Block Model. These findings make progress toward bridging our theoretical understanding of novices' comprehension process in small programs and expert's code comprehension process in large code bases by presenting a high sample size investigation of intermediate programmers' PC processes in a large, existing code base. Instructors can use our findings to understand which blocks in the Block Model are cover PC techniques, which can enable targeted teaching activities to impart PC skills.

Understanding students' testing processes in a CS1 course is crucial in helping instructors of introductory courses determine the necessary content to teach. Prior work highlights the importance of teaching testing practices to students, as there is concern for students' testing abilities upon graduation of an university CS program. Given that testing is an implicit programming process, we aim to examine how students in CS1 go about testing their code in programming assignments. Because of the consistent research showing the achievement gap between students with and without prior experience in introductory classes, our analysis also aims to understand specific differences in testing processes between the two groups. Leveraging a dataset of over 300 students with over 50,000 snapshots of student code during their development process, we applied metrics related to incremental testing and determined the usage of diagnostic print statements and the usage of designing test cases beyond the given tests (in which we refer to as ' custom test cases '). A large majority of the students used neither diagnostic print statements nor custom test cases in their programming assignments. Additionally, the three testing practices we examined do not seem to significantly contribute to the achievement gap due to prior experience to students' success, suggesting a need for further investigation into which practices do account for that success.

Large language models (LLMs) are already heavily used by professional software engineers. An important skill for new university graduates to possess will be the ability to use such LLMs to effectively navigate and modify a large code base. While much of the prior work related to LLMs in computing education focuses on novice programmers learning to code, less work has focused on how upper-division students use and trust these tools, especially while working with large code bases. In this study, we taught students about various GitHub Copilot features, including Copilot chat, in an upper-division software engineering course and asked students to add a feature to a large code base using Copilot. Our analysis revealed a novel interaction pattern that we call one-shot prompting, in which students ask Copilot to implement the entire feature at once and spend the next few prompts asking Copilot to debug the code or asking Copilot to regenerate its incorrect response. Finally, students reported significantly more trust in the code comprehension features than code generation features of Copilot, perhaps due to the presence of trust affordances in the Copilot chat that are absent in the code generation features. Our study takes the first steps in understanding how upper-division students use Github Copilot so that our instruction can adequately prepare students for a career in software engineering.

We present results of an intervention integrating a tool promoting metacognitive study behaviors, CompassX, in a Data Structures and Algorithms (CS2) course. Metacognition---commonly referred to as ''thinking about thinking''---has been consistently linked to improved learning strategies and student achievement. However, no prior literature has practiced an intervention that addresses all three commonly-accepted phases of metacognition. Thus in this work, we share the key features of CompassXthat promote metacognitive study behaviors, how our users engaged with those features, and how continued practice of metacognition using those features is related to improved learning outcomes. Students used CompassX voluntarily and some students did not fully engage with all metacognition-based features. However, continued engagement with a metacognitive feature appears to be indicative of higher exam scores. This also implies the possibility of utilizing a metacognitive tool to improve the performance of student outcome modeling by collecting a new type of behavioral data.

Characterising code quality is a challenge that was addressed by Börstler et al. 's working group in 2017. As emerged from their study, educators, developers and students have different perceptions of the manifold aspects involved, and a major conclusion of that WG was that "code quality should be discussed more thoroughly in educational programs" [2, p. 70]. However, the lack of materials and the time constraints have slowed down progress in that regard.

Live coding is a recommended teaching practice in which an instructor dynamically programs in front of students. However, findings related to students' engagement during live coding are mixed. Some works have reported that live codingseems to improve student engagement while others regard live coding as an activity in which students passively observe the instructor without asking questions or following along. Active live coding, in which students extend a live coding example and discuss with peers, incorporates active learning with the traditional live coding approach. We conducted a quasi-experimental study in which one section of an advanced introductory programming course was taught using active live coding (ALC) and the other was taught using traditional live coding (TLC). The goal of this work is to compare students' behavioral engagement in the two lectures using a classroom observation protocol called the Behavioral Engagement Related to Instruction (BERI) protocol. Our results from the 2,790 observations we collected indicate that traditional live coding engages only 65% of students, on average. However, we found a "persisting engagement'' effect of active live coding, where students were significantlymore engaged in the traditional live coding components of a lecture up to 20 minutesafter the active live coding component. Notably, the two lecture groups performed similarly on the Post-Lecture Questions, which were administered after each lecture as a review of the lecture material. Therefore, our results indicate an improved student engagement due to active live coding, but do not show a corresponding improvement in conceptual knowledge.

Since March of 2020, universities around the world have offered remote versions of courses to help limit the spread of COVID-19. Two years later, in the Spring 2022 quarter, the lectures in the CS1 course at our large, public research-intensive university were taught via two modalities---an in-person modality in which students attended traditional, in-person lectures and a blended modality in which students attended a remote lecture on Zoom. Every other course component---labs, discussions, office hours---were held in-person for both groups. The unique setup of the CS1 course allowed us to perform a comparative analysis of the outcomes and attitudes between the two groups. In this paper, we analyze the difference in course outcomes, peer support, competitive feelings in class, and students' sense of belonging between the groups. Our results indicate that students in the blended learning group attended lectures more frequently than their in-person counterparts yet performed 4-7%worse on the midterm and final exams. The blended learning group also experienced significantlyless feelings of competitiveness than their in-person counterparts. Interestingly, we discovered a consistent trend among our results indicating that the gap in grades, peer support, and classroom competitiveness between the blended group and in-person group was more pronounced among first- and second-year undergraduates than third- and four-year students. Despite the two learning groups having different instructors, our results shed light on the potential advantages and drawbacks of a blended learning experience in CS1 that instructors should consider when deciding on the format of their course.

Cognitive Apprenticeship (CA) is an instructional model that outlines how experts can transfer their skills and knowledge to a learner for reasoning-based tasks, such as reading comprehension or mathematical problem solving. Specifically, CA includes 6 teaching methods---modeling, scaffolding, coaching, reflection, articulation, and exploration---that facilitate learners' observation, acquisition, and externalization of implicit processes and techniques for completing a task. In this paper, we present a systematic literature review of 143 conference papers across ACM and IEEE venues about CA in computer science education literature. Specifically, we aim to understand which teaching methods are typically referenced, the theory level (i.e., depth of CA theory discussion) present in the literature, and the key findings related to CA-based teaching approaches. Our review reveals that CA has been cited in computing education research as a guiding theory for various course designs, though there is a clear emphasis on papers related to modeling, scaffolding, and coaching whereas reflection, articulation, and exploration are under-explored. We found that CA methods have been effective in improving students' enthusiasm towards computing, improving pass-rates in courses, and improving instructors' capacity to accommodate more students by reducing instructor workload. However, a key challenge of CA approaches that emerged from our review is the difficulty in scaling the approach in settings with a high student to instructor ratio. Through this literature review, we aim to highlight effective CA approaches and how future initiatives can leverage CA to improve student learning.

Prior work has highlighted the gap between industry expectations for recent university graduates and the abilities those recent graduates possess. These works have even specifically recommended that students be given the opportunity to work on large, pre-existing code bases in their undergraduate career. This paper presents our experience teaching a newly-created course calledWorking with Large Code Bases. Guided by a Cognitive Apprenticeship approach to provide an authentic classroom experience that emphasizes the implicit processes and techniques involved in real-world software engineering, the course serves as a practical introduction to the skills and workflow involved in navigating and understanding a large code base. The goal of this experience report is to provide the motivation for key course design decisions, an overview of the course content, and a detailed description of key course components. We present student feedback indicating improved confidence in navigating a large code base and course outcomes related to specific tools and techniques students used in the course. Finally, we provide the full set of course materials we used and actionable recommendations for instructors to administer this course at their own institution, even with limited TA support.

Programming requires many skills. Programmers must not only be able to write and develop code using processes such as incremental development, debugging, and testing, but also may need to use effective verbal and written communication skills in their profession. I hypothesize that the methods of Cognitive Apprenticeship, which are modeling, coaching, scaffolding, articulation, reflection, and exploration, may offer a path to imparting these skills to students.

Background and Context. Live coding is a teaching method in which an instructor dynamically writes code in front of students in an effort to impart skills such as incremental development and debugging. By contrast, traditional, static-code examples typically involve an instructor annotating or explaining components of pre-written code. Despite recommendations to use live coding and a wealth of qualitative analyses that identify perceived learning benefits of it, there are a lack of empirical evaluations to confirm those learning benefits, especially with respect to students’ programming processes.

Online discussion boards, designed to facilitate learning from peers and instructors in an accessible space, are a vital part of course design, especially in large scale computer science classes. Previous work has shown that women in computer science tend to use anonymity more often than men on these boards, a trend not found in humanities, social science or business courses. In this work, we build on these findings using an intersectional lens, analyzing both gender and race/ethnicity. We find this combined analysis reveals differences in anonymity that are not apparent when examining gender alone. For example, we find a significantly greater difference in anonymity use between Hispanic men and women than would be expected from analyzing race/ethnicity and gender independently. We additionally analyze type of content (e.g., questions, answers), course, platform, and data source to characterize the many factors at play in measuring students’ choice to participate anonymously. In doing so, we show that different approaches used in prior work for eliciting information on gender — whether using registrar data, a survey, or imputing gender based on name — changes how over of students are classified, particularly affecting nonbinary students and Asian students. Understanding when students participate anonymously can help educators and platform designers to make students’ experience of online discussion boards more welcoming.

Live coding---a pedagogical technique in which an instructor plans, writes, and executes code in front of a class---is generally considered a best practice when teaching programming. However, only a few studies have evaluated the effect of live coding on student learning in a controlled experiment and most of the literature relating to live coding identifies students' perceived benefits of live-coding examples. In order to empirically evaluate the impact of live coding, we designed a controlled experiment in a CS1 course taught in Python at a large public university. In the two remote lecture sections for the course, one was taught using live-coding examples and the other was taught using static-code examples. Throughout the term, we collected code snapshots from students' programming assignments, students' grades, and the questions that they asked during the remote lectures. We then applied a set of process-oriented programming metrics to students' programming data to compare students' adherence to effective programming processes in the two learning groups and categorized each question asked in lectures following an open-coding approach. Our results revealed a general lack of difference between the two groups across programming processes, grades, and lecture questions asked. However, our experiment uncovered minimal effects in favor of the live-coding group indicating improved programming processes but lower performance on assignments and grades. Our results suggest an overall insignificant impact of the style of presenting code examples, though we reflect on the threats to validity in our study that should be addressed in future work.

Incremental development is the process of writing a small snippet of code and testing it before moving on. For students in introductory programming courses, the value of incremental development is especially higher as they may suffer from more syntax errors, lack the proficiency to address complicated bugs, and may be more prone to frustration when struggling to correct code. However, to evaluate the effectiveness of interventions that aim to teach programming processes such as incremental development, we need to develop measures to assess such processes. In this paper, we present a way to measure incremental development. By qualitatively analyzing 15 student coding interviews, we identified common behaviors in the programming process that relate to incremental development. We then leveraged a dataset of over 1000 development sessions -- about 52,000 code snapshots at compilation time -- to automatically detect the common behaviors identified in our qualitative analysis. Finally, we crafted a formal metric, called the "Measure of Incremental Development'' (MID), to quantify how effectively a student used incremental development during a programming session. The MID detects common non-incremental development patterns such as excessive debugging after large additions of code to automatically assess a sequence of snapshots. The MID aligns with human evaluations of incrementality with over 80% accuracy. Our metric enables new research directions and interventions focused on improving students' development practices.