Code-generating Artificial Intelligence has gained popularity within both professional and educational programming settings over the past several years. While research and pedagogy are beginning to cope with this change, computing students are left to bear the unforeseen consequences of AI amidst a dearth of empirical evidence about its effects. Though pair programming between students is well studied and known to be beneficial to self-efficacy and academic achievement, it remains underutilized and further threatened by the proposition that AI can replace a human programming partner. In this paper, we present a controlled pair programming study with 22 participants who wrote Python code under time pressure in teams of two and individually with GitHub Copilot for 20 minutes each. They were incentivized by bonus compensation to balance performance with understanding and were retested individually on the programming tasks after a retention interval of one week. Subjective measures of workload and emotion as well as objective measures of performance and learning (retest performance) were collected. When first programming with AI, participants clearly performed better and had an easier time. On retest, they were not dramatically worse in raw score from the human-human condition, but human-AI pairs lost more of their initial advantage. Additionally, the emotional effect of the human teammate was significantly more positive and arousing as compared to working with Copilot. We recommend that educators strongly consider revisiting pair programming as an educational tool in addition to embracing modern AI.

GenAI is playing an increasingly important role in computing courses at all levels, offering new opportunities to support teaching and learning. However, using GenAI effectively raises important concerns regarding trust, academic integrity, and broader social and ethical dimensions. This Working Group was formed to report on the current state of the art in using GenAI in upper-level computing courses to aid educators. The working group will undertake a methodological review of published work and solicit input from the computing educational community as part of the report.

This research full paper explores the connection between office hours use and online forum engagement in introductory computer science courses. Office hours (OH) and online question-and-answer (Q&A) forums provide a platform for students to interact with their classmates and instructors. We investigate the relationship between student engagement in an online discussion forum (Piazza) and utilization of office hours across 5 semesters of an introductory CS course. We explored the correlation between Piazza utilization and OH attendance, discerned disparities between in-person and online OH involvement, and analyzed the distinct approaches of men and women in engaging with course resources. We found that active Piazza users visit OH more than inactive Piazza users. More specifically, students who interact above average on Piazza in each metric observed - asks, answers, posts, and views - attend OH more than those who are below average in each metric. Additionally, students who attend OH at least once tend to post, ask, answer, and view posts on Piazza more frequently than those who have never attended OH. This indicates that above average help-seeking students on Piazza and in OH tend to engage with available resources more than those who did not seek help as often. This quantifies how often - and through which methods - students seek help. Based on prior research and our findings, we find it likely that students often begin by seeking answers on Piazza. If they find the response unsatisfactory, they then resort to OH for clarification. We also examined the modality of office hours, comparing in-person and online interactions; there is no significant statistical difference in the number of OH visits between those who attend virtually vs those who attend in person. However, online OH visits tended to take longer than in-person visits. In terms of the relationship between engagement and gender, our findings show that women visit OH more than men, both in person and online, and take longer in their OH visits. These findings emphasize the importance of course engagement resources in assisting with learning while also highlighting factors that affect engagement, such as gender, mode of engagement, and usage of other resources, giving instructors a better understanding of which populations tend to engage with specific course resources.

The ITiCSE '23 final keynote raised teaching soft skills, or professional dispositions, to help students face challenges in modern programming. This project addresses helping computing students develop professional dispositions through collaborative learning (CL) since some in the industry observe entry-level engineers struggling due to their fragile professional dispositions. We are motivated to understand professional expectations from entry-level engineers and present the academia-industry gap to support practitioners and researchers in advancing CL in Computing Education, encouraging positive curricula and policy changes that promote DEIA. We will present CL practices alongside their supported professional dispositions to assist practitioners in adoption. We will present the academia-industry gap in CL for future research opportunities, helping researchers advance CL practices to integrate professional dispositions the industry expects from entry-level engineers.

Artificial Intelligence-driven Development Environments (AIDEs) offer developers revolutionary computer programming assistance. There is great potential in incorporating AIDEs into Computer Science education; however, the effects of these tools should be fully examined before doing so. Here, a within-subjects study was conducted to compare the programming performance, workload, emotion, and self-efficacy of seventeen novices coding with and without use of the GitHub Copilot AIDE under time pressure. Results showed that using the AIDE significantly increased programming efficiency and reduced effort and mental workload but did not significantly impact emotion or self-efficacy. However, participants' performance improved with more experience using the AI, and their self-efficacy followed. The results suggest that students who try AIDEs will likely be tempted to use them for time-sensitive work. There is no evidence that providing AIDEs will aid struggling students, but there is a clear need for students to practice with AI to become competent and confident using it.

Project-based (PB) learning has become increasingly popular in computer science education, particularly as studies have found that the teaching style better prepares students for future careers and improves learning outcomes through increased student engagement. Online forum usage is one measurable component of engagement. In order to study the impact of PB learning on online forum engagement, Piazza usage data from seven online computer science courses at a higher education institution were collected and examined. We analyzed the differences in online forum usage between PB and assignment-based (AB) learning, in addition to differences between men and women in each course type. Specifically, this study builds upon and replicates a previous study on Piazza that measured student engagement, anonymity usage, and peer parity. We found that students in PB courses were less actively engaged in online forums than students in AB courses; they were less likely to ask and answer questions on Piazza but were more likely to view posts and be logged on more days. Across both course types, students posted anonymously a similar amount as a proportion of the total number of questions and answers and experienced a proportionally similar amount of peer parity. Our findings mirror prior results on gender engagement on Piazza. Across both PB and AB courses, women were more engaged, asked and viewed more questions, posted anonymously more frequently, and were less likely to experience peer parity than men.

The recent advent of highly accurate and scalable large language models (LLMs) has taken the world by storm. From art to essays to computer code, LLMs are producing novel content that until recently was thought only humans could produce. Recent work in computing education has sought to understand the capabilities of LLMs for solving tasks such as writing code, explaining code, creating novel coding assignments, interpreting programming error messages, and more. However, these technologies continue to evolve at an astonishing rate leaving educators little time to adapt. This working group seeks to document the state-of-the-art for code generation LLMs, detail current opportunities and challenges related to their use, and present actionable approaches to integrating them into computing curricula.

Metacognition and self-regulation are important skills for successful learning and have been discussed and researched extensively in the general education literature for several decades. More recently, there has been growing interest in understanding how metacognitive and self-regulatory skills contribute to student success in the context of computing education. This article presents a thorough systematic review of metacognition and self-regulation work in the context of computer programming and an in-depth discussion of the theories that have been leveraged in some way. We also discuss several prominent metacognitive and self-regulation theories from the literature outside of computing education—for example, from psychology and education—that have yet to be applied in the context of programming education. In our investigation, we built a comprehensive corpus of papers on metacognition and self-regulation in programming education, and then employed backward snowballing to provide a deeper examination of foundational theories from outside computing education, some of which have been explored in programming education, and others that have yet to be but hold much promise. In addition, we make new observations about the way these theories are used by the computing education community, and present recommendations on how metacognition and self-regulation can help inform programming education in the future. In particular, we discuss exemplars of studies that have used existing theories to support their design and discussion of results as well as studies that have proposed their own metacognitive theories in the context of programming education. Readers will also find the article a useful resource for helping students in programming courses develop effective strategies for metacognition and self-regulation.

Online discussion forums are being increasingly used in classrooms as a way to encourage collaborative learning and community. Piazza is one such forum that was built specifically for academic institutions, and has been widely adopted. Students have the opportunity to ask questions and seek answers from peers and instructors alike online, allowing them to find the information they need even if they do not know fellow students in the class or if they cannot make an instructor's office hours. However, recent analysis of the popular online discussion site Stack Overflow, suggests that women are more likely than men to withdraw from such a community if they do not identify other members of the same gender. Women are often a minority in computer science courses and may express difficulty interacting with or seeking help from their peers who are predominantly men. Considering the importance of providing equal access to students regardless of gender and the value of resources like Piazza in one's education, it is imperative to assess the representation and impact of gender on Piazza. We analyzed data from over 2,500 Piazza users across three computer science courses at the University of Virginia and found that women on Piazza post more questions than men, spend more time on the discussion site, and achieve higher reputation scores on average. However, they are more likely than men to both ask and answer questions anonymously and less likely to receive responses from members of the same gender.

Diagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for decades. Unfortunately these messages which include error, warning, and runtime messages, present substantial difficulty and could be more effective, particularly for novices. Recent years have seen increased number of papers in the area including studies on the effectiveness of these messages, improving or enhancing them, and their usefulness as a part of programming process data that can be used to predict student performance. Despite this increased interest, the long history of literature is quite scattered and has not been brought together in any digestible form. We argue that in order to help the community proceed with more work on diagnostic messages, the literature needs to be presented in a state-of-the-art report. In addition we will synthesize and present the existing evidence for these messages including the difficulties they present and their effectiveness. We will also formulate a set of guidelines based on this evidence that can be used when designing or enhancing diagnostic messages. This work can serve as a starting point for those who wish to conduct research on such messages, those who wish to design better messages or those that aim to measure their effectiveness, more effectively.

When solving programming problems, novices are often not aware of where they are in the problem-solving process. For instance, students who misinterpret the problem prompt will most likely not form a valid conceptual model of the task and fail to make progress towards a working solution. Avoiding such errors, and recovering from them once they occur, requires metacognitive skills that enable students to reflect on their problem-solving processes. For these reasons, developing metacognitive awareness is crucially important for novice students. Previous research has shown that explicitly teaching key steps of programming problem-solving, and having students reflect on where they are in the problem-solving process, can help students complete future programming assignments. Such metacognitive awareness training can be done through personal tutoring, but can be difficult to implement without a high ratio of instructors to students. We explore a more scalable approach, making use of an automated assessment tool, and conduct a controlled experiment to see whether scaffolding the problem-solving process would increase metacognitive awareness and improve student performance. We collected all code submissions by students in both control and experimental groups, as well as data from direct observation using a think-aloud protocol. We found that students who received the intervention showed a higher degree of understanding of the problem prompt and were more likely to complete the programming task successfully.

Most novice programmers are not explicitly aware of the problem-solving process used to approach programming problems and cannot articulate to an instructor where they are in that process. Many are now arguing that this skill, called metacognitive awareness, is crucial for novice learning. However, novices frequently learn in university CS1 courses that employ automated assessment tools (AATs), which are not typically designed to provide the cognitive scaffolding necessary for novices to develop metacognitive awareness. This paper reports on an experiment designed to understand what difficulties novice programmers currently face when learning to code with an AAT. We describe the experiences of CS1 students who participated in a think-aloud study where they were observed solving a programming problem with an AAT. Our observations show that some students mentally augmented the tool when it did not explicitly support their metacognitive awareness, while others stumbled due to the tool's lack of such support. We use these observations to formulate difficulties faced by novices that lack metacognitive awareness, compare these results to other related studies, and look toward future work in modifying AATs.

The difficulty in understanding compiler error messages can be a major impediment to novice student learning. To alleviate this issue, multiple researchers have run experiments enhancing compiler error messages in automated assessment tools for programming assignments. The conclusions reached by these published experiments appear to be conducting. We examine these experiments and propose five potential reasons for the inconsistent conclusions concerning enhanced compiler error messages: (1) students do not read them, (2) researchers are measuring the wrong thing, (3) the effects are hard to measure, (4) the messages are not properly designed, (5) the messages are properly designed, but students do not understand them in context due to increased cognitive load. We constructed mixed-methods experiments designed to address reasons 1 and 5 with a specific automated assessment tool, Athene, that previously reported inconclusive results. Testing student comprehension of the enhanced compiler error messages outside the context of an automated assessment tool demonstrated their effectiveness over standard compiler error messages. Quantitative results from a 60 minute one-on-one think-aloud study with 31 students did not show substantial increase in student learning outcomes over the control. However, qualitative results from the one-on-one think-aloud study indicated that most students are reading the enhanced compiler error messages and generally make effective changes after encountering them.

One common frustration students face when first learning to program in a compiled language is the difficulty in interpreting the compiler error messages they receive. Attempts to improve error messages have produced differing results. Two recently published papers showed conflicting results, with one showing measurable change in student behavior, and the other showing no measurable change. We conducted an experiment comparable to these two over the course of several semesters in a CS1 course. This paper presents our results in the context of previous work in this area. We improved the clarity of the compiler error messages the students receive, so that they may more readily understand their mistakes and be able to make effective corrections. Our goal was to help students better understand their syntax mistakes and, as a reasonable measure of our success, we expected to document a decrease in the number of times students made consecutive submissions with the same compilation error. By doing this, we could demonstrate that this enhancement is effective. After collecting and thoroughly analyzing our own experimental data, we found that--despite anecdotal stories, student survey responses, and instructor opinions testifying to the tool's helpfulness--enhancing compiler error messages shows no measurable benefit to students. Our results validate one of the existing studies and contradict another. We discuss some of the reasons for these results and conclude with projections for future research.

As automated tools for grading programming assignments become more widely used, it is imperative that we better understand how students are utilizing them. Other researchers have provided helpful data on the role automated assessment tools (AATs) have played in the classroom. In order to investigate improved practices in using AATs for student learning, we sought to better understand how students iteratively modify their programs toward a solution by analyzing more than 45,000 student submissions over 7 semesters in an introductory (CS1) programming course. The resulting metrics allowed us to study what steps students took toward solutions for programming assignments. This paper considers the incremental changes students make and the correlating score between sequential submissions, measured by metrics including source lines of code, cyclomatic (McCabe) complexity, state space, and the 6 Halstead measures of complexity of the program. We demonstrate the value of throttling and show that generating software metrics for analysis can serve to help instructors better guide student learning.