Educational puzzles can be a powerful way to develop the situational interest of learners by presenting an authentic and challenging experience. This is important because situational interest has been shown to be a key predictor of engagement and performance. To this end, we developed CS Connections, inspired by the New York Times Connections puzzle game. CS Connections is a tool that provides educators with a flexible way to create custom interactive puzzles in a familiar format.

Timely, personalized feedback is essential for students learning programming. LLM-powered tools like ChatGPT offer instant support, but reveal direct answers with code, which may hinder deep conceptual engagement. We developed CodeAid, an LLM-powered programming assistant delivering helpful, technically correct responses, without revealing code solutions. CodeAid answers conceptual questions, generates pseudo-code with line-by-line explanations, and annotates student’s incorrect code with fix suggestions. We deployed CodeAid in a programming class of 700 students for a 12-week semester. A thematic analysis of 8,000 usages of CodeAid was performed, further enriched by weekly surveys, and 22 student interviews. We then interviewed eight programming educators to gain further insights. Our findings reveal four design considerations for future educational AI assistants: D1) exploiting AI’s unique benefits; D2) simplifying query formulation while promoting cognitive engagement; D3) avoiding direct responses while encouraging motivated learning; and D4) maintaining transparency and control for students to asses and steer AI responses.

In 2017, Edwards et al. studied a large corpus of Java programs collected through an automated submission and assessment system that integrated static analysis feedback. They found that errors reported were most commonly related to formatting, but that the frequency of errors they categorized as "Coding Flaws" correlated with program correctness grades. They argued that static analysis feedback could detect problems relating to code correctness and could therefore be useful beyond evaluating conformance to style rules, but that students may overlook non-cosmetic error messages because of the relative volume of formatting errors. In this paper we perform a conceptual replication of the Edwards et al. study with 1270 CS1 students learning Python. We confirm that almost a decade later and even after being instructed to use the auto-formatting options within their IDE, students still encounter mostly formatting errors when using a static analysis tool. We find that the second- most common category of errors detected are "Coding Flaws", and, like Edwards et al., that the frequency of coding flaws identified by the static analysis tool correlates to program correctness. When we examine trends based on levels of prior programming experience, we find that all students tend to make more formatting errors than other kinds of errors, but that students with no prior programming experience have more errors reported across all error categories.

Studies show that both test anxiety (TA) and self-efficacy (SE) have an impact on academic performance and that different students experience TA at different levels. For example, TA has consistently been shown to be higher and SE to be lower for women than men. In our study, we explore how TA and SE are experienced by CS1 students in a computer-based testing environment and how this changes by demographic group. We build on our prior work, focusing on groups of students based on their prior programming experience (PE) and whether or not they are first in their family (FIF) to attend a post-secondary institution. We measure self-reported TA and SE at five points and relate these to grades. We find that while novices report higher TA, lower SE, and lower grades than their peers, TA is only correlated with grade for experienced students.

The recent advent of highly accurate and scalable large language models (LLMs) has taken the world by storm. From art to essays to computer code, LLMs are producing novel content that until recently was thought only humans could produce. Recent work in computing education has sought to understand the capabilities of LLMs for solving tasks such as writing code, explaining code, creating novel coding assignments, interpreting programming error messages, and more. However, these technologies continue to evolve at an astonishing rate leaving educators little time to adapt. This working group seeks to document the state-of-the-art for code generation LLMs, detail current opportunities and challenges related to their use, and present actionable approaches to integrating them into computing curricula.

In the context of problem sets in first- and second-year computer science theory courses, we investigate the factors influencing students' decisions to work individually or in a small group. Through analysis of open-ended survey responses from over 1,300 students, we have gained a more nuanced understanding of these factors. We observed three categories of factors: workload and time management, optimizing learning and assignment marks, and social and affective factors. We identified two modifiers, online learning and previous group experiences, that amplify the impact of factors in the three main categories. We highlight notable student quotations and discuss barriers to group formation.

Mental health crises among post-secondary Computer Science students are persistent and growing concerns as students are prone to high stress levels and feelings of anxiety and depression [1]. Although this issue is not unique to Computer Science, the prevalence of mental health issues in STEM [1] makes it extremely important for CS educators to find ways to support student well-being within their courses. One potential technique to help alleviate some of the negative feelings students face is for courses to incorporate mental wellness interventions that aim to improve students’ psychological well-being. This poster discusses an attempt at engaging students in such an intervention – weekly gratitude journaling – in an online CS1 course.

We investigate the impact of allowing students to optionally work in small groups on problem sets in first- and second-year computer science theory courses. After each homework assignment, students reported on their experience working on that problem set either individually or in a group of two or three. Over 1,300 students from two courses participated. We explore who chooses to work in a group and why, how students work in groups, and differences in learning, drop-rates, help-seeking, and satisfaction with problem set submissions between groups and students working individually.

Metacognition and self-regulation are important skills for successful learning and have been discussed and researched extensively in the general education literature for several decades. More recently, there has been growing interest in understanding how metacognitive and self-regulatory skills contribute to student success in the context of computing education. This article presents a thorough systematic review of metacognition and self-regulation work in the context of computer programming and an in-depth discussion of the theories that have been leveraged in some way. We also discuss several prominent metacognitive and self-regulation theories from the literature outside of computing education—for example, from psychology and education—that have yet to be applied in the context of programming education. In our investigation, we built a comprehensive corpus of papers on metacognition and self-regulation in programming education, and then employed backward snowballing to provide a deeper examination of foundational theories from outside computing education, some of which have been explored in programming education, and others that have yet to be but hold much promise. In addition, we make new observations about the way these theories are used by the computing education community, and present recommendations on how metacognition and self-regulation can help inform programming education in the future. In particular, we discuss exemplars of studies that have used existing theories to support their design and discussion of results as well as studies that have proposed their own metacognitive theories in the context of programming education. Readers will also find the article a useful resource for helping students in programming courses develop effective strategies for metacognition and self-regulation.

Metacognition and self-regulation are popular areas of interest in programming education, and they have been extensively researched outside of computing. While computing education researchers should draw upon this prior work, programming education is unique enough that we should explore the extent to which prior work applies to our context. The goal of this systematic review is to support research on metacognition and self-regulation in programming education by synthesizing relevant theories, measurements, and prior work on these topics. By reviewing papers that mention metacognition or self-regulation in the context of programming, we aim to provide a benchmark of our current progress towards understanding these topics and recommendations for future research. In our results, we discuss eight common theories that are widely used outside of computing education research, half of which are commonly used in computing education research. We also highlight 11 theories on related constructs (e.g., self-efficacy) that have been used successfully to understand programming education. Towards measuring metacognition and self-regulation in learners, we discuss seven instruments and protocols that have been used and highlight their strengths and weaknesses. To benchmark the current state of research, we examined papers that primarily studied metacognition and self-regulation in programming education and synthesize the reported interventions used and results from that research. While the primary intended contribution of this paper is to support research, readers will also learn about developing and supporting metacognition and self-regulation of students in programming courses.

The goal of many computing education researchers is to improve how computing is taught and learned. To do that, researchers must engage with teachers, coaches, and mentors who design instructional materials and deliver lessons. However researchers may not be investigating problems that are directly of interest or utility to practitioners, and thus may not deliver results that are as impactful as possible in their contexts. To find out what research most interests today's practitioners, we conducted a two-stage survey. The first stage gathered questions that practitioners want researchers to investigate, and the second stage ranked these questions in terms of importance. We found that today's practitioners are more interested in student behavior, student understanding, and pedagogy than in languages and tools, curriculum, and inclusivity, and that there is little overlap between the questions ranked as highly interesting by researchers and those ranked highly by practitioners. Our results indicate that researchers need to better communicate why the questions they are pursuing are important, look for opportunities to collaborate with those who teach but have little direct connection with research, and examine the relevance of their research questions to practitioners.

Studies in other disciplines demonstrate that writing summaries of videos or readings is an effective strategy for increasing student understanding. We explore a related strategy in computer science where we ask students to write explanations of program behaviour. Our experiments are conducted through an online application used by students to prepare for class by watching videos and solving follow-up exercises. With students randomly assigned to treatment groups, we evaluate the effect of different kinds of video follow-up questions by analyzing pre- and post-performance on subsequent select-all-that-apply multiple-choice questions. In experiment one, we found no difference in post-performance between students who wrote explanations and those who were not asked to do so. When the same prompt for explanations was accompanied by a motivational sentence, the quality of the student answers increased, but post-performance was unchanged. In a second experiment, students received either no experimental questions, questions requiring a written explanation or short-answer questions with a single correct response. Again, we found no difference in post-performance across the different treatments. This non-significant result might be explained by the post-performance question difficulty -- the question might not be hard enough to reveal differences in understanding. The question timing might also be a factor, since students complete all the questions soon after watching the video and there may not be enough time for students to need the extra retention benefit gained by doing the experimental questions.

Online coding tools are an increasingly common feature of programming courses, providing students with rapid feedback and flexible practice opportunities and providing instructors with useful analytics. However, little research has explored the complexity of online exercises provided to students and the order in which students are exposed to new ideas. In this paper, we investigate the benefits of using a short sequence of practice exercises, each targeting a distinct topic, prior to having students solve a goal task that combines the concepts. As expected, we find students solve the goal task with fewer errors and in less time after completing the practice tasks. However, we also find that the practice tasks reduce the likelihood of students delaying work on the goal task, and these effects are particularly large for less-experienced students.

Replication research is rare in CS education. For this reason, it is often unclear to what extent our findings generalize beyond the context of their generation. The present paper is a replication and extension of Achievement Goal Theory research on CS1 students. Achievement goals are cognitive representations of desired competence (e.g., topic mastery, outperforming peers) in achievement settings, and can predict outcomes such as grades and interest. We study achievement goals and their effects on CS1 students at six institutions in four countries. Broad patterns are maintained --- mastery goals are beneficial while appearance goals are not --- but our data additionally admits fine-grained analyses that nuance these findings. In particular, students' motivations for goal pursuit can clarify relationships between performance goals and outcomes.

We report our experience developing and teaching a computing elective course for students enrolled in a Doctor of Medicine (MD) program. Students participated in a series of workshops to learn and practice programming, and gained additional experience by completing programming assignments. Students then participated in a novel seminar series delivered by experts who each discussed one application of computing to medicine. Each seminar included a corresponding programming project where students worked with the ideas introduced in the seminar and practiced their newly-acquired programming skills. We found that by streaming the students into levels based on prior experience, carefully scaffolding project handouts, and having each seminar co-led by a faculty member, we are able to support students --- even beginners --- to succeed. Students report that the topics are relevant, they appreciate the medical context of the programming exercises, and they would recommend the program to others.

A good suite of test inputs is an indispensable tool both for manual and automated assessment of student submissions to programming assignments. Yet, without a way to evaluate our test suites, it is difficult to know how well we are doing, much less improve our practice. We present a technique for evaluating a hand-generated test suite by comparing its ability to find defects against that of a test suite generated automatically using Constraint Logic Programming (CLP). We describe our technique and present a case study using student submissions for an assignment from a second-year programming course. Our results show that a CLP-generated test suite was able to identify significant defects that the instructor-generated suite missed, despite having similar code coverage.

An exam wrapper is a structured activity that students engage in after their instructor has graded and returned an exam, and is designed to promote self-reflection and improve study practices. This paper describes two studies examining the efficacy and student perceptions of exam wrappers. The studies were conducted at two major Canadian universities, using complementary research designs. We report that neither study produced evidence that exam wrappers have a significant effect on final exam scores or on course drop rates. However, we also find that the use of wrappers was associated with improved rates of test pickup and increased scores on a course evaluation question regarding the fairness of evaluation methods. Given these results, we advise instructors who are considering the use of exam wrappers to review the evidence for other possible interventions that may more effectively serve the same goals.

It is commonly thought that CS grades are bimodal. We statistically analyzed 778 distributions of final course grades from a large research university, and found only 5.8% of the distributions passed tests of multimodality. We then devised a psychology experiment to understand why CS educators believe their grades to be bimodal. We showed 53 CS professors a series of histograms displaying ambiguous distributions and asked them to categorize the distributions. A random half of participants were primed to think about the fact that CS grades are commonly thought to be bimodal; these participants were more likely to label ambiguous distributions as "bimodal". Participants were also more likely to label distributions as bimodal if they believed that some students are innately predisposed to do better at CS. These results suggest that bimodal grades are instructional folklore in CS, caused by confirmation bias and instructor beliefs about their students.

Enrollment in post-secondary online courses has been increasing, but several studies have found that the drop rates in online courses are higher than in face-to-face. In our previous study comparing an online section of CS1 with a face-to-face flipped section, we also found the drop rate higher in the online section. Given that we plan to continue offering online options for our students, we aim to identify factors associated with success in online CS1. In this paper, we examine factors that are under students' own control such as how fully they participate in ungraded but important learning activities, and other factors that we may be able to manipulate and improve, such as students' skills for self-regulated learning, and their sense of community in the course. We found important differences between the online and flipped sections regarding what behaviours and attributes were associated with success. While completion of unmarked practice exercises was a factor for both sections, test anxiety and self-efficacy were factors only for the online section, and intrinsic goal orientation was a factor only for the flipped section.

Increasing enrollments and adoption of online resources have encouraged the use of multiple choice questions as a means of providing scalable assessment. However, in contexts where formative feedback is desired, standard multiple choice questions may lead students to a false sense of confidence -- a result of their small solution space and the temptation to guess. We propose the use of multiple-answer multiple choice questions in situations where formative feedback is desired and present evidence that these questions are well suited for that role.

In addition to their role as a summative measure, midterm tests can provide formative feedback that can be used by students to identify areas of weakness and adjust studying approaches. Unfortunately, low levels of test pickup often preclude this type of learning from tests. Even when students do collect their marked tests, it is unclear how much they reflect on or learn from the feedback.

When students can choose to take a course online or face-to-face, who chooses each format? Why do students select one format instead of the other? We compare an online section of CS1 to a concurrent face-to-face section and find that the populations in the two sections are very different. In particular, students in the online section have less prior programming experience and are less likely to intend to major in computer science. We also examine the reasons why students choose their section, many of which relate to convenience, desire for interaction with others, and degree of familiarity with the course material. Finally, we compare course outcomes for the two sections. We find significant differences in drop rates, but not in final exam scores. We investigate whether the differences we find can be explained by differences in the populations who choose to take the course online vs face-to-face.

How to increase diversity in computer science is an important open question in CS education. A number of best practices have been suggested based on case studies; however, for scaling these efforts up in a sustainable fashion, it remains unclear which types of initiatives are most effective in which contexts. We examine gender diversity initiatives in CS education from a policy analysis perspective, adapting McDonnell and Elmore's 1987 notion of policy instruments, wherein the initiative is the unit of analysis. We present a conceptual framework for categorizing the different policy instruments by a cross of 'leverage' and 'targetedness', and discuss how different types of initiatives will scale. We argue that universally-targeted, high-leverage initiatives are most important for scaling up diversity initiatives in CS education, with medium-leverage being a stepping stone to high leverage change.

The C programming language is an important piece of many undergraduate CS programs, as it provides an environment for interacting directly with memory and exploring systems-programming concepts. However, while many common introductory languages have rich tools that support instruction, C has received relatively little attention [2, 1]. To provide students with rapid feedback and tools for understanding C, we have extended PCRS, a web-based platform for deploying programming exercises and content such as videos. Students submit C code to solve programming exercises and receive immediate feedback generated by running the submission against a set of instructor-defined testcases. Students also have access to graphical traces of execution, so they can explore how their code manipulates memory. The system has been deployed to two second-year systems-programming courses with a total enrollment over 600, and a set of modules, consisting of videos and exercises, is being developed for use by the community.

Much attention has been paid to the failure rate in CS1 and attrition between CS1 and CS2. In our study of 1236 CS1 students, we examine subgroups of students, to find out how characteristics such as prior experience and reason for taking the course influence who drops, fails, or passes, and who continues on to CS2. We also examine whether student characteristics influence outcomes differently in traditional vs. inverted offerings of the course. We find that more students in the inverted offering failed the midterm test, but those who failed were much more likely to either drop the course or recover and ultimately pass the course. While we find no difference between the offerings in the overall drop-fail-pass rates or in the percentage and types of students who go on to take CS2, there is a significant, widely felt, boost in exam grades in the inverted offering.

We compare a traditional CS1 offering with an inverted offering delivered the following year to a comparable student population. We measure student attitudes, grades, and final course outcomes and find that, while students in the inverted offering do not report increased enjoyment and are no more likely to pass, learning as measured by final exam performance increases significantly. This increase is not simply a function of a more onerous inverted offering, as students report spending similar time per week in the traditional and inverted offerings. Contrary to our hypotheses, however, we find no evidence that the the inverted offering disproportionally helps beginners or those not fully fluent in English.

We present a history of female participation in North American CS, with a focus on the social forces involved. For educators to understand the status quo, and how to change it, we must understand the historical forces that have led us here. We begin with the female ''computers'' of the 19th century, then cover the rise of computing machines, establishment of CS, and a history of CS education with regard to gender. In our discussion of academic CS, we contemplate academic generations of female computer scientists and describe their differential experiences.

This case study explores an inverted classroom offering of an introductory programming course (CS1). Students prepared for lecture by watching short lecture videos and completing required in-video quiz questions. During lecture, the students worked through exercises with the support of the instructor and teaching assistants. We describe the course implementation and its assessment, including pre- and post-course surveys. We also discuss lessons learned, modifications that we plan to make for the next offering, and recommendations for others teaching inverted courses.

In the interest of better understanding why many students fail to complete CS1 successfully, we analyze a class of 555 CS1 students, 127 of whom either failed or dropped the course. We find that students with previous experience are more likely to pass even if the experience is not a formal programming course. Contrary to intuition, students who drop have enrolled in fewer courses and have fewer hours of non-school commitments than those who stay. We present a classification scheme to categorize student outcomes based on their performance at a course midpoint.

Comparing and contrasting different solution approaches is known in math education and cognitive science to increase student learning -- what about CS? In this experiment, we replicated work from Rittle-Johnson and Star, using a pretest--intervention--posttest--follow-up design (n=241). Our intervention was an in-class workbook in CS2. A randomized half of students received questions in a compare-and-contrast style, seeing different code for different algorithms in parallel. The other half saw the same code questions sequentially, and evaluated them one at a time. Students in the former group performed better with regard to procedural knowledge (code reading & writing), and flexibility (generating, recognizing & evaluating multiple ways to solve a problem). The two groups performed equally on conceptual knowledge. Our results agree with those of Rittle-Johnson and Star, indicating that the existing work in this area generalizes to CS education.

Every time I re-use a handout, I look it over and make a few little "improvements". I play around with code demos and entertain myself with different slide transitions. However, inevitably, I return to the conclusion that most of what my students learn in my course comes from the assignments. Great assignments are hard to dream up and time-consuming to develop. With that in mind, the Nifty Assignments session is all about promoting and sharing the ideas and ready-to-use materials of successful assignments.

From an ongoing research project on teaching hash tables using worked examples, we present four preliminary observations. First, that rather than there being a small set of common misconceptions, student misconceptions are diverse and often unique to the student. Second, that students' naive language about hash tables when given a pretest is influenced by words from the Internet (e.g. "hashtag"). Third, we observed that students' language on concept questions evolves with repeated testing, becoming more conceptually accurate but technically less precise. And finally, that students' code code correctness is not correlated to code style, but is correlated to how students performed on the concept questions.

This paper presents observations about teaching program tracing to novices drawn from a study of knitting patterns. Due to changes in audience, knitting patterns have evolved from vague, chatty discourse written for experts to precise, line-by-line procedures that are akin to programs. The modern knitting community has developed numerous conventions for articulating iteration, expressing conditions, and documenting design decisions. "Executing" one of these patterns is analogous to tracing, since the knitter must demonstrate understanding of the instructions. We argue that the conventions adopted by knitters to make their patterns more understandable to non-experts provide useful insight to computer scientists teaching novices. Our observations suggest that phrasing conditions as termination cases ("until" instead of "while") and partially unrolling loops may help beginners understand code and that some structures, like parameters to functions, may be unfamiliar because they have no common analog.

In this paper, we explore the use of sequences of small code writing questions ("concept questions") designed to incrementally evaluate single programming concepts. We report on a study of student performance on a CS1 final examination that included a traditional code-writing question and four intentionally corresponding concept questions. We find that the concept questions are significant predictors of performance on both the corresponding code-writing question and the final exam as a whole. We argue that concept questions provide more accurate formative feedback and simplify marking by reducing the number of variants that must be considered. An analysis of responses categorized by the students' previous programming experience suggests that inexperienced students have the most to gain from the use of concept questions.

I worry over topics for the syllabus, fretting over demos and presentations. And yet, I always come back to the fact that most of what my students learn and remember from my course comes from the assignments. Great assignments are hard to dream up and time-consuming to develop. With that in mind, the Nifty Assignments session is all about promoting and sharing the ideas and concrete materials of successful assignments.

Many factors have been cited for poor performance of students in CS1. To investigate how assessment mechanisms may impact student performance, nine experienced CS1 instructors reviewed final examinations from a variety of North American institutions. The majority of the exams reviewed were composed predominantly of high-value, integrative code-writing questions, and the reviewers regularly underestimated the number of CS1 concepts required to answer these questions. An evaluation of the content and cognitive requirements of individual questions suggests that in order to succeed, students must internalize a large amount of CS1 content. This emphasizes the need for focused assessment techniques to provide students with the opportunity to demonstrate their knowledge.

Instructors often put students into groups for coursework. Several tools exist to facilitate this process, but they typically limit the criteria one can use for forming groups. We have defined a general mathematical model for group formation: a set of attribute types, group-formation criteria, and fitness measures. We have implemented an optimizer that uses an evolutionary algorithm to create groups according to the instructor's criteria. Our experiments support the hypothesis that, even with a general model, reasonably optimal solutions to the group-formation problem can be found in reasonable time. Several instructors have used the tool to form groups for their courses. In all cases, they were impressed by the expressiveness of the model and pleased with the quality of the groups produced.

In order to address the under-representation of women in Computer Science, we have created a program for middle-school girls that specifically aims to change their attitudes about CS and encourages them to see it as a potential career. Our assessment of the program shows that it did indeed have a significant, positive impact and suggests that this was still in effect three months later. This paper describes the program and its assessment, and makes suggestions for those considering offering a similar program.

Trying to improve undergraduate teaching based on feedback collected by traditional student course evaluations can be a frustrating experience. Unclear, contradictory and ill-informed student comments leave instructors confused and discouraged. We designed and then implemented an evaluation mechanism where an independent CS faculty peer visits a lecture and holds an evaluation discussion with the students. These facilitated discussions begin by looking at overall strengths and weaknesses for the course but quickly focus on the key student concerns and suggestions for improvement. After conducting thirty four facilitated discussions, we find them appreciated by students who feel heard and valued. A survey of participating faculty indicates that the written discussion report is more useful to them than standard student survey results. Faculty report that they have made changes based on the recommendations and limited quantitative data suggests that teaching has improved and its value in the departmental culture has increased. In this paper we describe the evaluation process, discuss our experiences and offer some concrete suggestions for those who might want to try this approach in their own department.

This paper focuses on the use of code features for automatic plagiarism detection. Instead of the text-based analyses employed by current plagiarism detectors, we propose a system that is based on properties of assignments that course instructors use to judge the similarity of two submissions. This system uses neural network techniques to create a feature-based plagiarism detector and to measure the relevance of each feature in the assessment. The system was trained and tested on assignments from an introductory computer science course, and produced results that are comparable to the most popular plagiarism detectors.