Researchers use replication to confirm, strengthen, and advance Computing Education Research (CER). However, prior research shows that replication is infrequently used in CER, even though the community encourages its use. Previous research suggests that the CER community uses different terms to describe replication, which we aim to confirm in this Working Group (WG). We will conduct a Systematic Literature Review (SLR) across influential international CER venues to understand how researchers present and conduct replication studies, possibly identifying venues receptive to papers applying this research design. In addition, we will interview Computing Education researchers, conference leaders, and journal editors to understand their experiences and perceptions with replication. We expect to collect suggestions and recommendations on how CER can encourage more replication in future studies. Our work will highlight and confirm the terms the CER community uses to present replication studies, enabling researchers and educators to better identify these studies.

GenAI is playing an increasingly important role in computing courses at all levels, offering new opportunities to support teaching and learning. However, using GenAI effectively raises important concerns regarding trust, academic integrity, and broader social and ethical dimensions. This Working Group was formed to report on the current state of the art in using GenAI in upper-level computing courses to aid educators. The working group will undertake a methodological review of published work and solicit input from the computing educational community as part of the report.

Every year, millions of students learn how to write programs. Learning activities for beginners almost always include programming tasks that require a student to write a program to solve a particular problem. When learning how to solve such a task, many students need feedback on their previous actions, and hints on how to proceed. In the case of programming, the feedback should take the steps a student has taken towards implementing a solution into account, and the hints should help a student to complete or improve a possibly partial solution. Only a limited number of learning environments for programming give feedback and hints on intermediate steps students take towards a solution, and little is known about the quality of the feedback provided. To determine the quality of feedback of such tools and to help further developing them, we create and curate data sets that show what kinds of steps students take when solving programming exercises for beginners, and what kind of feedback and hints should be provided. This working group aims to 1) select or create several data sets with steps students take to solve programming tasks, 2) introduce a method to annotate students' steps in these data sets, 3) attach feedback and hints to these steps, 4) set up a method to utilize these data sets in various learning environments for programming, and 5) analyse the quality of hints and feedback in these learning environments.

Diagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for decades. Unfortunately these messages which include error, warning, and runtime messages, present substantial difficulty and could be more effective, particularly for novices. Recent years have seen increased number of papers in the area including studies on the effectiveness of these messages, improving or enhancing them, and their usefulness as a part of programming process data that can be used to predict student performance. Despite this increased interest, the long history of literature is quite scattered and has not been brought together in any digestible form. We argue that in order to help the community proceed with more work on diagnostic messages, the literature needs to be presented in a state-of-the-art report. In addition we will synthesize and present the existing evidence for these messages including the difficulties they present and their effectiveness. We will also formulate a set of guidelines based on this evidence that can be used when designing or enhancing diagnostic messages. This work can serve as a starting point for those who wish to conduct research on such messages, those who wish to design better messages or those that aim to measure their effectiveness, more effectively.

The experience of transitioning into and starting higher education is a very individual one, with some applicants viewing the prospect of higher education as an unknown entity; for those who are first in their family or community to consider higher education, this can seem an "alien environment". These are just some of the issues that lead to applicants experiencing levels of concern when considering a transition into higher education. This international working group proposes to consider concerns across a wide range of participants: namely students who are in the process of transitioning into a higher education environment, and students who have recently completed this transition.

Contextualization of problems is widely studied in mathematics education. In computer science it is taken for granted that authentic, contextualized programming assignments will increase student interest and therefore enhance performance in programming assignments. This paper examines whether contextualization is, in fact, beneficial for students. We present a study that compares novice programmers' ability to code a solution given two versions of a problem. One version is contextualized, the other is non-contextualized, using "just the numbers." The results presented indicate that there is no difference in success rates for the two types of programming assignments.

Peer Instruction (PI) is a student-centric pedagogy in which students move from the role of passive listeners to active participants in the classroom. Over the past five years, there have been a number of research articles regarding the value of PI in computer science. The present work adds to this body of knowledge by examining outcomes from seven introductory programming instructors: three novices to PI and four with a range of PI experience. Through common measurements of student perceptions, we provide evidence that introductory computing instructors can successfully implement PI in their classrooms. We find encouraging minimum (74%) and average (92%) levels of success as measured through student valuation of PI for their learning. This work also documents and hypothesizes reasons for comparatively poor survey results in one course, highlighting the importance of the choice of grading policy (participation vs. correctness) for new PI adopters.

This study examines the extent to which novice computing students with minimal computer science coursework and no training in user interface (UI) evaluation consider UI concepts such as usability, user experience, and the context in which software will be used when evaluating an interface. In analyzing the responses of 149 novice computer science students who were asked to evaluate two interfaces for converting temperatures between Fahrenheit and Celsius, we observed that students generally considered usability and user experience factors, but were less likely to consider context. For educators, this exact task could be given to a class in order to initiate discussion of user-centered design; the study also provides a framework for structuring the discussion. More generally, the results of this study provide insight into some opportunities and challenges in teaching good interface design and evaluation skills.

This paper investigates what students understand about algorithm efficiency before receiving any formal instruction on the topic. We gave students a challenging search problem and two solutions, then asked them to identify the more efficient solution and to justify their choice. Many students did not use the standard worst-case analysis of algorithms; rather they chose other metrics, including average-case, better for more cases, better in all cases, one algorithm being more correct, and better for real-world scenarios. Students were much more likely to choose the correct algorithm when they were asked to trace the algorithms on specific examples; this was true even if they traced the algorithms incorrectly.

As the third in a series of projects investigating commonsense computing -- the relevant knowledge that students have before any formal study of computing -- we examine students' commonsense understanding of concurrency. Specifically, we replicated (with modifications) an experiment by Ben-David Kolikant. [2] Ben-David Kolikant's data were gathered from high-school seniors who had previously studied computing, at the beginning of an advanced class in concurrent and distributed programming. Modifying one of her questions to reflect our students' lack of background, we asked students at five different institutions, in the first week of CS1, to describe in English the problems that might arise when more than one person is selling seats to a concert.Almost all students (97%) identified the problem of interest -- that a race condition may occur between sellers. 73% of students identified at least one possible solution. We found that the categorizations developed by Ben-David Kolikant were also meaningful when applied to our data, that our beginning CS1 students are more likely to give centralized solutions (as opposed to decentralized ones) than Ben-David Kolikant's concurrency students, and that the granularity of solutions is finer among the more experienced students.

Abstract: This paper presents a case study of the use of a repeated single-criterion card sort with an unusually large, diverse participant group. The study, whose goal was to elicit novice programmers' knowledge of programming concepts, involved over 20 researchers from four continents and 276 participants drawn from 20 different institutions. In this paper we present the design of the study and the unexpected result that there were few discernible systematic differences in the population. The study was one of the activities of the National Science Foundation funded Bootstrapping Research in Computer Science Education project (2003).

Previous Computer Science Education research incorporating Common Learning Experiences (CLEs) into freshman level breadth-first Computer Science course inspired the author to undertake similar work for a depth-first course. This paper presents a set of CLEs developed for this purpose, namely the 'Living Flowcharts CLEs', along with a discussion of the pilot study and the preliminary results.