Modern programming tools use generative AI (GenAI) to suggest code to the user as they type, interrupting their problem-solving behavior and undermining the development of their programming critical thinking skills. In this paper, we present a scaffolded programming exercise designed to support student differentiation between good and bad GenAI code suggestions based on negative expertise–that identifying why an answer is wrong is part of developing conceptual knowledge. We compare a version of the tool that showed one suggestion (correct or not), to a version that showed three suggestions (one of which was correct). We present results on performance and error rates as well as qualitative findings centered on Pintrich and DeGroot’s theory of self-regulation. Students reported that the single suggestion version better aligned with industry tools and presented a lower cognitive load. Students also reported that the multiple suggestion version caused them to slow down and think critically about the line under consideration, the overall purpose of the code, and the benefits of planning.

Publisher Copyright: © 2026 Owner/Author.

Code quality is an essential aspect of programming education, impacting source code maintainability and readability. However, providing manual feedback is time-consuming for instructors, and automated approaches to code quality feedback remain inconsistent and underdeveloped. We adapt a defect catalogue for novice Python programmers and construct an automated detection pipeline that integrates multiple static analysis tools and a custom natural language identifier detector. Using a standardised set of defect examples, we evaluate the detection coverage of the selected tools. Our results show that our integrated toolset detects 64 defect types, with 30 undetected. Applying the pipeline to over 86,000 student submissions, we analyse the prevalence and distribution of code quality defects in real coursework. To better align with pedagogical priorities, we introduce a multi-dimensional prioritisation framework that combines defect frequency, student coverage, and instructor-rated importance. Our findings provide a reference for improving feedback mechanisms and instructional strategies for code quality in introductory programming education.

• Motivation mainly focuses on enhancing understanding of abstract concepts. • Most research focuses on transformations, leaving other topics underexplored. • Unity-based mobile tools with marker tracking are widely used. • Evidence shows improved performance, but comparisons with traditional methods are mixed. • Mixed evidence on cognitive load and recurring usability challenges.

Grading computer graphics programming assessments and generating formative and summative feedback can require significant effort on the part of human experts.Since these assessments generate visual outputs that can be static or animated, determining correctness may be subjective.For feedback to be effective, it must be delivered in a timely manner.This can be a challenge for introductory computer graphics-based courses since cohort size can be substantial, errors in visual output can be subtle, and causes of errors are often not obvious.In this paper, we explore the feasibility of an automated system for marking visual output and providing program implementation feedback for learners in an introductory computer graphics-based design course in three short programming assessments, including static and animated scenes.To assess the effectiveness of our approach, we compare the marks generated by our tool with those assigned by a human expert.We show that it is possible to automate marking, providing both a grade based on the visual output and formative feedback on source code in the style of a human marker.This can improve objective consistency, grade reproducibility, and reduce marking time, enabling a course to scale to support large cohorts without the need for more resourcing for human markers.We describe lessons learnt and potential pitfalls to assist educators with introducing automated marking for their courses.Finally, we identify areas for future refinement and development of our automated system.

Code reading and comprehension skills are essential for novices learning programming, and explain-in-plain-English tasks (EiPE) are a well-established approach for assessing these skills. However, manual grading of EiPE tasks is time-consuming and this has limited their use in practice. To address this, we explore an approach where students explain code samples to a large language model (LLM) which generates code based on their explanations. This generated code is then evaluated using test suites, and shown to students along with the test results. We are interested in understanding how automated formative feedback from an LLM guides students' subsequent prompts towards solving EiPE tasks. We analyzed 177 unique attempts on four EiPE exercises from 21 students, looking at what kinds of mistakes they made and how they fixed them. We found that when students made mistakes, they identified and corrected them using either a combination of the LLM-generated code and test case results, or they switched from describing the purpose of the code to describing the sample code line-by-line until the LLM-generated code exactly matched the obfuscated sample code. Our findings suggest both optimism and caution with the use of LLMs for unmonitored formative feedback. We identified false positive and negative cases, helpful variable naming, and clues of direct code recitation by students. For most students, this approach represents an efficient way to demonstrate and assess their code comprehension skills. However, we also found evidence of misconceptions being reinforced, suggesting the need for further work to identify and guide students more effectively.

Providing timely and personalized feedback to large numbers of students is a long-standing challenge in programming courses. Relying on human teaching assistants (TAs) has been extensively studied, revealing a number of potential shortcomings. These include inequitable access for students with low confidence when needing support, as well as situations where TAs provide direct solutions without helping students to develop their own problem-solving skills. With the advent of powerful large language models (LLMs), digital teaching assistants configured for programming contexts have emerged as an appealing and scalable way to provide instant, equitable, round-the-clock support. Although digital TAs can provide a variety of help for programming tasks, from high-level problem solving advice to direct solution generation, the effectiveness of such tools depends on their ability to promote meaningful learning experiences. If students find the guardrails implemented in digital TAs too constraining, or if other expectations are not met, they may seek assistance in ways that do not help them learn. Thus, it is essential to identify the features that students believe make digital teaching assistants valuable. We deployed an LLM-powered digital assistant in an introductory programming course and collected student feedback ($n=813$) on the characteristics of the tool they perceived to be most important. Our results highlight that students value such tools for their ability to provide instant, engaging support, particularly during peak times such as before assessment deadlines. They also expressed a strong preference for features that enable them to retain autonomy in their learning journey, such as scaffolding that helps to guide them through problem-solving steps rather than simply being shown direct solutions.

Parsons problems provide useful scaffolding for introductory programming students learning to write code. However, generating large numbers of high-quality Parsons problems that appeal to the diverse range of interests in a typical introductory course is a significant challenge for educators. Large language models (LLMs) may offer a solution, by allowing students to produce on-demand Parsons problems for topics covering the breadth of the introductory programming curriculum, and targeting thematic contexts that align with their personal interests. In this paper, we introduce PuzzleMakerPy, an educational tool that uses an LLM to generate unlimited contextualized drag-and-drop programming exercises in the form of Parsons Problems, which introductory programmers can use as a supplemental learning resource. We evaluated PuzzleMakerPy by deploying it in a large introductory programming course, and found that the ability to personalize the contextual framing used in problem descriptions was highly engaging for students, and being able to customize the programming topics was reported as being useful for their learning.

Characterising code quality is a challenge that was addressed by Börstler et al. 's working group in 2017. As emerged from their study, educators, developers and students have different perceptions of the manifold aspects involved, and a major conclusion of that WG was that "code quality should be discussed more thoroughly in educational programs" [2, p. 70]. However, the lack of materials and the time constraints have slowed down progress in that regard.

Folk-wisdom in software engineering suggests that small functions that adhere to the principle of single-responsibility have several advantages over longer, monolithic functions, including improvement in code comprehension. Despite this widespread view, empirical research on the impact of functional decomposition on understanding code is sparse, yet it is central to software development practices.

Large language models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards reading, comprehending and evaluating LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.

Introductory programming courses often require students to solve many small programming exercises as part of their learning. Researchers have previously suggested that the context used in the problem description for these exercises is likely to impact student engagement and motivation. Furthermore, supplying programming exercises that use a broad range of contexts or even allowing students to select contexts to personalize their own exercises, may support the interests of a diverse student population. Unfortunately, it is time-consuming for instructors to create large numbers of programming exercises that provide a wide range of contextualized problems. However, recent work has shown that large language models may be able to automate the mass production of programming exercises, reducing the burden on instructors. In this research, we explore the potential of OpenAI's GPT-4 to create high-quality and novel programming exercises that implement various contexts. Finally, through prompt engineering, we compare different prompting strategies used to generate many programming exercises with various contextualized problem descriptions and then evaluate the quality of the exercises generated.

Teaching large cohorts in upper-level computing courses is challenging, as providing rapid feedback and marking at scale is difficult without significant resources. Many institutions lack funds to employ a large number of skilled markers or such markers are simply not available.

Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.

The recent advent of highly accurate and scalable large language models (LLMs) has taken the world by storm. From art to essays to computer code, LLMs are producing novel content that until recently was thought only humans could produce. Recent work in computing education has sought to understand the capabilities of LLMs for solving tasks such as writing code, explaining code, creating novel coding assignments, interpreting programming error messages, and more. However, these technologies continue to evolve at an astonishing rate leaving educators little time to adapt. This working group seeks to document the state-of-the-art for code generation LLMs, detail current opportunities and challenges related to their use, and present actionable approaches to integrating them into computing curricula.

The introductory programming sequence has been the focus of much research in computing education. The recent advent of several viable and freely-available AI-driven code generation tools present several immediate opportunities and challenges in this domain. In this position paper we argue that the community needs to act quickly in deciding what possible opportunities can and should be leveraged and how, while also working on overcoming otherwise mitigating the possible challenges. Assuming that the effectiveness and proliferation of these tools will continue to progress rapidly, without quick, deliberate, and concerted efforts, educators will lose advantage in helping shape what opportunities come to be, and what challenges will endure. With this paper we aim to seed this discussion within the computing education community.

Student performance prediction aims to build models to help educators identify struggling students so they can be better supported. However, prior work in the space frequently evaluates features and models on data collected from a single semester, of a single course, taught at a single university. Without evaluating these methods in a broader context there is an open question of whether or not performance prediction methods are capable of generalising to new data. We test three methods for evaluating student performance models on data from introductory programming courses from two universities with a total of 3,323 students. Our results suggest that using cross-validation on one semester is insufficient for gauging model performance in the real world. Instead, we suggest that where possible future work in student performance prediction collects data from multiple semesters and uses one or more as a distinct hold-out set. Failing this, bootstrapped cross-validation should be used to improve confidence in models' performance. By recommending stronger methods for evaluating performance prediction models, we hope to bring them closer to practical use and assist teachers to understand struggling students in novice programming courses.

Qualitative research methodologies that collect data through interviews, focus groups and ethnographic studies, are valuable approaches to understand the educational experience. Participant drawings can be used as prompts to access implicit understandings and have been shown to contribute to effective data collection in related fields, but are not widely used in computing education research. We report on a case study in which sixteen New Zealand primary and secondary school digital technologies teachers were interviewed and asked to draw their ideal computing student. This case study illustrates how drawings can be used to enhance semi-structured interviews in the context of computing education. The contributions made here come in the form of a case study demonstrating the draw-a-person method in a computing education context, along with recommendations for researchers considering applying this method in their work.

Debugging is a core skill required by programmers, yet we know little about how to effectively teach the process of debugging. The challenges of learning debugging are compounded for novices who lack experience and are still learning the tools they need to program effectively. In this work, we report a case study in which we used a think-aloud protocol to gain insight into the behaviour of three students engaged in debugging tasks. Our qualitative analysis reveals a variety of helpful practices and barriers that limit the effectiveness of debugging. We observe that comprehension, evidence-based activities, and workflow practices all contribute to novice debugging success. Lack of sustained effort, precision, and methodical processes negatively impact debugging effectiveness. We anticipate that understanding how students engage in debugging tasks will aid future work to address ineffective behaviours and promote effective debugging activities.

Computer programming is a fundamental tool for life scientists, allowing them to carry out essential research tasks. However, despite various educational efforts, learning to write code can be a challenging endeavor for students and researchers in life-sciences disciplines. Recent advances in artificial intelligence have made it possible to translate human-language prompts to functional code, raising questions about whether these technologies can aid (or replace) life scientists' efforts to write code. Using 184 programming exercises from an introductory-bioinformatics course, we evaluated the extent to which one such tool-OpenAI's ChatGPT-could successfully complete programming tasks. ChatGPT solved 139 (75.5%) of the exercises on its first attempt. For the remaining exercises, we provided natural-language feedback to the model, prompting it to try different approaches. Within 7 or fewer attempts, ChatGPT solved 179 (97.3%) of the exercises. These findings have implications for life-sciences education and research. Instructors may need to adapt their pedagogical approaches and assessment techniques to account for these new capabilities that are available to the general public. For some programming tasks, researchers may be able to work in collaboration with machine-learning models to produce functional code.

Previous research has shown a correlation between spatial reasoning skills and success in many STEM subjects including computing. More recent research suggests not just a correlation but also a causation, i.e., students' achievements in computing can be improved by improving spatial reasoning skills. So far most of the literature in the education field has treated spatial skills as a single entity, e.g., authors often only measured or trained one specific skill, or did not differentiate between different skills during assessment. We investigate how students' success in introductory computer graphics is correlated to different types of spatial skills. The results of a user study with 36 participants suggest that success in introductory computer graphics is correlated to students' abilities in mental rotation and spatial visualisation but is not correlated with spatial perception skills. We also found a large variation of spatial abilities in the student cohort. We discuss possible reasons for our results and implications for teaching and learning.

In order to transfer problem solving knowledge effectively across tasks, students must be able to identify when two problems can be solved in similar ways. As instructors, it is helpful to know how well students can identify such similarities. Unfortunately, current efforts to understand how students categorise similar problems are laborious and time-consuming, involving intensive data collection and analysis. Our own prior work on analysing student categorisations of simple algorithmic patterns involved 35 hour-long interviews and time-consuming open coding of the resulting transcripts. It is clear that traditional approaches to elicit such knowledge are not scalable or reproducible.

When a student reads a programming problem statement, something has to happen; that something could be abject confusion, the beginnings of a search for a solution, or a well-formed understanding of what the problem is asking and how to solve it. Barring abject confusion, several theories explain the differences between these responses all revolving around the existence or non-existence of a problem schema – some mental concept or knowledge structure which encodes what it is to be a particular type of problem which gets solved in a particular type of way. Learners often lack appropriate schemata to call upon when solving problems, instead resorting to generic problem-solving techniques. Not only is this an inefficient method of solving problems, it can even inhibit the development of schemata. In line with constructivist theories of learning, effective teaching should build on the existing knowledge of learners; to do so, we must understand the nature of what they know – what do their schemata, as undeveloped as they may be, ‘look like’ and what concepts do they have about problems? In this paper, we explore the categories students identify when sorting simple algorithmic computing problem statements and the language they use to describe those categories. We conduct an interpretivist study involving a card sorting exercise, in which 35 computing students across four years of tertiary-level study grouped problem statements into categories they identified as meaningful, followed up with semi-structured interviews. Results of qualitative analysis revealed several students do demonstrate productive knowledge for identifying and reasoning about common tasks such as filtering, mapping, aggregating, and searching; however, this knowledge is fragile and concrete, and does not demonstrate the existence of pre-established problem schemata or abstract knowledge of algorithmic patterns. One implication of this work is that instruction may benefit from a more explicit focus on patterns and plans, and an established language with which students can communicate and reason about them.

The practice of peer code review has been shown to deliver a variety of benefits to programming students. These include learning from producing and receiving feedback, and from being exposed to a range of problem-solving approaches and solutions. However, the success of a peer code review activity depends on the quality and accuracy of the reviews that students produce, and prior work has shown that these can sometimes be poor. One approach for addressing this problem is to incorporate motivational incentives directly into the design of the code review platform. In this research, we explore the use of gamification in an online peer code review tool, where game-like elements are used to reward students for generating accurate and helpful reviews. We report the results of a randomized controlled study (n=171) that measures both review accuracy and the perceived value of the feedback produced. Although quantitative ratings of the review quality did not differ significantly between control and experimental conditions, we observed interesting trends relating to the perceived value of the feedback. Students in both groups had similar views regarding the usefulness of the feedback they received on their own work, however students in the experimental condition tended to express more positive sentiments towards the quality of the feedback they produced for their peers and observed from other reviewers.

The introduction of programming into secondary schools in New Zealand (NZ) has meant that many teachers are preparing programming students at senior level for the New Zealand Certificate of Educational Achievement (NCEA). A recent study showed that many teachers struggle with providing feedback to students on code quality, and indicated a possible issue with the available resources. In this paper, we describe a study to help us understand this phenomenon. We first analysed the data from the earlier study concerning teachers' viewpoints on the available resources. We then analysed the defining curriculum and assessment documents for NCEA, extracting quality-relating terms. We found that all teachers interviewed reported issues with the resources and that there was no clear mapping for quality concepts between the two sets of resources. Our contribution is to expose gaps in the online resources that are problematic from the perspective of teacher and student outcomes.

There have been four principal studies on pass rates in introductory programming courses, in 2007, 2014, and two in 2019. These studies, the first two in particular, are extremely widely cited, but it appears that some of the citing papers misrepresent what was written in the original papers. We present an analysis of nearly 600 papers that cite one or more of these four papers, indicating what sorts of message they impute to the papers. We find that well over half of the citing papers fail to accurately represent the papers' findings, instead citing the papers as evidence either of a commonly accepted belief or of their own -- sometimes quite bizarre -- assertions.

A comprehensive understanding of computing has become a necessary skill for students in an increasingly digital world. Consequently, many countries around the world have updated their national pre-tertiary curriculum to include the topic of computing. However, the structure and presentation of these curricula varies greatly. This makes it difficult to compare the timing that students are exposed to computing and as a result, increases the difficulty for producing effective and well-timed educational resources that are relevant for varying implementations. This research investigates the structure of international computing curricula, identifying and comparing the timing that students are exposed to computing. This will better inform researchers and resource developers about where and when their interventions are most applicable. This will also provide valuable insights for the creation of future computing curricula, by informing curriculum designers about the perspectives of various curricula and the most common approaches they have taken.

The computing education community is growing, but there is little information about the geographic distribution of the community or collaboration between members. Our research investigates three computer science education conferences (SIGCSE Technical Symposium, ITiCSE and ICER) by analysing authorship and affiliation details for publications in the proceedings over the lifetime of the respective conferences, totalling over 4500 publications. We examine the geographic location of authors and model the scientific collaboration network of each conference. We conclude that the community is open to newcomers, and both the number of authors, and the overall level of collaboration is growing.

The ACM Special Interest Group on Computer Science Education (SIGCSE) is one of the oldest and largest SIGs, and the SIGCSE Technical Symposium is the oldest and largest of the four SIGCSE conferences. However, the vast majority of Symposium attendees and contributors are from the United States. Because SIGCSE is an international organization, this lack of geographic diversity and representation is troubling because it may stifle collaboration, membership, professional development, and dissemination of research, and have many other adverse effects.

Gamification has been used in introductory programming courses, for example, to increase engagement with study materials, reduce procrastination, and increase attendance to practice sessions. Indeed, with the rapidly growing adoption of digital tools in such courses, the use of various game elements and mechanics to drive participation is increasing. Previous studies on gamification in computing have examined the effects over the whole student population. Prior work in other disciplines has found that the benefits associated with gamification may only be realized for some students, while others may even experience reduced motivation. The Hexad user types survey attempts to tackle this problem by grouping users into six different types for whom gamification should have different effects. The goal is to personalize the game elements for different user types, thus creating gamified experiences more suitable for individual learners. In this work, we study whether the Hexad survey could be used to guide the personalization of gamification in an introductory programming course. Specifically, we examine the quality of students' answers to the Hexad survey and explore whether they can be used to predict students' preferences for enabling gamification in the platform where they complete assignments. In our specific computing education context, we find that classifying students using the Hexad survey does not appear to be an effective approach for the automatic personalization of gamification.

Learning computer graphics requires a diverse range of skills, such as mathematics, programming, problem-solving and 3D reasoning skills. One fundamental area students struggle with is three-dimensional (3D) transformations, partially due to a lack of visuospatial skills.

Despite decades of literature calling for research on teaching debugging, we still lack clear guidelines on how to teach debugging processes effectively. This is a significant problem as debugging is an activity that is a key component of software development and a frustrating and time-consuming barrier to students who are learning to program. In this paper we reflect on teaching and learning debugging through the lens of student beliefs about their own debugging practice. We find that few students use systematic approaches such as the scientific method of hypothesis generation and testing that is informally adopted by many software developers. Some students appeared to be satisfied with their ad hoc approaches while others expressed a desire to develop more thorough approaches.

Authors of academic papers are generally required to nominate several keywords that characterize the paper, but are rarely offered guidance on how to select those keywords. We analyzed the keywords in the past 15 years of selected computing education publications: the 1274 papers published in the proceedings of ICER and ITiCSE, including the ITiCSE working group reports. As well as the keywords assigned by the authors, we mined the abstracts of these papers to extract a separate list of keywords. Our work has two goals: to frame the thematic landscape of the field, using keywords that communicate the work conducted; and to detect differences between the human judgement and interpretation of keywords and the machine 'intelligence' on handling those keywords, with respect to the clusters of thematic topics identified in each case. The analysis shows that the field is dominated by learning approaches (e.g., active learning, collaborative learning), aspects of programming (e.g., debugging, misconceptions), computational thinking, feedback, and assessment, while other areas that have attracted attention include academic integrity (e.g., plagiarism) and diversity (e.g., female students, underrepresented groups). It was observed that the keywords chosen by authors are often too general to provide information about the paper (e.g., 'concerns', 'course', 'fun', 'justice'). We elaborate on the findings and begin a discussion on how authors can improve the communication of their research and make access to it more transparent.

Cybersecurity is an area of growing international importance. In response to global shortages of Cybersecurity skills, many universities have introduced degree programmes in Cybersecurity. These programmes aim to prepare students to become Cybersecurity practitioners with advanced skills in a timely manner. Several universities offer Cybersecurity degrees, but these have been developed ad hoc, as there is currently no internationally accepted Cybersecurity curriculum. Recently, an ITiCSE working group on global perspectives on Cybersecurity education developed a competency-based framework that aims to help institutions to implement Cybersecurity programmes. In this report, we present a case study of a Cybersecurity programme at the University of Auckland. We discuss how the curriculum and resource management of this programme evolved, and we present some challenges for the design and delivery of a Cybersecurity programme in the light of this competency-based framework.

SIGCSE is a global organization with members from well over one hundred countries, but attendance at SIGCSE conferences is not always reflective of membership as a whole. Attendance at the SIGCSE Technical Symposium is overwhelmingly from the United States, with more than 92% of all attendees in recent years having a U.S. affiliation. This panel, which includes members of the SIGCSE Board and the Symposium International Committee, will present the state of Symposium participation from outside the U.S. in an effort towards understanding what can be done to expand global participation in the Symposium. The panelists span the range from students to long-time faculty and together have computing education experience from seven countries across six continents. The panel will also solicit views from attendees on barriers to global Symposium participation and ways to improve non-U.S. participation in SIGCSE's flagship conference.

Peer review is the standard process within academia for maintaining publication quality, but it is also widely employed in other settings, such as education and industry, for improving work quality and for generating actionable feedback to content authors. For example, in the software industry peer review of program source code—or peer code review—is a key technique for detecting bugs and maintaining coding standards. In a programming education context, although peer code review offers potential benefits to both code reviewers and code authors, individuals are typically less experienced, which presents a number of challenges. Some of these challenges are similar to those reported in the educational literature on peer review in other academic disciplines, but reviewing code presents unique difficulties. Better understanding these challenges and the conditions under which code review can be taught and implemented successfully in computer science courses is of value to the computing education community. In this work, we conduct a systematic review of the literature on peer code review in higher education to examine instructor motivations for conducting peer code review activities, how such activities have been implemented in practice, and the primary benefits and difficulties that have been reported. We initially identified 187 potential studies and analyzed 51 empirical studies pertinent to our goals. We report the most commonly cited benefits (e.g., the development of programming-related skills) and barriers (e.g., low student engagement), and we identify a wide variety of tools that have been used to facilitate the peer code review process. While we argue that more empirical work is needed to validate currently reported results related to learning outcomes, there is also a clear need to address the challenges around student motivation, which we believe could be an important avenue for future research.

Vast numbers of publications in computing education begin with the premise that programming is hard to learn and hard to teach. Many papers note that failure rates in computing courses, and particularly in introductory programming courses, are higher than their institutions would like. Two highly distinct research projects have established that average success rates in introductory programming courses world-wide are in the region of 67%. However, there is little published work comparing pass rates in computing courses with those in other STEM disciplines. As institutions continually ask computing educators to justify the atypical failure rates in their courses, a thoroughly researched comparison of this sort could prove useful in demonstrating whether the phenomenon is real, and, if so, whether it extends somewhat beyond the boundaries of individual institutions. This working group will gather information on pass rates in computing courses, particularly introductory programming courses, and in courses at comparable levels in other STEM disciplines. Members of the group will be required to gather the information from their own institutions, and further data will be gathered by way of a broad survey. The data will be analysed to see whether global patterns can be established, and the group will survey the literature to gather and summarise postulated explanations for any difference between pass rates in computing and in other STEM disciplines.

One means to determine whether a student understands the fundamentals of good object-oriented design is to assess designs the student has created. However, providing reliable assessment of designs efficiently is difficult due to the many viable designs that are possible and the high level of expertise required. Consequently, design assessment tends to be limited to identifying the most basic of design problems. We propose a technique---"object counts''---that involves counting the objects created at runtime. This is more efficient than manual grading because the data is gathered automatically and more reliable than using rubrics because it is based on objective data. The data is relevant because it captures the fundamental property of an object-oriented program---the creation of objects---and so provides good insight into the student's design decisions. This provides support for both summative and formative feedback. We demonstrate the technique on two corpora containing submissions for a typical first assignment of an introductory course on object-oriented design.

Online coding tools are an increasingly common feature of programming courses, providing students with rapid feedback and flexible practice opportunities and providing instructors with useful analytics. However, little research has explored the complexity of online exercises provided to students and the order in which students are exposed to new ideas. In this paper, we investigate the benefits of using a short sequence of practice exercises, each targeting a distinct topic, prior to having students solve a goal task that combines the concepts. As expected, we find students solve the goal task with fewer errors and in less time after completing the practice tasks. However, we also find that the practice tasks reduce the likelihood of students delaying work on the goal task, and these effects are particularly large for less-experienced students.

Debugging software is challenging, particularly for novices. Despite the importance of debugging, most novice programmers are not formally taught any debugging skills. This paper describes an online tool, Ladebug, that is designed to scaffold the learning of debugging skills. In this environment, students follow a structured debugging process to find and fix errors in predefined exercises. Overall, we find that students are positive about the tool, and report the exercises to be engaging and helpful.

A broad review of research on the teaching and learning of programming was conducted by Robins et al. in 2003. Since this work there have been several reviews of research concerned with the teaching and learning of programming, in particular introductory programming. However, these reviews have focused on highly specific aspects, such as student misconceptions, teaching approaches, program comprehension, potentially seminal papers, research methods applied, automated feedback for exercises, competency-enhancing games, and program visualisation. While these aspects encompass a wide range of issues, they do not cover the full scope of research into novice programming. Some notable areas that have not been reviewed are assessment, academic integrity, and novice student attitudes to programming. There does not appear to have been a comprehensive review of research into introductory programming since that of Robins et al. It is therefore timely to conduct and present such a review in order to gain an understanding of the research focuses, to highlight advances in knowledge since 2003, and to indicate possible future directions for research. The working group will conduct a systematic literature review based on the guidelines proposed by Kitchenham et al. This research project is well suited to an ITiCSE working group as the synthesis and discussion of the literature will benefit from input from a variety of researchers drawn from different backgrounds and countries.

Teaching and learning computer graphics is often considered challenging due to it requiring a diverse range of skills such as mathematics, programming, problem solving, and art and design. Assignments are a popular tool to support learning and to assess students' understanding. The value of such assignments depends on the ability to give fast (and ideally formative) feedback, and enabling students to interactively explore the solution space. This is often a problem, in particular for large classes, where assignment marking can take many days or even weeks. By the time feedback is received students often don't remember details, and there is usually no opportunity to resubmit and hence little motivation to reflect on and correct mistakes.

Teaching cyber security techniques can be challenging due to the complexity associated with building secure systems. The major issue is these systems could easily be broken if proper protection techniques are not employed. This requires students to understand the offensive approaches that can be used to breach security in order to better understand how to properly defend against cyber attacks. We present a novel approach to teaching cyber security in a graduate course using an innovative assessment task that engages students in both software obfuscation and reverse engineering of obfuscated code. Students involved in the activities gain an appreciation of the challenges in defending against attacks. Our results demonstrate a positive change in the students' perception during the learning process.

Design smells are design structures that indicate poor design quality. Many identified smells are difficult to teach as they require a degree of experience and judgement that novices, by definition, do not have. We have identified a design smell, which we call "unencapsulated collection", that is common in novice designs. It is simple to describe, allowing it to be objectively detected, and the refactoring steps needed to remove the smell are usually simple to illustrate. We give a description of the smell and present the results of an empirical study showing its prevalence. We outline the general steps for refactoring the smell, and illustrate it with a case study. The simplicity of this smell makes it a good candidate for teaching good design principles to novices.

Current CS1 learning outcomes are relatively general, specifying tasks such as designing, implementing, testing and debugging programs that use some fundamental programming constructs. These outcomes impact what we teach, our expectations, and our assessments. Although prior work has demonstrated the utility of single concept assessments, most assessments used in formal examinations combine numerous heterogeneous concepts, resulting in complex and difficult tasks.

The orthodox view that "programming is difficult to learn" leads to uncritical teaching practices and poor student outcomes. It may also impact negatively on diversity and equity within the Computer Science discipline. But learning to program is easy --- so easy that children can do it. We make our introductory courses difficult by establishing unrealistic expectations for novice programming students. By revisiting the expected norms for introductory programming we may be able to substantially improve outcomes for novice programmers, address negative impressions of disciplinary practices and create a more equitable environment.

Pen-based computing allows users to seamlessly add freeform ink annotations to dynamic digital documents. When these documents change, the annotations should adapt, or “refit”, in response to these changes. While previous studies have demonstrated ways of refitting, there have been few investigations in what the users expect to happen. In this paper, we report on an investigation into user expectations for refitting freeform ink annotations. We explored how the four main categories of annotations can be refitted and what user thought of each approach. Our findings indicate the approach depends on the user's overall preference and the category of annotation. Furthermore we suggest ways to implement freeform ink annotations on dynamic digital documents.

Debugging is an important skill for novice programmers to acquire. Error messages help novices to locate and correct errors, but compiler messages are frequently inadequate. We have developed a system that provides enhanced error messages, including concrete examples that illustrate the kind of error that has occurred and how that kind of error could be corrected. We evaluate the effectiveness of the enhanced error messages with a controlled empirical study and find no significant effect.

Background: The quality of data sets used in software engineering research is of the utmost importance. To ensure credibility of results obtained from use of data sets, the quality of the data must be examined. Objective: This study provides an overview of recent research(2008-2012) involving data quality in software engineering datasets, with the goal of generally understanding what research there is that addresses data quality, and in particular to determine to what degree researchers have addressed any data quality issues in order to evaluate the trustworthiness of their results. Method: We performed a systematic mapping study to investigate treatment of data quality issues in software engineering research. A total of 64 papers published from 2008 to 2012explicitly address issues with the quality of data and use software engineering data sets. These studies were classified according to the data quality topic, data set and data quality problem. Results: We found only 31 studies gave serious consideration for how the quality of the data affected their results. We observed that there is a lack of clear and consistent terminology regarding data quality, especially with respect to the kinds of quality problems a data set might have. As a first step to address this problem, we propose a model that describes the lifecycle that research data goes through when used in research. Conclusions: The results suggest that researchers should give more attention to the quality of data sets in order to produce trustworthy data for reliable empirical research, and that the research community needs to better understand and communicate issues with data quality.

We describe the design of a web-based interactive learning environment that guides students' learning of visual aesthetics. With aesthetics, the whole is greater than the sum of its parts. However, initially we take an element-by-element approach that aligns well to the SOLO Taxonomy and software components. Using the colour element as an exemplar, we trialled our online learning environment with 36 undergraduate students undertaking a Human-Computer Interaction course. The results suggest that students enjoy and engage in the process, indicating the potential of our structured approach to learning visual aesthetics within Human Computer Interaction education.

We know that students solve problems in different ways, but we know little about the kinds of variation, or the degree of variation between these student generated solutions. In this paper, we propose a taxonomy that classifies the variation between correct student solutions in objective terms, and we show how the application of the taxonomy provides instructors with additional insight about the differences between student solutions. This taxonomy may be used to inform instructors in selecting examples of code for teaching purposes, and provides the possibility of automatically applying the taxonomy to existing solution sets.

Identifying and correcting syntax errors is a challenge all novice programmers confront. As educators, the more we understand about the nature of these errors and how students respond to them, the more effective our teaching can be. It is well known that just a few types of errors are far more frequently encountered by students learning to program than most. In this paper, we examine how long students spend resolving the most common syntax errors, and discover that certain types of errors are not solved any more quickly by the higher ability students. Moreover, we note that these errors consume a large amount of student time, suggesting that targeted teaching interventions may yield a significant payoff in terms of increasing student productivity.

Recent research has focused on tools that support the creation, review and sharing of student-generated content for peer learning. However, we know little about the student perspective of such activities. In this paper, we identify what students believe is most helpful for their learning by analysing open-ended comments from students engaged in creating, answering and reviewing exam-style questions generated by their peers. Students report learning about content and appropriate standards of work, both individually and through interaction with peer generated resources.

Recent interest in student-centric pedagogies have resulted in the development of numerous tools that support student generated questions. Previous evaluations of such tools have reported strong correlations between student participation and exam performance, yet the level of student engagement with other learning activities in the course is a potential confounding factor. We show such correlations may be explained by other factors, and we undertake a deeper analysis that reveals evidence of the positive impact question-generation activities have on student performance.

Mastering syntax is one of the earliest challenges facing the novice programmer. Problem solving and algorithms are the focus of many first year programming classes, leaving students to learn syntax on their own while they practice writing code. In this paper we investigate the frequency with which students encounter syntax errors during a drill and practice activity. We find that students struggle with syntax to a greater extent than we anticipated, even when writing short fragments of code.

Although a number of existing systems support student-generated multiple choice questions, such questions tend to focus on lower-order cognitive skills. Free response questions are frequently used to evaluate higher-order thinking, but supporting student-generated free-response questions is challenging. StudySieve is a web-based tool that extends student-generated questions to the free-response domain.

Drill and practice exercises enable students to master skills needed for more sophisticated programming. A barrier to providing such activities is the effort required to set up the programming environment. Testing is an important component to writing good software, but it is difficult to motivate students to write tests. In this paper we describe and evaluate Code Write, a web-based tool that provides drill and practice support for Java programming, and for which testing plays a central role in its use. We describe how we have used Code Write in a CS1 course, and demonstrate its effectiveness in providing good coverage of the language features presented in the course.

Students in a large introductory programming course were asked twice to predict their scores on the final exam: once at the beginning of a six-week module, and once at the end. In between, students in only one of the two lecture streams recorded subjective confidence in their answers to individual questions on weekly quizzes. Students' predictions were moderately correlated with their scores. Students who attended more quizzes had not only higher exam scores, but improved their predictions more than those who attended fewer quizzes. Practice recording confidence on individual quiz questions did not yield significantly more improvement in exam predictions. Several findings from previous work are confirmed, including that women were significantly more underconfident than men.

While popularity of peer assessment in Computer Science has increased in recent years, the validity of peer assessed marks remain a significant concern to instructors and source of anxiety to students. We report here on a large-scale study (1,500 students and 10,000 reviews) involving three introductory programming classes which recorded grades and feedback comments for both student and tutor reviews of novice programs. Using a paired analysis, we compare the quantitative marks given by students with those given by tutors, for both functional and non-functional aspects of the program. We also report on an analysis of the lexical sophistication of feedback comments.

A recent approach to engaging students in deep learning involves an online tool, PeerWise, through which students contribute multiple-choice questions to a shared question bank. Earlier work demonstrated a strong correlation between the use of PeerWise and student performance. In this study we investigate the quality of the MCQ repository created by students in an introductory programming course by analysing the range of topics on which students chose to write questions (i.e.~the repository coverage) without guidance from an instructor. We assess the repository coverage by comparing it with a common list of typical introductory programming topics, and by looking at its extent. We find that, despite having freedom to choose any topic, students created a repository that covered all the major topics in the curriculum.

Computer games are widely used in programming assignments to motivate students. Traditionally, these have been text-based games such as hangman, but as Java has become widespread, the use of graphics and graphical games has increased correspondingly. We report on a framework used as scaffolding to help students understand how to design and implement a variety of interactive games. We share our experiences using this framework for assignments over a number of years.

PeerWise is a system in which students create multiple choice questions and answer those created by their peers. In this paper, we report on some quantitative results which suggest that students who use PeerWise actively perform better in final examinations than students who are not active. We note a significant correlation between performance in written (not just multiple choice) questions and PeerWise activity, suggesting that active use of the system may contribute to deep (and not just drill-and-practise) learning.

Common exam practice centres around two question types: code tracing (reading) and code writing. It is commonly believed that code tracing is easier than code writing, but it seems obvious that different skills are needed for each. These problems also differ in their value on an exam. Pedagogically, code tracing on paper is an authentic task whereas code writing on paper is less so. Yet, few instructors are willing to forgo the code writing question on an exam. Is there another way, easier to grade, that captures the "problem solving through code creation process" we wish to examine? In this paper we propose Parson's puzzle-style problems for this purpose. We explore their potential both qualitatively, through interviews, and quantitatively through a set of CS1 exams. We find notable correlation between Parsons scores and code writing scores. We find low correlation between code writing and tracing and between Parsons and tracing. We also make the case that marks from a Parsons problem make clear what students don't know (specifically, in both syntax and logic) much less ambiguously than marks from a code writing problem. We make recommendations on the design of Parsons problems for the exam setting, discuss their potential uses and urge further investigations of Parsons problems for assessment of CS1 students.

PeerWise is a web-based system that supports the creation of student-generated test banks of multiple choice questions. Students contribute question stems and answers, provide explanations, answer questions contributed by other students, rate questions for difficulty and quality, and participate in on-line discussions of all these activities.

This paper presents the results of a replicated pair programming experiment conducted at the University of Auckland (NZ) during the first semester of 2005. It involved 190 second year Computer Science students attending a software design and construction course. We replicated the experiment described in [18], investigating similar issues to those reported in [32] and employing a subset of the questionnaires used in [32]. Our results confirm the use of pair programming as an effective programming/design learning technique.

This paper presents the results of a pair programming experiment conducted at the University of Auckland (NZ) during the first semester of 2004. It involved 300 second year Computer Science students attending a software design and construction course. We investigated similar issues to those reported in [26] and employed a subset of the questionnaires used by Laurie Williams et al. on the experiments presented in [26]. Our results support the use of pair programming as an effective programming/design learning technique.