Code-generating Artificial Intelligence has gained popularity within both professional and educational programming settings over the past several years. While research and pedagogy are beginning to cope with this change, computing students are left to bear the unforeseen consequences of AI amidst a dearth of empirical evidence about its effects. Though pair programming between students is well studied and known to be beneficial to self-efficacy and academic achievement, it remains underutilized and further threatened by the proposition that AI can replace a human programming partner. In this paper, we present a controlled pair programming study with 22 participants who wrote Python code under time pressure in teams of two and individually with GitHub Copilot for 20 minutes each. They were incentivized by bonus compensation to balance performance with understanding and were retested individually on the programming tasks after a retention interval of one week. Subjective measures of workload and emotion as well as objective measures of performance and learning (retest performance) were collected. When first programming with AI, participants clearly performed better and had an easier time. On retest, they were not dramatically worse in raw score from the human-human condition, but human-AI pairs lost more of their initial advantage. Additionally, the emotional effect of the human teammate was significantly more positive and arousing as compared to working with Copilot. We recommend that educators strongly consider revisiting pair programming as an educational tool in addition to embracing modern AI.

Learning to communicate with code-generating AI models is an emerging skill for novice programmers. One recent pedagogical approach, Prompt Problems, has students solve computational tasks by writing natural-language prompts for code-generating AI models. However, little is known about the specific prompt-level mistakes novice programmers make, the kinds of computational details they fail to communicate, and what strategies they use to recover when generated code is incorrect. In a CS1 course, we studied attempts by more than 900 students to solve dialogue-based Prompt Problems. We analyzed student reflections, unsuccessful prompts, and reported debugging strategies. Compared to traditional coding tasks, students generally found prompting easier, more enjoyable, and better targeted at developing problem-solving skills. The most common mistakes are related to the omission of key details, suggesting both a failure to acknowledge their importance and over-reliance on AI to infer them. When prompts failed, students focused more on clarifying their intent and reflecting on the provided problem details than on tracing generated code or examining test cases.

The rise of generative artificial intelligence (GenAI) has sparked a rapid change in computing curricula and teaching approaches. GenAI coding tools can accurately complete assignments, answer test questions, and perform other tasks traditionally associated with learning programming, especially at the introductory level. Because GenAI is still so new, researchers investigating student usage of GenAI have used informal rubrics and questionnaires. To advance, the field needs validated instruments that measure student perception and use of GenAI. This paper presents the development and initial validation of an instrument to measure self-efficacy while using GenAI to learn programming. Self-efficacy is an important construct in education research because it robustly correlates with student success, across disciplines and ages, including undergraduate computing education. Computing education researchers have presented several validated self-efficacy instruments, most recently by Steinhorst et al. in 2020. Critically, this instrument was created before the rise of GenAI’s popularity in 2022. To complement this instrument, we created a GenAI scale similar in style to the Steinhorst self-efficacy instrument, consisting originally of 11 items and revised to 5 items. We report two important findings in this paper. First, we found strong support for the validity of the existing Steinhorst instrument in a new context, specifically an introductory programming course that fully integrates GenAI. Second, the new GenAI scale shows strong internal reliability, discriminant validity with items in the Steinhorst subscales, and criterion validity with students’ GenAI usage patterns. Based on statistical analysis and cognitive probing interviews, we argue for the validity of the five-item scale to measure students’ GenAI self-efficacy in the context of programming.

Modern programming tools use generative AI (GenAI) to suggest code to the user as they type, interrupting their problem-solving behavior and undermining the development of their programming critical thinking skills. In this paper, we present a scaffolded programming exercise designed to support student differentiation between good and bad GenAI code suggestions based on negative expertise–that identifying why an answer is wrong is part of developing conceptual knowledge. We compare a version of the tool that showed one suggestion (correct or not), to a version that showed three suggestions (one of which was correct). We present results on performance and error rates as well as qualitative findings centered on Pintrich and DeGroot’s theory of self-regulation. Students reported that the single suggestion version better aligned with industry tools and presented a lower cognitive load. Students also reported that the multiple suggestion version caused them to slow down and think critically about the line under consideration, the overall purpose of the code, and the benefits of planning.

Program decomposition is a core skill in computer science that overlaps with program comprehension, refactoring, and design. Decomposition takes many forms, with one common task at the introductory level being the identification and extraction of meaningful abstraction into separate functions. However, there is limited empirical evidence about how the algorithmic structuring of code affects the difficulty of producing a meaningful decomposition. In this paper, we empirically study how different algorithmic implementations of the same underlying task affect students' ability to reason about good abstractions through method extraction. Grounded in a recent framework on code structuring, we design three functionally equivalent versions of the same task that differ only in how two latent functional patterns relate to each other: sequentially, hierarchically, or interleaved. Using a large-scale controlled study with 994 introductory programming students, we examine students' effort in decomposing the programs, the approaches they adopt, and their perceptions of whether the resulting decomposition improves readability and understanding. Our results show that interleaved functional composition is more difficult to decompose, providing initial empirical support for the framework's hypothesized ordering of composition pattern difficulty. Students also perceive their decompositions of the interleaved version to be less easy to read and understand. We also find that students describe many different approaches of decomposing, and we discuss several implications for teaching and future research.

Publisher Copyright: © 2026 Owner/Author.

Large language models (LLMs) are increasingly integrated into computing education, yet nearly all prior research has focused on text-based interactions.As voice-enabled interfaces become more capable and more common, there is growing interest in understanding how voice input might shape students' use of LLM-powered tools.In this exploratory study, we investigated how introductory programming students interact with Prompt Problems, which are programming tasks that require crafting natural-language prompts to generate correct code.Students (N = 919) solved a series of Prompt Problems with the freedom to select or switch between text and voice input modalities.We collected their prompt submissions as well as post-activity survey responses, then analysed differences in prompt accuracy, persistence, and perspectives by modality.For two of the three problems, we found that students who typed their prompts using text were more likely to have those prompts succeed on the first attempt than students who submitted unedited voice prompts.There was no difference in success rate if students edited their transcribed voice prompts before submission.Across the problems, we found evidence that students who tried voice prompting varied in their usage of modality -perhaps indicating a complementary, or non-preferential approach.However, most students only tried and reported preferring text.Our qualitative analysis revealed how students' perceived the roles of voice and text input in shaping their problem-solving process, as well as the reported drawbacks and advantages of each modality.We discuss implications for future multimodal tools and instructional design in computing education.

Student disengagement remains a persistent challenge in higher education. This paper presents Genius-Coin, an open-source, self-hostable blockchain courseware designed to promote student motivation and engagement through gamification and token-economy mechanisms. The system connects verifiable learning activities to digital rewards: classroom attendance is recorded through dynamic QR code scanning, while online participation from learning management and discussion platforms is also captured. Once participation is verified, digital tokens and achievement badges are automatically distributed through smart contracts and can later be redeemed for rewards through an online store. Designed for practical institutional use, Genius-Coin supports straightforward deployment on institutional servers and provides a dedicated wallet application that simplifies student onboarding. A pilot deployment with 65 postgraduate students suggests that the system fostered intrinsic engagement, as many students appeared more motivated to attend class and participate in course activities than to redeem tokens for external rewards; only 33.1% of distributed tokens were redeemed.

Code quality is an essential aspect of programming education, impacting source code maintainability and readability. However, providing manual feedback is time-consuming for instructors, and automated approaches to code quality feedback remain inconsistent and underdeveloped. We adapt a defect catalogue for novice Python programmers and construct an automated detection pipeline that integrates multiple static analysis tools and a custom natural language identifier detector. Using a standardised set of defect examples, we evaluate the detection coverage of the selected tools. Our results show that our integrated toolset detects 64 defect types, with 30 undetected. Applying the pipeline to over 86,000 student submissions, we analyse the prevalence and distribution of code quality defects in real coursework. To better align with pedagogical priorities, we introduce a multi-dimensional prioritisation framework that combines defect frequency, student coverage, and instructor-rated importance. Our findings provide a reference for improving feedback mechanisms and instructional strategies for code quality in introductory programming education.

Decomposition and abstraction is an essential component of computational thinking, yet it is not always emphasized in introductory programming courses. In addition, as generative AI further reduces the focus on syntax and increases the importance of higher-level code reasoning, there is renewed opportunity to teach DA explicitly. In this paper, we introduce a framework for systematically assessing the complexity of code structuring tasks, where students must identify and separate meaningful abstractions within existing, unstructured code. The framework defines three dimensions of task complexity, each with multiple levels: repetition, code pattern, and data dependency. To support practical use, we provide example tasks mapped to these levels and offer an interactive tool for generating and exploring DA problems. The framework is designed to support the development of educational tasks that build students' skills with DA in the procedural paradigm.

Large Language Models (LLMs) have shown the potential to generate code explanations that surpass those of peers in quality, offering promising opportunities for computer science education. Inspired by this, we explore whether combining multiple diverse explanations, each emphasizing distinct aspects (e.g., function, concept, goal), can enhance students' understanding of programming exercises compared to generic explanations that do not emphasize distinct conceptual aspects. Insights from other fields, such as computational creativity, suggest that diverse ideas may be more beneficial than relying solely on a single, high-quality option. Variation Theory holds that learners grasp a concept when they see systematic variation that exposes its critical features, helping them distinguish it from related ideas. In creative domains, uniform or homogeneous exemplars can lead to design fixation, whereas varied inputs support more flexible reasoning. In a study with 971 first-year computing students, participants were randomly assigned either diverse or generic LLM-generated explanations for two programming exercises. Students completed multiple-choice (MCQ) and open-ended (OE) questions for each exercise to assess understanding, followed by Likert-scale questions and OE reflections to understand preferences and perception. Across participants, performance was consistently 7.7% higher when students received diverse explanations, and there was no difference in perceived cognitive load. Performance on the closed-form multiple-choice questions was similar for diverse and generic explanations.

Large language models (LLMs) have demonstrated strong capabilities in language understanding and generation, and their potential in educational contexts is increasingly being explored. One promising area is learnersourcing, where students engage in creating their own educational content, such as multiple-choice questions. A critical step in this process is generating effective explanations for the solutions to these questions, as such explanations aid in peer understanding and promote deeper conceptual learning. However, students often find it difficult to craft high-quality explanations due to limited understanding or gaps in their subject knowledge. To support this task, we introduce ``ILearner-LLM,'' a framework that uses iterative enhancement with LLMs to improve generated explanations. The framework combines an explanation generation model and an explanation evaluation model fine-tuned using student preferences for quality, where feedback from the evaluation model is fed back into the generation model to refine the output. Our experiments with LLaMA2-13B and GPT-4 using five large datasets from the PeerWise MCQ platform show that ILearner-LLM produces explanations of higher quality that closely align with those written by students. Our findings represent a promising approach for enriching the learnersourcing experience for students and for leveraging the capabilities of large language models for educational applications.

Introductory programming courses often rely on small code-writing exercises that have clearly specified problem statements. This limits opportunities for students to practice how to clarify ambiguous requirements -- a critical skill in real-world programming. In addition, the emerging capabilities of large language models (LLMs) to produce code from well-defined specifications may harm student engagement with traditional programming exercises. This study explores the use of ``Probeable Problems'', automatically gradable tasks that have deliberately vague or incomplete specifications. Such problems require students to submit test inputs, or `probes', to clarify requirements before implementation. Through analysis of over 40,000 probes in an introductory course, we identify patterns linking probing behaviors to task success. Systematic strategies, such as thoroughly exploring expected behavior before coding, resulted in fewer incorrect code submissions and correlated with course success. Feedback from nearly 1,000 participants highlighted the challenges and real-world relevance of these tasks, as well as benefits to critical thinking and metacognitive skills. Probeable Problems are easy to set up and deploy at scale, and help students recognize and resolve uncertainties in programming problems.

Publisher Copyright: © 2025 Copyright held by the owner/author(s).

In today's digital world, computing education offers critical opportunities, yet systemic inequities exclude under-represented communities, especially in rural, under-resourced regions. Early engagement is vital for building interest in computing careers and achieving equitable participation. Recent work has shown that the use of sensor-enabled tools and block-based programming can improve engagement and self-efficacy for students from under-represented groups, but these findings lack replication in diverse, resource-constrained settings. This study addresses this gap by implementing sensor-based programming workshops with rural students in Sri Lanka. Replicating methods from the literature, we conduct a between-group study (sensor vs. non-sensor) using Scratch and real-time environmental sensors. We found that students in both groups reported significantly higher confidence in programming in Scratch after the workshop. In addition, average changes in both self-efficacy and outcome expectancy were higher in the experimental (sensor) group than in the control (non-sensor) group, mirroring trends observed in the original study being replicated. We also found that using the sensors helped to enhance creativity and inspired some students to express an interest in information and communications technology (ICT) careers, supporting the value of such hands-on activities in building programming confidence among under-represented groups.

Computing students increasingly rely on generative AI tools for programming assistance, often without formal instruction or guidance. This highlights a need to teach students how to effectively interact with AI models, particularly through natural language prompts, to generate and critically evaluate code for solving computational tasks. To address this, we developed a novel platform for prompt programming that enables authentic dialogue-based interactions, supports problems involving multiple interdependent functions, and offers on-request execution of generated code. Data analysis from over 900 students in an introductory programming course revealed high engagement, with the majority of prompts occurring within multi-turn dialogues. Problems with multiple interdependent functions encouraged iterative refinement, with progression graphs highlighting several common strategies. Students were highly selective about the code they chose to test, suggesting that on-request execution of generated code promoted critical thinking. Given the growing importance of learning dialogue-based programming with AI, we provide this tool as a publicly accessible resource, accompanied by a corpus of programming problems for educational use.

Reading and understanding code are fundamental skills for novice programmers, and especially important with the growing prevalence of AI-generated code and the need to evaluate its accuracy and reliability. ''Explain in Plain English'' questions are a widely used approach for assessing code comprehension, but providing automated feedback, particularly on comprehension levels, is a challenging task. This paper introduces a novel method for automatically assessing the comprehension level of responses to ''Explain in Plain English'' questions. Central to this is the ability to distinguish between two response types: multi-structural, where students describe the code line-by-line, and relational, where they explain the code's overall purpose. Using a Large Language Model (LLM) to segment both the student's description and the code, we aim to determine whether the student describes each line individually (many segments) or the code as a whole (fewer segments). We evaluate this approach's effectiveness by comparing segmentation results with human classifications, achieving substantial agreement. We conclude with how this approach, which we release as an open source Python package, could be used as a formative feedback mechanism.

''Explain in Plain English'' (EiPE) questions are widely used to assess code comprehension skills but are challenging to grade automatically. Recent approaches like Code Generation Based Grading (CGBG) leverage large language models (LLMs) to generate code from student explanations and validate its equivalence to the original code using unit tests. However, this approach does not differentiate between high-level, purpose-focused responses and low-level, implementation-focused ones, limiting its effectiveness in assessing comprehension level. We propose a modified approach where students generate function names, emphasizing the function's purpose over implementation details. We evaluate this method in an introductory programming course and analyze it using Item Response Theory (IRT) to assess the difficulty and discrimination of function naming exercises as exam items and to compare their alignment with traditional EiPE grading standards. We also publish this work as an open source Python package for auto-grading EiPE questions, providing a scalable solution for adoption.

Peer Reviewed

Novice programmers often struggle to develop computational thinking (CT) skills in introductory programming courses. This study investigates the use of Large Language Models (LLMs) to provide scalable, strategy-driven feedback to teach CT. Through think-aloud interviews with 17 students solving code comprehension and writing tasks, we found that LLMs effectively guided decomposition and program development tool usage. Challenges included students seeking direct answers or pasting feedback without considering suggested strategies. We discuss how instructors should integrate LLMs into assessments to support students' learning of CT.

Grading computer graphics programming assessments and generating formative and summative feedback can require significant effort on the part of human experts.Since these assessments generate visual outputs that can be static or animated, determining correctness may be subjective.For feedback to be effective, it must be delivered in a timely manner.This can be a challenge for introductory computer graphics-based courses since cohort size can be substantial, errors in visual output can be subtle, and causes of errors are often not obvious.In this paper, we explore the feasibility of an automated system for marking visual output and providing program implementation feedback for learners in an introductory computer graphics-based design course in three short programming assessments, including static and animated scenes.To assess the effectiveness of our approach, we compare the marks generated by our tool with those assigned by a human expert.We show that it is possible to automate marking, providing both a grade based on the visual output and formative feedback on source code in the style of a human marker.This can improve objective consistency, grade reproducibility, and reduce marking time, enabling a course to scale to support large cohorts without the need for more resourcing for human markers.We describe lessons learnt and potential pitfalls to assist educators with introducing automated marking for their courses.Finally, we identify areas for future refinement and development of our automated system.

Code reading and comprehension skills are essential for novices learning programming, and explain-in-plain-English tasks (EiPE) are a well-established approach for assessing these skills. However, manual grading of EiPE tasks is time-consuming and this has limited their use in practice. To address this, we explore an approach where students explain code samples to a large language model (LLM) which generates code based on their explanations. This generated code is then evaluated using test suites, and shown to students along with the test results. We are interested in understanding how automated formative feedback from an LLM guides students' subsequent prompts towards solving EiPE tasks. We analyzed 177 unique attempts on four EiPE exercises from 21 students, looking at what kinds of mistakes they made and how they fixed them. We found that when students made mistakes, they identified and corrected them using either a combination of the LLM-generated code and test case results, or they switched from describing the purpose of the code to describing the sample code line-by-line until the LLM-generated code exactly matched the obfuscated sample code. Our findings suggest both optimism and caution with the use of LLMs for unmonitored formative feedback. We identified false positive and negative cases, helpful variable naming, and clues of direct code recitation by students. For most students, this approach represents an efficient way to demonstrate and assess their code comprehension skills. However, we also found evidence of misconceptions being reinforced, suggesting the need for further work to identify and guide students more effectively.

The emergence of large language models (LLMs) has transformed research and practice across a wide range of domains. Within the computing education research (CER) domain, LLMs have garnered significant attention, particularly in the context of learning programming. Much of the work on LLMs in CER, however, has focused on applying and evaluating proprietary models. In this article, we evaluate the efficiency of open-source LLMs in generating high-quality feedback for programming assignments and judging the quality of programming feedback, contrasting the results with proprietary models. Our evaluations on a dataset of students' submissions to introductory Python programming exercises suggest that state-of-the-art open-source LLMs are nearly on par with proprietary models in both generating and assessing programming feedback. Additionally, we demonstrate the efficiency of smaller LLMs in these tasks and highlight the wide range of LLMs accessible, even for free, to educators and practitioners.

Debugging is an essential skill when learning to program, yet its instruction and emphasis often vary widely across introductory courses. In the era of code-generating large language models (LLMs), the ability for students to reason about code and identify errors is increasingly important. However, students frequently resort to trial-and-error methods to resolve bugs without fully understanding the underlying issues. Developing the ability to identify and hypothesize the cause of bugs is crucial but can be time-consuming to teach effectively through traditional means. This paper introduces BugSpotter, an innovative tool that leverages an LLM to generate buggy code from a problem description and verify the synthesized bugs via a test suite. Students interact with BugSpotter by designing failing test cases, where the buggy code's output differs from the expected result as defined by the problem specification. This not only provides opportunities for students to enhance their debugging skills, but also to practice reading and understanding problem specifications. We deployed BugSpotter in a large classroom setting and compared the debugging exercises it generated to exercises hand-crafted by an instructor for the same problems. We found that the LLM-generated exercises produced by BugSpotter varied in difficulty and were well-matched to the problem specifications. Importantly, the LLM-generated exercises were comparable to those manually created by instructors with respect to student performance, suggesting that BugSpotter could be an effective and efficient aid for learning debugging.

Productive Failure (PF) is a learning approach where students initially tackle novel problems targeting concepts they have not yet learned, followed by a consolidation phase where these concepts are taught. Recent application in STEM disciplines suggests that PF can help learners develop more robust conceptual knowledge. However, empirical validation of PF for programming education remains under-explored. In this paper, we investigate the use of PF to teach Python lists to undergraduate students with limited prior programming experience. We designed a novel PF-based learning activity that incorporated the unobtrusive collection of real-time heart-rate data from consumer-grade wearable sensors. This sensor data was used both to make the learning activity engaging and to infer cognitive load. We evaluated our approach with 20 participants, half of whom were taught Python concepts using Direct Instruction (DI), and the other half with PF. We found that although there was no difference in initial learning outcomes between the groups, students who followed the PF approach showed better knowledge retention and performance on delayed but similar tasks. In addition, physiological measurements indicated that these students also exhibited a larger decrease in cognitive load during their tasks after instruction. Our findings suggest that PF-based approaches may lead to more robust learning, and that future work should investigate similar activities at scale across a range of concepts.

Providing students with hands-on experience in kernel programming within a real-world operating system is highly beneficial in an Operating Systems (OS) course for teaching core operating system concepts and developing practical skills. However, accessing suitable devices for such hands-on experimentation poses significant challenges. Traditional solutions involve hosting virtual machines on cloud platforms, which are expensive and do not scale well with increasing student numbers. Additionally, many students' personal devices, such as Macs or iPads, have limited support for running Linux, creating further barriers. In this paper, we introduce KernelVM, a novel cost-effective platform that offers students a Linux virtual machine with full superuser access and pre-configured kernel programming toolchains. KernelVM is accessible via any modern browser on any device. It performs all computations locally within the user's browser, thus eliminating cloud computing costs. KernelVM provides a robust learning environment by incorporating interactive virtual hardware components and an automatic evaluation system, supporting a wide range of tasks, including multi-threaded cryptographic kernel modules and Linux drivers for hardware interaction. We detail the design of KernelVM, and describe our experiences incorporating it for the first time into an OS course with 159 undergraduate students. We found that KernelVM was instrumental in improving the quality and efficiency of hands-on learning experiences, with students reporting increased satisfaction and engagement due to the immediate feedback and the ability to experiment in a risk-free environment. Our experience suggests that KernelVM not only addresses the logistical challenges of kernel programming education, but it helps foster a highly interactive and engaging learning experience.

Leveraging a Large Language Model (LLM) for personalized learning in computing education is promising, yet cloud-based LLMs pose risks around data security and privacy. To address these concerns, we developed and deployed a locally stored Small Language Model (SLM) utilizing Retrieval-Augmented Generation (RAG) methods to support computing students' learning. Previous work has demonstrated that SLMs can match or surpass popular LLMs (gpt-3.5-turbo and gpt-4-32k) in handling conversational data from a CS1 course. We deployed SLMs with RAG (SLM + RAG) in a large course with more than 250 active students, fielding nearly 2,000 student questions, while evaluating data privacy, scalability, and feasibility of local deployments. This paper provides a comprehensive guide for deploying SLM + RAG systems, detailing model selection, vector database choice, embedding methods, and pipeline frameworks. We share practical insights from our deployment, including scalability concerns, accuracy versus context length trade-offs, guardrails and hallucination reduction, as well as data privacy maintenance. We address the "Impossible Triangle" in RAG systems, which states that achieving high accuracy, short context length, and low time consumption simultaneously is not feasible. Furthermore, our novel RAG framework, Intelligence Concentration (IC), categorizes information into multiple layers of abstraction within Milvus collections mitigating trade-offs and enabling educational assistants to deliver more relevant and personalized responses to students quickly.

Learnersourcing offers great potential for scalable education through student content creation. However, predicting student performance on learnersourced questions, which is essential for personalizing the learning experience, is challenging due to the inherent noise in student-generated data. Moreover, while conventional graph-based methods can capture the complex network of student and question interactions, they often fall short under cold start conditions where limited student engagement with questions yields sparse data. To address both challenges, we introduce an innovative strategy that synergizes the potential of integrating Signed Graph Neural Networks (SGNNs) and Large Language Model (LLM) embeddings. Our methodology employs a signed bipartite graph to comprehensively model student answers, complemented by a contrastive learning framework that enhances noise resilience. Furthermore, LLM's contribution lies in generating foundational question embeddings, proving especially advantageous in addressing cold start scenarios characterized by limited graph data. Validation across five real-world datasets sourced from the PeerWise platform underscores our approach's effectiveness. Our method outperforms baselines, showcasing enhanced predictive accuracy and robustness.

Timely, personalized feedback is essential for students learning programming. LLM-powered tools like ChatGPT offer instant support, but reveal direct answers with code, which may hinder deep conceptual engagement. We developed CodeAid, an LLM-powered programming assistant delivering helpful, technically correct responses, without revealing code solutions. CodeAid answers conceptual questions, generates pseudo-code with line-by-line explanations, and annotates student’s incorrect code with fix suggestions. We deployed CodeAid in a programming class of 700 students for a 12-week semester. A thematic analysis of 8,000 usages of CodeAid was performed, further enriched by weekly surveys, and 22 student interviews. We then interviewed eight programming educators to gain further insights. Our findings reveal four design considerations for future educational AI assistants: D1) exploiting AI’s unique benefits; D2) simplifying query formulation while promoting cognitive engagement; D3) avoiding direct responses while encouraging motivated learning; and D4) maintaining transparency and control for students to asses and steer AI responses.

Providing timely and personalized feedback to large numbers of students is a long-standing challenge in programming courses. Relying on human teaching assistants (TAs) has been extensively studied, revealing a number of potential shortcomings. These include inequitable access for students with low confidence when needing support, as well as situations where TAs provide direct solutions without helping students to develop their own problem-solving skills. With the advent of powerful large language models (LLMs), digital teaching assistants configured for programming contexts have emerged as an appealing and scalable way to provide instant, equitable, round-the-clock support. Although digital TAs can provide a variety of help for programming tasks, from high-level problem solving advice to direct solution generation, the effectiveness of such tools depends on their ability to promote meaningful learning experiences. If students find the guardrails implemented in digital TAs too constraining, or if other expectations are not met, they may seek assistance in ways that do not help them learn. Thus, it is essential to identify the features that students believe make digital teaching assistants valuable. We deployed an LLM-powered digital assistant in an introductory programming course and collected student feedback ($n=813$) on the characteristics of the tool they perceived to be most important. Our results highlight that students value such tools for their ability to provide instant, engaging support, particularly during peak times such as before assessment deadlines. They also expressed a strong preference for features that enable them to retain autonomy in their learning journey, such as scaffolding that helps to guide them through problem-solving steps rather than simply being shown direct solutions.

Publisher Copyright: © 2024 Owner/Author.

Grasping complex computing concepts often poses a challenge for students who struggle to anchor these new ideas to familiar experiences and understandings. To help with this, a good analogy can bridge the gap between unfamiliar concepts and familiar ones, providing an engaging way to aid understanding. However, creating effective educational analogies is difficult even for experienced instructors. We investigate to what extent large language models (LLMs), specifically ChatGPT, can provide access to personally relevant analogies on demand. Focusing on recursion, a challenging threshold concept, we conducted an investigation analyzing the analogies generated by more than 350 first-year computing students. They were provided with a code snippet and tasked to generate their own recursion-based analogies using ChatGPT, optionally including personally relevant topics in their prompts. We observed a great deal of diversity in the analogies produced with student-prescribed topics, in contrast to the otherwise generic analogies, highlighting the value of student creativity when working with LLMs. Not only did students enjoy the activity and report an improved understanding of recursion, but they described more easily remembering analogies that were personally and culturally relevant.

Introducing students to new concepts in computer science can often be challenging, as these concepts may differ significantly from their existing knowledge and conceptual understanding. To address this, we employed analogies to help students connect new concepts to familiar ideas. Specifically, we generated analogies using large language models (LLMs), namely ChatGPT, and used them to help students make the necessary connections. In this poster, we present the results of our survey, in which students were provided with two analogies relating to different computing concepts, and were asked to describe the extent to which they were accurate, interesting, and useful. This data was used to determine how effective LLM-generated analogies can be for teaching computer science concepts, as well as how responsive students are to this approach.

Parsons problems provide useful scaffolding for introductory programming students learning to write code. However, generating large numbers of high-quality Parsons problems that appeal to the diverse range of interests in a typical introductory course is a significant challenge for educators. Large language models (LLMs) may offer a solution, by allowing students to produce on-demand Parsons problems for topics covering the breadth of the introductory programming curriculum, and targeting thematic contexts that align with their personal interests. In this paper, we introduce PuzzleMakerPy, an educational tool that uses an LLM to generate unlimited contextualized drag-and-drop programming exercises in the form of Parsons Problems, which introductory programmers can use as a supplemental learning resource. We evaluated PuzzleMakerPy by deploying it in a large introductory programming course, and found that the ability to personalize the contextual framing used in problem descriptions was highly engaging for students, and being able to customize the programming topics was reported as being useful for their learning.

Large language models (LLMs) have shown great potential for the automatic generation of feedback in a wide range of computing contexts. However, concerns have been voiced around the privacy and ethical implications of sending student work to proprietary models. This has sparked considerable interest in the use of open source LLMs in education, but the quality of the feedback that such open models can produce remains understudied. This is a concern as providing flawed or misleading generated feedback could be detrimental to student learning. Inspired by recent work that has utilised very powerful LLMs, such as GPT-4, to evaluate the outputs produced by less powerful models, we conduct an automated analysis of the quality of the feedback produced by several open source models using a dataset from an introductory programming course. First, we investigate the viability of employing GPT-4 as an automated evaluator by comparing its evaluations with those of a human expert. We observe that GPT-4 demonstrates a bias toward positively rating feedback while exhibiting moderate agreement with human raters, showcasing its potential as a feedback evaluator. Second, we explore the quality of feedback generated by several leading open-source LLMs by using GPT-4 to evaluate the feedback. We find that some models offer competitive performance with popular proprietary LLMs, such as ChatGPT, indicating opportunities for their responsible use in educational settings.

We explored how undergraduate introductory programming students naturalistically used generative AI to solve programming problems. We focused on the relationship between their use of AI to their self-regulation strategies, self-efficacy, and fear of failure in programming. In this repeated-measures, mixed-methods research, we examined students' patterns of using generative AI with qualitative student reflections and their self-regulation, self-efficacy, and fear of failure with quantitative instruments at multiple times throughout the semester. We also explored the relationships among these variables to learner characteristics, perceived usefulness of AI, and performance. Overall, our results suggest that student factors affect their baseline use of AI. In particular, students with higher self-efficacy, lower fear of failure, or higher prior grades tended to use AI less or later in the problem-solving process and rated it as less useful than others. Interestingly, we found no relationship between students' self-regulation strategies and their use of AI. Students who used AI less or later in problem-solving also had higher grades in the course, but this is most likely due to prior characteristics as our data do not suggest that this is a causal relationship.

Computer science (CS) capstone courses offer students a valuable opportunity to gain hands-on experience in software development, practice essential soft skills, and enhance their employability prospects. They are a core component in many CS undergraduate degrees and address the ACM curricula requirements of inculcating professional dispositions in students and making them aware of the broader societal implications of computing. However, coordinating a capstone course, especially for a large student cohort, can be a daunting task for academic staff. It demands considerable time and energy for planning and coordinating activities between students, academic staff, and any external stakeholders. In this experience report, we outline the iterative development and refinement of our capstone course as it grew substantially in size over a span of six consecutive sessions. We outline the pedagogies that helped us to enhance student engagement and motivation in the course as assessed by end-of-course surveys and students' written reflections. We share the lessons that we have learnt and provide recommendations to educators who are designing new capstone courses or looking to scale existing ones.

Folk-wisdom in software engineering suggests that small functions that adhere to the principle of single-responsibility have several advantages over longer, monolithic functions, including improvement in code comprehension. Despite this widespread view, empirical research on the impact of functional decomposition on understanding code is sparse, yet it is central to software development practices.

aendees to leave the workshop with a practical understanding of how to engage with learnersourcing.Participants will get hands-on experience with current tools,

Learning to program requires the development of a variety of skills including the ability to read, comprehend, and communicate the purpose of code. In the age of large language models (LLMs), where code can be generated automatically, developing these skills is more important than ever for novice programmers. The ability to write precise natural language descriptions of desired behavior is essential for eliciting code from an LLM, and the code that is generated must be understood in order to evaluate its correctness and suitability. In introductory computer science courses, a common question type used to develop and assess code comprehension skill is the 'Explain in Plain English' (EiPE) question. In these questions, students are shown a segment of code and asked to provide a natural language description of that code's purpose. The adoption of EiPE questions at scale has been hindered by: 1) the difficulty of automatically grading short answer responses and 2) the ability to provide effective and transparent feedback to students. To address these shortcomings, we explore and evaluate a grading approach where a student's EiPE response is used to generate code via an LLM, and that code is evaluated against test cases to determine if the description of the code was accurate. This provides a scalable approach to creating code comprehension questions and enables feedback both through the code generated from a student's description and the results of test cases run on that code. We evaluate students' success in completing these tasks, their use of the feedback provided by the system, and their perceptions of the activity.

Large language models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards reading, comprehending and evaluating LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.

In the past year, large language models (LLMs) have taken the world by storm, demonstrating their potential as a transformative force in many domains including computing education. Computing education researchers have found that LLMs can solve most assessments in introductory programming courses, including both traditional code writing tasks and other popular tasks such as Parsons problems. As more and more students start to make use of LLMs, the question instructors might ask themselves is "what can I do?". We propose that one promising way forward is to integrate LLMs into teaching practice, providing all students with an equal opportunity to learn how to interact productively with LLMs as well as encounter and understand their limitations. In this workshop, we first present state-of-the-art research results on how to utilize LLMs in computing education practice, after which participants will take part in hands-on activities using LLMs. We end the workshop by brainstorming ideas with participants around adapting their classrooms to most effectively integrate LLMs while avoiding some common pitfalls.

Introductory programming courses often require students to solve many small programming exercises as part of their learning. Researchers have previously suggested that the context used in the problem description for these exercises is likely to impact student engagement and motivation. Furthermore, supplying programming exercises that use a broad range of contexts or even allowing students to select contexts to personalize their own exercises, may support the interests of a diverse student population. Unfortunately, it is time-consuming for instructors to create large numbers of programming exercises that provide a wide range of contextualized problems. However, recent work has shown that large language models may be able to automate the mass production of programming exercises, reducing the burden on instructors. In this research, we explore the potential of OpenAI's GPT-4 to create high-quality and novel programming exercises that implement various contexts. Finally, through prompt engineering, we compare different prompting strategies used to generate many programming exercises with various contextualized problem descriptions and then evaluate the quality of the exercises generated.

Teaching large cohorts in upper-level computing courses is challenging, as providing rapid feedback and marking at scale is difficult without significant resources. Many institutions lack funds to employ a large number of skilled markers or such markers are simply not available.

In a previous Birds of a Feather discussion, we delved into the nascent applications of generative AI, contemplating its potential and speculating on future trajectories. Since then, the landscape has continued to evolve revealing the capabilities and limitations of these models. Despite this progress, the computing education research community still faces uncertainty around pivotal aspects such as (1) academic integrity and assessments, (2) curricular adaptations, (3) pedagogical strategies, and (4) the competencies students require to instill responsible use of these tools. The goal of this Birds of a Feather discussion is to unravel these pressing and persistent issues with computing educators and researchers, fostering a collaborative exploration of strategies to navigate the educational implications of advancing generative AI technologies. Aligned with this goal of building an inclusive learning community, our BoF is led by globally distributed leaders to facilitate multiple coordinated discussions that can lead to a broader conversation about the role of LLMs in CS education.

Large language models (LLMs) have recently taken many fields, including computer science, by storm. Most recent work on LLMs in computing education has shown that they are capable of solving most introductory programming (CS1) exercises, exam questions, Parsons problems, and several other types of exercises and questions. Some work has investigated the ability of LLMs to solve CS2 problems as well. However, it remains unclear how well LLMs fare against more advanced upper-division coursework, such as proofs in algorithms courses. After all, while known to be proficient in many programming tasks, LLMs have been shown to have more difficulties in forming mathematical proofs.

Much of the recent work investigating large language models and AI Code Generation tools in computing education has focused on assessing their capabilities for solving typical programming problems and for generating resources such as code explanations and exercises. If progress is to be made toward the inevitable lasting pedagogical change, there is a need for research that explores the instructor voice, seeking to understand how instructors with a range of experiences plan to adapt. In this paper, we report the results of an interview study involving 12 instructors from Australia, Finland and New Zealand, in which we investigate educators' current practices, concerns, and planned adaptations relating to these tools. Through this empirical study, our goal is to prompt dialogue between researchers and educators to inform new pedagogical strategies in response to the rapidly evolving landscape of AI code generation tools.

Technology integration in educational settings has led to the development of novel sensor-based tools that enable students to measure and interact with their environment. Although reports from using such tools can be positive, evaluations are often conducted under controlled conditions and short timeframes. There is a need for longitudinal data collected in realistic classroom settings. However, sustained and authentic classroom use requires technology platforms to be seen by teachers as both easy to use and of value. We describe our development of a sensor-based platform to support science teaching that followed a 14-month design process. We share insights from this design and development approach, and report findings from a six-month large-scale evaluation involving 35 schools and 1245 students. We share lessons learnt, including that technology integration is not an educational goal per se and that technology should be a transparent tool to enable students to achieve their learning goals.

Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.

Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course (n ≈ 1000) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.

Recent breakthroughs in deep learning have led to the emergence of generative AI models that exhibit extraordinary performance at producing human-like outputs. Using only simple input prompts, it is possible to generate novel text, images, video, music, and source code, as well as tackle tasks such as answering questions and translating and summarising text.

The recent advent of highly accurate and scalable large language models (LLMs) has taken the world by storm. From art to essays to computer code, LLMs are producing novel content that until recently was thought only humans could produce. Recent work in computing education has sought to understand the capabilities of LLMs for solving tasks such as writing code, explaining code, creating novel coding assignments, interpreting programming error messages, and more. However, these technologies continue to evolve at an astonishing rate leaving educators little time to adapt. This working group seeks to document the state-of-the-art for code generation LLMs, detail current opportunities and challenges related to their use, and present actionable approaches to integrating them into computing curricula.

The recent emergence of code generation tools powered by large language models has attracted wide attention. Models such as OpenAI Codex can take natural language problem descriptions as input and generate highly accurate source code solutions, with potentially significant implications for computing education. Given the many complexities that students face when learning to write code, they may quickly become reliant on such tools without properly understanding the underlying concepts. One popular approach for scaffolding the code writing process is to use Parsons problems, which present solution lines of code in a scrambled order. These remove the complexities of low-level syntax, and allow students to focus on algorithmic and design-level problem solving. It is unclear how well code generation models can be applied to solve Parsons problems, given the mechanics of these models and prior evidence that they underperform when problems include specific restrictions. In this paper, we explore the performance of the Codex model for solving Parsons problems over various prompt variations. Using a corpus of Parsons problems we sourced from the computing education literature, we find that Codex successfully reorders the problem blocks about half of the time, a much lower rate of success when compared to prior work on more free-form programming tasks. Regarding prompts, we find that small variations in prompting have a noticeable effect on model performance, although the effect is not as pronounced as between different problems.

Knowledge of programming and computing is becoming increasingly valuable in today's world, and thus it is crucial that students from all backgrounds have the opportunity to learn. As the teaching of computing at high-school becomes more common, there is a growing need for approaches and tools that are effective and engaging for all students. Especially for students from groups that are traditionally underrepresented at university level, positive experiences at high-school can be an important factor for their future academic choices. In this paper we report on a hands-on programming workshop that we ran over multiple sessions for Maori and Pasifika high-school students who are underrepresented in computer science at the tertiary level in New Zealand. In the workshop, participants developed Scratch programs starting from a simple template we provided. In order to control the action in their programs, half of the participants used standard mouse and keyboard inputs, and the other half had access to plug-and-play sensors that provided real-time environmental data. We explore how students' perceptions of self-efficacy and outcome expectancy -- both key constructs driving academic career choices -- changed during the workshop and how these were impacted by the availability of the sensor toolkit. We found that participants enjoyed the workshop and reported improved self-efficacy with or without use of the toolkit, but outcome expectancy improved only for students who used the sensor toolkit.

A key part of learning to program is learning to understand programming error messages. They can be hard to interpret and identifying the cause of errors can be time-consuming. One factor in this challenge is that the messages are typically intended for an audience that already knows how to program, or even for programming environments that then use the information to highlight areas in code. Researchers have been working on making these errors more novice friendly since the 1960s, however progress has been slow. The present work contributes to this stream of research by using large language models to enhance programming error messages with explanations of the errors and suggestions on how to fix them. Large language models can be used to create useful and novice-friendly enhancements to programming error messages that sometimes surpass the original programming error messages in interpretability and actionability. These results provide further evidence of the benefits of large language models for computing educators, highlighting their use in areas known to be challenging for students. We further discuss the benefits and downsides of large language models and highlight future streams of research for enhancing programming error messages.

The introductory programming sequence has been the focus of much research in computing education. The recent advent of several viable and freely-available AI-driven code generation tools present several immediate opportunities and challenges in this domain. In this position paper we argue that the community needs to act quickly in deciding what possible opportunities can and should be leveraged and how, while also working on overcoming otherwise mitigating the possible challenges. Assuming that the effectiveness and proliferation of these tools will continue to progress rapidly, without quick, deliberate, and concerted efforts, educators will lose advantage in helping shape what opportunities come to be, and what challenges will endure. With this paper we aim to seed this discussion within the computing education community.

GitHub Copilot is an artificial intelligence tool for automatically generating source code from natural language problem descriptions. Since June 2022, Copilot has officially been available for free to all students as a plug-in to development environments like Visual Studio Code. Prior work exploring OpenAI Codex, the underlying model that powers Copilot, has shown it performs well on typical CS1 problems thus raising concerns about its potential impact on how introductory programming courses are taught. However, little is known about the types of problems for which Copilot does not perform well, or about the natural language interactions that a student might have with Copilot when resolving errors. We explore these questions by evaluating the performance of Copilot on a publicly available dataset of 166 programming problems. We find that it successfully solves around half of these problems on its very first attempt, and that it solves 60% of the remaining problems using only natural language changes to the problem description. We argue that this type of prompt engineering, which we believe will become a standard interaction between human and Copilot when it initially fails, is a potentially useful learning activity that promotes computational thinking skills, and is likely to change the nature of code writing skill development.

The introduction of Large Language Models (LLMs) has generated a significant amount of excitement both in industry and among researchers. Recently, tools that leverage LLMs have made their way into the classroom where they help students generate code and help instructors generate learning materials. There are likely many more uses of these tools -- both beneficial to learning and possibly detrimental to learning. To help ensure that these tools are used to enhance learning, educators need to not only be familiar with these tools, but with their use and potential misuse. The goal of this BoF is to raise awareness about LLMs and to build a learning community around their use in computing education. Aligned with this goal of building an inclusive learning community, our BoF is led by globally distributed discussion leaders, including undergraduate researchers, to facilitate multiple coordinated discussions that can lead to a broader conversation about the role of LLMs in CS education.

Recent breakthroughs in Large Language Models (LLMs), such as GPT-3 and Codex, now enable software developers to generate code based on a natural language prompt. Within computer science education, researchers are exploring the potential for LLMs to generate code explanations and programming assignments using carefully crafted prompts. These advances may enable students to interact with code in new ways while helping instructors scale their learning materials. However, LLMs also introduce new implications for academic integrity, curriculum design, and software engineering careers. This workshop will demonstrate the capabilities of LLMs to help attendees evaluate whether and how LLMs might be integrated into their pedagogy and research. We will also engage attendees in brainstorming to consider how LLMs will impact our field.

Advances in natural language processing have resulted in large language models (LLMs) that can generate code and code explanations. In this paper, we report on our experiences generating multiple code explanation types using LLMs and integrating them into an interactive e-book on web software development. Three different types of explanations -- a line-by-line explanation, a list of important concepts, and a high-level summary of the code -- were created. Students could view explanations by clicking a button next to code snippets, which showed the explanation and asked about its utility. Our results show that all explanation types were viewed by students and that the majority of students perceived the code explanations as helpful to them. However, student engagement varied by code snippet complexity, explanation type, and code snippet length. Drawing on our experiences, we discuss future directions for integrating explanations generated by LLMs into CS classrooms.

Reading a programming error message is the first step in understanding what it is trying to tell the programmer about how to fix an error in their code. However, these are often difficult to read, especially for novices which is not surprising given that error messages in many of the most popular languages in which novices learn to code were not written with readability in mind. As a result, novices frequently struggle to understand them. This is a long-standing problem, with researchers highlighting concerns about programming error message readability over the last six decades. Very recent work has put forward evidence of the need for measuring readability in error messages and a framework for doing so. This framework consists of four factors of readability for programming error messages: message length, vocabulary, jargon, and sentence construction. We use this framework to implement an approach to automatically assess the readability of programming error messages. Using established readability factors as predictors in a machine learning model, we train several models using a dataset of C and Java error messages. We examine the performance of these models, and apply the best performing model to a previously published set of messages evaluated for readability by experts, non-experts and students. Our results validate the previously proposed readability factors, and our model classifies messages similarly to human raters. Finally, we discuss future work needed to improve the accuracy of the model.

Student performance prediction aims to build models to help educators identify struggling students so they can be better supported. However, prior work in the space frequently evaluates features and models on data collected from a single semester, of a single course, taught at a single university. Without evaluating these methods in a broader context there is an open question of whether or not performance prediction methods are capable of generalising to new data. We test three methods for evaluating student performance models on data from introductory programming courses from two universities with a total of 3,323 students. Our results suggest that using cross-validation on one semester is insufficient for gauging model performance in the real world. Instead, we suggest that where possible future work in student performance prediction collects data from multiple semesters and uses one or more as a distinct hold-out set. Failing this, bootstrapped cross-validation should be used to improve confidence in models' performance. By recommending stronger methods for evaluating performance prediction models, we hope to bring them closer to practical use and assist teachers to understand struggling students in novice programming courses.

Computer programming is a fundamental tool for life scientists, allowing them to carry out essential research tasks. However, despite various educational efforts, learning to write code can be a challenging endeavor for students and researchers in life-sciences disciplines. Recent advances in artificial intelligence have made it possible to translate human-language prompts to functional code, raising questions about whether these technologies can aid (or replace) life scientists' efforts to write code. Using 184 programming exercises from an introductory-bioinformatics course, we evaluated the extent to which one such tool-OpenAI's ChatGPT-could successfully complete programming tasks. ChatGPT solved 139 (75.5%) of the exercises on its first attempt. For the remaining exercises, we provided natural-language feedback to the model, prompting it to try different approaches. Within 7 or fewer attempts, ChatGPT solved 179 (97.3%) of the exercises. These findings have implications for life-sciences education and research. Instructors may need to adapt their pedagogical approaches and assessment techniques to account for these new capabilities that are available to the general public. For some programming tasks, researchers may be able to work in collaboration with machine-learning models to produce functional code.

Automated question quality rating (AQQR) aims to evaluate question quality through computational means, thereby addressing emerging challenges in online learnersourced question repositories. Existing methods for AQQR rely solely on explicitly-defined criteria such as readability and word count, while not fully utilising the power of state-of-the-art deep-learning techniques. We propose DeepQR, a novel neural-network model for AQQR that is trained using multiple-choice-question (MCQ) datasets collected from PeerWise, a widely-used learnersourcing platform. Along with designing DeepQR, we investigate models based on explicitly-defined features, or semantic features, or both. We also introduce a self-attention mechanism to capture semantic correlations between MCQ components, and a contrastive-learning approach to acquire question representations using quality ratings. Extensive experiments on datasets collected from eight university-level courses illustrate that DeepQR has superior performance over six comparative models.

Background and Context. Metacognitive skills are important for all students learning to program and interest in applying pedagogical approaches in early programming courses that focus on metacognitive aspects is growing. However, most studies of such approaches are not rigorously based in theory, and when they are, almost always utilize foundational education and psychology theories from as far back as the 1970s. More recent theory is less tested, and not all relevant metacognitive theories have been explored in the computing education research literature.

This article explores the natural language generation capabilities of large language models with application to the production of two types of learning resources common in programming courses. Using OpenAI Codex as the large language model, we create programming exercises (including sample solutions and test cases) and code explanations, assessing these qualitatively and quantitatively. Our results suggest that the majority of the automatically generated content is both novel and sensible, and in some cases ready to use as is. When creating exercises we find that it is remarkably easy to influence both the programming concepts and the contextual themes they contain, simply by supplying keywords as input to the model. Our analysis suggests that there is significant value in massive generative machine learning models as a tool for instructors, although there remains a need for some oversight to ensure the quality of the generated content before it is delivered to students. We further discuss the implications of OpenAI Codex and similar tools for introductory programming education and highlight future research streams that have the potential to improve the quality of the educational experience for both teachers and students alike.

Programming is a complex task that requires the development of many skills including knowledge of syntax, problem decomposition, algorithm development, and debugging. Code-writing activities are commonly used to help students develop these skills, but the difficulty of writing code from a blank page can overwhelm many novices. Parsons problems offer a simpler alternative to writing code by providing scrambled code blocks that must be placed in the correct order to solve a problem. The extensive literature on Parsons problems documents numerous benefits to using them as both formative and summative assessments. These include more efficient learning, the possibility to dynamically adapt to learner needs, and more reliable grading. Despite these positive findings, further research is needed in order to draw broader inferences. Most work has been conducted at single institutions under unique conditions that are not easily replicated, and some prior studies have been inconclusive or had limitations that affected data validity. To address this, we propose a multi-institutional and multi-national study of the effectiveness of Parsons problems for novice programmers. We will focus on introductory programming courses (CS0/1/2) that use Java, Python, and C/C++ as these are the most common teaching languages. The working group will collaborate to refine the scope, methodology and research questions, and contribute to data collection and analysis.

Novice programmers naturally focus on writing code that is functionally correct over writing code that is readable and efficient. This emphasis on correctness over style is encouraged by the ubiquitous use of automated grading tools. Consequently, teaching students to write succinct, readable code that uses language-appropriate idioms is a challenge for educators. Prior work in computing education has catalogued the common code smells and anti-patterns often present in code produced by novices. However, there remains a need for good teaching resources and, importantly, evidence that these resources are both usable by students and produce the intended outcomes. In this paper, we propose a lab-based resource to help novices identify and refactor code smells when writing conditional statements. Our resource consists of four rules for simplifying conditionals that are derived from prior literature, and three accompanying practice tasks to which students can apply the rules. We describe our first experience using this resource in a large introductory programming course and explore (1) how well students use the provided rules to simplify correct but inelegant code, and (2) whether any effects persist on later code writing tasks. We found that students understood the rules and were able to apply them effectively to examples containing common code smells. We also found evidence that students continued to write more succinct code two weeks after using the resource. Finally, we make our resource available so that it can be adopted by other educators.

The first annual workshop on Learnersourcing: Student-generated Content @ Scale is taking place at Learning @ Scale 2022. This hybrid workshop will expose attendees to the ample opportunities in the learnersourcing space, including instructors, researchers, learning engineers, and many other roles. We believe participants from a wide range of backgrounds and prior knowledge on learnersourcing can both benefit and contribute to this workshop, as learnersourcing draws on work from education, crowdsourcing, learning analytics, data mining, ML/NLP, and many more fields. Additionally, as the learnersourcing process involves many stakeholders (students, instructors, researchers, instructional designers, etc.), multiple viewpoints can help to inform what future student-generated content might be useful, new and better ways to assess the quality of the content and spark potential collaboration efforts between attendees. We ultimately want to show how everyone can make use of learnersourcing and have participants gain hands-on experience using existing tools, create their own learnersourcing activities using them or their own platforms, and take part in discussing the next challenges and opportunities in the learnersourcing space. Our hope is to attract attendees interested in scaling the generation of instructional and assessment content and those interested in the use of online learning platforms.

How students are assessed has a powerful effect on their strategies for studying and their learning. When designing assessments, instructors should consider how different approaches for providing feedback to students could encourage positive learning behaviours. One such design is the use of interim deadlines that enable students to receive and respond to feedback. This is used to encourage students to start early and thus reduce the negative effects of procrastination. If multiple submissions are allowed, penalty schemes can be included to encourage students to reflect deeply on the feedback they receive, rather than developing an over-reliance on autograders. In this work we describe two approaches to feedback used over two consecutive semesters for a final project in a large introductory programming course. In both semesters, the complexity and structure of the final project was similar and students received identical instruction. In the first instance of the course students could submit their work prior to two scheduled interim deadlines, after which they would receive automated feedback, before meeting a final third deadline. In the second instance, students received automated feedback immediately upon submission but with increasing penalties to discourage excessive submissions. In both cases, the ability to receive automated feedback -- both scheduled and immediate -- was designed to encourage early participation with the project. Under the two feedback schemes, we observed different patterns of behaviour -- particularly for the lower performing students. We explore the benefits and drawbacks of the two schemes and consider implications for future project grading.

Humans adjust how they speak depending on context. Two key facets of this are utilizing different vocabulary and speaking rates depending on the audience. Exactly how we use language while teaching may depend on our students, their backgrounds and needs, and the subject matter. How we speak in the classroom likely affects student comprehension and may affect equity and accessibility.

As computing students progress through their studies they become proficient with multiple programming languages. Prior work investigating language transitions for novices has tended to analyze program artifacts rather than explore the benefits and difficulties as perceived by students in their own words, and has often overlooked problems that may arise in switching paradigms or where familiar syntax has a different meaning in the new language. In this paper, we ask students to reflect on the transition from an interpreted language and environment (MATLAB) to a compiled language (C), prompting comments on the aspects of learning the new language that they found both easier and harder. Analysis of over 70,000 words written by 771 students revealed that the highest-performing students expressed more negative sentiments towards the language transition -- a surprising result that we hypothesize is explained by their generally stronger metacognitive skills. We also report the most common difficulties described by students, which include challenges with syntax, error messages, and the process of compilation, and suggest teaching practices that might help students as they transition to a new programming language.

In order to transfer problem solving knowledge effectively across tasks, students must be able to identify when two problems can be solved in similar ways. As instructors, it is helpful to know how well students can identify such similarities. Unfortunately, current efforts to understand how students categorise similar problems are laborious and time-consuming, involving intensive data collection and analysis. Our own prior work on analysing student categorisations of simple algorithmic patterns involved 35 hour-long interviews and time-consuming open coding of the resulting transcripts. It is clear that traditional approaches to elicit such knowledge are not scalable or reproducible.

Block-based programming environments are a popular way to introduce programming as they provide helpful visual cues and remove the complexities of syntax, allowing learners to focus on being creative. However, input to programs in block-based environments is often limited to the keyboard and mouse, meaning programs respond only to the direct actions of the user. Allowing programs to respond to changes in the physical environment may influence the types of programs students create and their motivation and interest towards learning programming. We explore this idea by integrating real-time sensor data into Scratch. With our platform, students simply connect a sensor to their computer via USB and use custom blocks, alongside conventional Scratch code, to read and react to the sensor data in real-time. We evaluated this approach using a field study involving 25 students with limited prior experience in Scratch. We found students were highly motivated by the sensors and expressed a strong desire to use them in future projects. Analysis of the created programs revealed that, compared to using standard input sources, students explored a wider variety of Scratch blocks when controlling sprites using the sensors.

Metacognition and self-regulation are important skills for successful learning and have been discussed and researched extensively in the general education literature for several decades. More recently, there has been growing interest in understanding how metacognitive and self-regulatory skills contribute to student success in the context of computing education. This article presents a thorough systematic review of metacognition and self-regulation work in the context of computer programming and an in-depth discussion of the theories that have been leveraged in some way. We also discuss several prominent metacognitive and self-regulation theories from the literature outside of computing education—for example, from psychology and education—that have yet to be applied in the context of programming education. In our investigation, we built a comprehensive corpus of papers on metacognition and self-regulation in programming education, and then employed backward snowballing to provide a deeper examination of foundational theories from outside computing education, some of which have been explored in programming education, and others that have yet to be but hold much promise. In addition, we make new observations about the way these theories are used by the computing education community, and present recommendations on how metacognition and self-regulation can help inform programming education in the future. In particular, we discuss exemplars of studies that have used existing theories to support their design and discussion of results as well as studies that have proposed their own metacognitive theories in the context of programming education. Readers will also find the article a useful resource for helping students in programming courses develop effective strategies for metacognition and self-regulation.

Scientific inquiry involves prediction, observation and explanation (POE) of phenomena and data. Appropriate guidance through these steps is essential for helping students learn and form positive attitudes towards science. Sensor-based education toolkits are becoming a popular way to provide this guidance, but they typically present different interfaces for measurement and learning materials which places a high cognitive demand on learners. To address this challenge, we developed a web application to integrate the scientific inquiry method where students are guided step-by-step, using a scaffolded-learning approach, through slide-based worksheets that provide direct interaction with real-time sensor measurements. We evaluate this approach through a qualitative analysis of data collected from two field studies in classrooms with a total of 42 students. We show that our approach encouraged positivity and further learning in science. Students displayed and expressed interest to conduct science experiments outside of class. We identify design implications for seamless learning, storytelling and integration of POE guided scientific inquiry with sensor-based toolkits.

Programming error messages play an important role in learning to program. The cycle of program input and error message response completes a loop between the programmer and the compiler/interpreter and is a fundamental interaction between human and computer. However, error messages are notoriously problematic, especially for novices. Despite numerous guidelines citing the importance of message readability, there is little empirical research dedicated to understanding and assessing it. We report three related experiments investigating factors that influence programming error message readability. In the first two experiments we identify possible factors, and in the third we ask novice programmers to rate messages using scales derived from these factors. We find evidence that several key factors significantly affect message readability: message length, jargon use, sentence structure, and vocabulary. This provides novel empirical support for previously untested long-standing guidelines on message design, and informs future efforts to create readability metrics for programming error messages.

When a student reads a programming problem statement, something has to happen; that something could be abject confusion, the beginnings of a search for a solution, or a well-formed understanding of what the problem is asking and how to solve it. Barring abject confusion, several theories explain the differences between these responses all revolving around the existence or non-existence of a problem schema – some mental concept or knowledge structure which encodes what it is to be a particular type of problem which gets solved in a particular type of way. Learners often lack appropriate schemata to call upon when solving problems, instead resorting to generic problem-solving techniques. Not only is this an inefficient method of solving problems, it can even inhibit the development of schemata. In line with constructivist theories of learning, effective teaching should build on the existing knowledge of learners; to do so, we must understand the nature of what they know – what do their schemata, as undeveloped as they may be, ‘look like’ and what concepts do they have about problems? In this paper, we explore the categories students identify when sorting simple algorithmic computing problem statements and the language they use to describe those categories. We conduct an interpretivist study involving a card sorting exercise, in which 35 computing students across four years of tertiary-level study grouped problem statements into categories they identified as meaningful, followed up with semi-structured interviews. Results of qualitative analysis revealed several students do demonstrate productive knowledge for identifying and reasoning about common tasks such as filtering, mapping, aggregating, and searching; however, this knowledge is fragile and concrete, and does not demonstrate the existence of pre-established problem schemata or abstract knowledge of algorithmic patterns. One implication of this work is that instruction may benefit from a more explicit focus on patterns and plans, and an established language with which students can communicate and reason about them.

For many decades, educational communities, including computing education, have debated the value of telling students what they need to know (i.e., direct instruction) compared to guiding them to construct knowledge themselves (i.e., constructivism). Comparisons of these two instructional approaches have inconsistent results. Direct instruction can be more efficient for short-term performance but worse for retention and transfer. Constructivism can produce better retention and transfer, but this outcome is unreliable. To contribute to this debate, we propose a new theory to better explain these research results. Our theory, multiple conceptions theory, states that learners develop better conceptual knowledge when they are guided to compare multiple conceptions of a concept during instruction. To examine the validity of this theory, we used this lens to evaluate the literature for eight instructional techniques that guide learners to compare multiple conceptions, four from direct instruction (i.e., test-enhanced learning, erroneous examples, analogical reasoning, and refutation texts) and four from constructivism (i.e., productive failure, ambitious pedagogy, problem-based learning, and inquiry learning). We specifically searched for variations in the techniques that made them more or less successful, the mechanisms responsible, and how those mechanisms promote conceptual knowledge, which is critical for retention and transfer. To make the paper directly applicable to education, we propose instructional design principles based on the mechanisms that we identified. Moreover, we illustrate the theory by examining instructional techniques commonly used in computing education that compare multiple conceptions. Finally, we propose ways in which this theory can advance our instruction in computing and how computing education researchers can advance this general education theory.

The practice of peer code review has been shown to deliver a variety of benefits to programming students. These include learning from producing and receiving feedback, and from being exposed to a range of problem-solving approaches and solutions. However, the success of a peer code review activity depends on the quality and accuracy of the reviews that students produce, and prior work has shown that these can sometimes be poor. One approach for addressing this problem is to incorporate motivational incentives directly into the design of the code review platform. In this research, we explore the use of gamification in an online peer code review tool, where game-like elements are used to reward students for generating accurate and helpful reviews. We report the results of a randomized controlled study (n=171) that measures both review accuracy and the perceived value of the feedback produced. Although quantitative ratings of the review quality did not differ significantly between control and experimental conditions, we observed interesting trends relating to the perceived value of the feedback. Students in both groups had similar views regarding the usefulness of the feedback they received on their own work, however students in the experimental condition tended to express more positive sentiments towards the quality of the feedback they produced for their peers and observed from other reviewers.

The computing education community is growing, but there is little information about the geographic distribution of the community or collaboration between members. Our research investigates three computer science education conferences (SIGCSE Technical Symposium, ITiCSE and ICER) by analysing authorship and affiliation details for publications in the proceedings over the lifetime of the respective conferences, totalling over 4500 publications. We examine the geographic location of authors and model the scientific collaboration network of each conference. We conclude that the community is open to newcomers, and both the number of authors, and the overall level of collaboration is growing.

Gamification has been used in introductory programming courses, for example, to increase engagement with study materials, reduce procrastination, and increase attendance to practice sessions. Indeed, with the rapidly growing adoption of digital tools in such courses, the use of various game elements and mechanics to drive participation is increasing. Previous studies on gamification in computing have examined the effects over the whole student population. Prior work in other disciplines has found that the benefits associated with gamification may only be realized for some students, while others may even experience reduced motivation. The Hexad user types survey attempts to tackle this problem by grouping users into six different types for whom gamification should have different effects. The goal is to personalize the game elements for different user types, thus creating gamified experiences more suitable for individual learners. In this work, we study whether the Hexad survey could be used to guide the personalization of gamification in an introductory programming course. Specifically, we examine the quality of students' answers to the Hexad survey and explore whether they can be used to predict students' preferences for enabling gamification in the platform where they complete assignments. In our specific computing education context, we find that classifying students using the Hexad survey does not appear to be an effective approach for the automatic personalization of gamification.

Data structures and algorithms (DSA) are fundamental concepts introduced in the early stages of any undergraduate computing degree. Many students find DSA concepts challenging to learn, partly due to their abstract nature but also due to issues of low motivation. We examine these and other common difficulties faced by educators teaching DSA, and present a novel solution which combines a game-based learning approach with the use of engaging visualizations of algorithms in 2.5D. Game-based learning and algorithm visualizations are often employed in existing educational tools, but are rarely combined. Our game, DeCode, presents DSA concepts as in-game objects and animations within a 2.5D game-based world alongside traditional game-like mechanics such as progression. Feedback from an initial pilot study (n=51) suggests that students regard DeCode as an enjoyable and effective tool for learning DSA concepts. We also find evidence that DeCode's visualizations help students find efficient solutions to problems that have received prior attention in the literature, such as shifting all of the elements in an array.

Online learning environments eliminate geographical barriers and enable new forms of collaboration between students at large scale. Self-presentation within such environments affects how students interact with learning content and with each other. We explore how anonymity/identifiability in user profile design impacts student interactions in a large multicultural classroom across two geographical locations. After triangulating 150,000 online interactions with questionnaires and focus groups, we provide three major findings. First, being identifiable had a significant impact on how students accessed and rated content created by their peers. Second, when identifiable, cultural differences became more prominent, leading some students to avoid content created by classmates of certain nationalities. Finally, when students interacted with their real identities, there were significant and negative gender effects which were absent when students were anonymous. These findings contribute to our understanding of social dynamics within multicultural learning environments, and raise practical implications for tool design.

Metacognition and self-regulation are popular areas of interest in programming education, and they have been extensively researched outside of computing. While computing education researchers should draw upon this prior work, programming education is unique enough that we should explore the extent to which prior work applies to our context. The goal of this systematic review is to support research on metacognition and self-regulation in programming education by synthesizing relevant theories, measurements, and prior work on these topics. By reviewing papers that mention metacognition or self-regulation in the context of programming, we aim to provide a benchmark of our current progress towards understanding these topics and recommendations for future research. In our results, we discuss eight common theories that are widely used outside of computing education research, half of which are commonly used in computing education research. We also highlight 11 theories on related constructs (e.g., self-efficacy) that have been used successfully to understand programming education. Towards measuring metacognition and self-regulation in learners, we discuss seven instruments and protocols that have been used and highlight their strengths and weaknesses. To benchmark the current state of research, we examined papers that primarily studied metacognition and self-regulation in programming education and synthesize the reported interventions used and results from that research. While the primary intended contribution of this paper is to support research, readers will also learn about developing and supporting metacognition and self-regulation of students in programming courses.

It is well known that programming error messages can be notoriously difficult for novices to understand, hampering progress and leading to frustration. In response, researchers have explored various approaches for enhancing such messages, yet results from this active strand of research are currently mixed. Direct comparisons of results between studies is challenging as these typically investigate different kinds of message enhancements and report results using different metrics. In addition, many prior studies have involved code writing tasks. In such cases, not all students encounter the same errors and messages, and it is difficult to isolate the time spent interpreting messages and resolving errors from the time spent writing code. In this research, we explore the effects of presenting novices with compiler error messages designed using the most recent collection of published guidelines - specifically, more easily readable, short, positive messages containing resolution hints. To accurately determine the time and effort required to read and respond to the messages, we utilise a debugging task where all students are presented the same code and therefore encounter the same errors. We present results of a randomised controlled experiment (n > 700) which shows that, compared to standard error messages, the messages we tested resulted in significantly shorter debugging times and higher self-reported scores of message usefulness for students in the very early stages of learning a new language.

The quality of the code impacts the cost of its maintenance, yet "code quality" is often not given attention in introductory programming courses, perhaps due to the difficulty of providing automated code quality feedback. We have been exploring how to provide automated feedback on complexity, one aspect of code quality. We have developed CompareCFG that provides feedback based on control flow graphs (CFGs). It generates visualisations of students' submissions and provides the means for a student to compare the CFG of their own code with CFGs of less complex submissions, helping to support their understanding of code complexity. CompareCFG also provides actionable feedback by indicating specific issues in a submission that can reduce its complexity. We evaluated CompareCFG in a pilot study. We found it provides useful feedback to participants that helped them reduce the complexity of their code. CompareCFG offers a convenient way to provide programming students with automated visual feedback on code quality.

Peer review is the standard process within academia for maintaining publication quality, but it is also widely employed in other settings, such as education and industry, for improving work quality and for generating actionable feedback to content authors. For example, in the software industry peer review of program source code—or peer code review—is a key technique for detecting bugs and maintaining coding standards. In a programming education context, although peer code review offers potential benefits to both code reviewers and code authors, individuals are typically less experienced, which presents a number of challenges. Some of these challenges are similar to those reported in the educational literature on peer review in other academic disciplines, but reviewing code presents unique difficulties. Better understanding these challenges and the conditions under which code review can be taught and implemented successfully in computer science courses is of value to the computing education community. In this work, we conduct a systematic review of the literature on peer code review in higher education to examine instructor motivations for conducting peer code review activities, how such activities have been implemented in practice, and the primary benefits and difficulties that have been reported. We initially identified 187 potential studies and analyzed 51 empirical studies pertinent to our goals. We report the most commonly cited benefits (e.g., the development of programming-related skills) and barriers (e.g., low student engagement), and we identify a wide variety of tools that have been used to facilitate the peer code review process. While we argue that more empirical work is needed to validate currently reported results related to learning outcomes, there is also a clear need to address the challenges around student motivation, which we believe could be an important avenue for future research.

The goal of many computing education researchers is to improve how computing is taught and learned. To do that, researchers must engage with teachers, coaches, and mentors who design instructional materials and deliver lessons. However researchers may not be investigating problems that are directly of interest or utility to practitioners, and thus may not deliver results that are as impactful as possible in their contexts. To find out what research most interests today's practitioners, we conducted a two-stage survey. The first stage gathered questions that practitioners want researchers to investigate, and the second stage ranked these questions in terms of importance. We found that today's practitioners are more interested in student behavior, student understanding, and pedagogy than in languages and tools, curriculum, and inclusivity, and that there is little overlap between the questions ranked as highly interesting by researchers and those ranked highly by practitioners. Our results indicate that researchers need to better communicate why the questions they are pursuing are important, look for opportunities to collaborate with those who teach but have little direct connection with research, and examine the relevance of their research questions to practitioners.

Diagnostic messages generated by compilers and interpreters such as syntax error messages have been researched for decades. Unfortunately these messages which include error, warning, and runtime messages, present substantial difficulty and could be more effective, particularly for novices. Recent years have seen increased number of papers in the area including studies on the effectiveness of these messages, improving or enhancing them, and their usefulness as a part of programming process data that can be used to predict student performance. Despite this increased interest, the long history of literature is quite scattered and has not been brought together in any digestible form. We argue that in order to help the community proceed with more work on diagnostic messages, the literature needs to be presented in a state-of-the-art report. In addition we will synthesize and present the existing evidence for these messages including the difficulties they present and their effectiveness. We will also formulate a set of guidelines based on this evidence that can be used when designing or enhancing diagnostic messages. This work can serve as a starting point for those who wish to conduct research on such messages, those who wish to design better messages or those that aim to measure their effectiveness, more effectively.

Academic dishonesty is a widely acknowledged problem in tertiary education, and a range of safeguards and tools exist to both deter and detect cheating. During tests and examinations students are often monitored closely by invigilators, however incidents of misconduct still occur. Multiple-choice questions (MCQs) are particularly susceptible in this regard due to the simplicity with which answers can be communicated. Nonetheless, they are commonly used in large classes due to their convenience. In this work we investigate multiple-variant MCQs, where answer options are randomly selected from predefined sets. This format is less prone to student cheating and overcomes several weaknesses of similar methods. We evaluate this approach in two large computer science courses through student questionnaires and simple item analysis. We find that multiple-variant MCQ exams are perceived as an effective way to reduce cheating behavior, however not all question variants are equally difficult. This introduces concerns around fairness and we discuss approaches to mitigate these issues in the future.

When solving programming problems, novices are often not aware of where they are in the problem-solving process. For instance, students who misinterpret the problem prompt will most likely not form a valid conceptual model of the task and fail to make progress towards a working solution. Avoiding such errors, and recovering from them once they occur, requires metacognitive skills that enable students to reflect on their problem-solving processes. For these reasons, developing metacognitive awareness is crucially important for novice students. Previous research has shown that explicitly teaching key steps of programming problem-solving, and having students reflect on where they are in the problem-solving process, can help students complete future programming assignments. Such metacognitive awareness training can be done through personal tutoring, but can be difficult to implement without a high ratio of instructors to students. We explore a more scalable approach, making use of an automated assessment tool, and conduct a controlled experiment to see whether scaffolding the problem-solving process would increase metacognitive awareness and improve student performance. We collected all code submissions by students in both control and experimental groups, as well as data from direct observation using a think-aloud protocol. We found that students who received the intervention showed a higher degree of understanding of the problem prompt and were more likely to complete the programming task successfully.

Preparing for exams is an important yet stressful time for many students. Self-testing is known to be an effective preparation strategy, yet some students lack motivation to engage or persist in self-testing activities. Adding game elements to a platform supporting self-testing may increase engagement and, by extension, exam performance. We conduct a randomized controlled experiment (n=701) comparing the effect of two game elements -- a points system and a badge system -- used individually and in combination. We find that the badge system elicits significantly higher levels of voluntary self-testing activity and this effect is particularly pronounced amongst a relatively small cohort. Importantly, this increased activity translates to a significant improvement in exam scores. Our data supports a causal relationship between gamification and learning outcomes, mediated by self-testing behavior. This provides empirical support for Landers' theory of gamified learning when the gamified activity is conducted prior to measuring learning outcomes.

One means to determine whether a student understands the fundamentals of good object-oriented design is to assess designs the student has created. However, providing reliable assessment of designs efficiently is difficult due to the many viable designs that are possible and the high level of expertise required. Consequently, design assessment tends to be limited to identifying the most basic of design problems. We propose a technique---"object counts''---that involves counting the objects created at runtime. This is more efficient than manual grading because the data is gathered automatically and more reliable than using rubrics because it is based on objective data. The data is relevant because it captures the fundamental property of an object-oriented program---the creation of objects---and so provides good insight into the student's design decisions. This provides support for both summative and formative feedback. We demonstrate the technique on two corpora containing submissions for a typical first assignment of an introductory course on object-oriented design.

Online coding tools are an increasingly common feature of programming courses, providing students with rapid feedback and flexible practice opportunities and providing instructors with useful analytics. However, little research has explored the complexity of online exercises provided to students and the order in which students are exposed to new ideas. In this paper, we investigate the benefits of using a short sequence of practice exercises, each targeting a distinct topic, prior to having students solve a goal task that combines the concepts. As expected, we find students solve the goal task with fewer errors and in less time after completing the practice tasks. However, we also find that the practice tasks reduce the likelihood of students delaying work on the goal task, and these effects are particularly large for less-experienced students.

Debugging software is challenging, particularly for novices. Despite the importance of debugging, most novice programmers are not formally taught any debugging skills. This paper describes an online tool, Ladebug, that is designed to scaffold the learning of debugging skills. In this environment, students follow a structured debugging process to find and fix errors in predefined exercises. Overall, we find that students are positive about the tool, and report the exercises to be engaging and helpful.

Students and instructors expend significant effort, respectively, preparing to be examined and preparing students for exams. This paper investigates question authoring, where students create practice questions as a preparation activity prior to an exam, in an introductory programming context. The key contribution of this study as compared to previous work is an improvement to the design of the experiment. Students were randomly assigned the topics that their questions should target, removing a selection bias that has been a limitation of earlier work. We conduct a large-scale between-subjects experiment (n = 700) and find that students exhibit superior performance on exam questions that relate to the topics they were assigned when compared to those students preparing questions on other assigned topics.

Increasing enrollments and adoption of online resources have encouraged the use of multiple choice questions as a means of providing scalable assessment. However, in contexts where formative feedback is desired, standard multiple choice questions may lead students to a false sense of confidence -- a result of their small solution space and the temptation to guess. We propose the use of multiple-answer multiple choice questions in situations where formative feedback is desired and present evidence that these questions are well suited for that role.

Written exams are a common form of assessment in introductory programming courses. Creating exam questions is normally the responsibility of the course instructor, however the process of authoring such questions may be a useful learning activity in itself. We explored this idea with a randomized controlled experiment (n > 700) in which a group of first-year programming students generated practice questions prior to an exam. Even though all questions were available to every student in the course for practice, the group that generated the questions performed significantly better on the exam. The effects were most pronounced when students answered exam questions on topics that were targeted by questions they had generated. We suggest that some existing tools for computer science education may benefit from incorporating related activities.

Web-based programming exercises are a useful way for students to practice and master essential concepts and techniques presented in introductory programming courses. Although these systems are used fairly widely, we have a limited understanding of how students use these systems, and what can be learned from the data collected by these systems.

Debugging is an important skill for novice programmers to acquire. Error messages help novices to locate and correct errors, but compiler messages are frequently inadequate. We have developed a system that provides enhanced error messages, including concrete examples that illustrate the kind of error that has occurred and how that kind of error could be corrected. We evaluate the effectiveness of the enhanced error messages with a controlled empirical study and find no significant effect.

Badge-based achievement systems are being used increasingly to drive user participation and engagement across a variety of platforms and contexts. Despite positive anecdotal reports, there is currently little empirical evidence to support their efficacy in particular domains. With the recent rapid growth of tools for online learning, an interesting open question for educators is the extent to which badges can positively impact student participation.

We know that students solve problems in different ways, but we know little about the kinds of variation, or the degree of variation between these student generated solutions. In this paper, we propose a taxonomy that classifies the variation between correct student solutions in objective terms, and we show how the application of the taxonomy provides instructors with additional insight about the differences between student solutions. This taxonomy may be used to inform instructors in selecting examples of code for teaching purposes, and provides the possibility of automatically applying the taxonomy to existing solution sets.

Automatically-tested online programming exercises can be useful in introductory programming courses as self-tests to accompany readings, for in-class assessment, for skills development, and to provide additional practice for students who need it. CloudCoder (http://cloudcoder.org) is an effort to build a community based on an open-source programming exercise system (currently supporting C, Java, and Python) tightly integrated with a repository of freely-redistributable programming exercises written and used by members of the community. The goal of the project is to make programming exercises easy and free to incorporate into any programming course.

Identifying and correcting syntax errors is a challenge all novice programmers confront. As educators, the more we understand about the nature of these errors and how students respond to them, the more effective our teaching can be. It is well known that just a few types of errors are far more frequently encountered by students learning to program than most. In this paper, we examine how long students spend resolving the most common syntax errors, and discover that certain types of errors are not solved any more quickly by the higher ability students. Moreover, we note that these errors consume a large amount of student time, suggesting that targeted teaching interventions may yield a significant payoff in terms of increasing student productivity.

Recent research has focused on tools that support the creation, review and sharing of student-generated content for peer learning. However, we know little about the student perspective of such activities. In this paper, we identify what students believe is most helpful for their learning by analysing open-ended comments from students engaged in creating, answering and reviewing exam-style questions generated by their peers. Students report learning about content and appropriate standards of work, both individually and through interaction with peer generated resources.

Recent interest in student-centric pedagogies have resulted in the development of numerous tools that support student generated questions. Previous evaluations of such tools have reported strong correlations between student participation and exam performance, yet the level of student engagement with other learning activities in the course is a potential confounding factor. We show such correlations may be explained by other factors, and we undertake a deeper analysis that reveals evidence of the positive impact question-generation activities have on student performance.

PeerWise (PW) is an online tool that allows students in a course to collaborate and learn by creating, sharing, answering and discussing multiple-choice questions (MCQs). Previous studies of PW at the introductory level have shown that students in computing courses like it, and report statistically significant learning gains in courses taught by the investigators at different institutions. However, we recently conducted three quasi-experimental studies of PW use in upper-division computing courses in the U.S. and failed to replicate these positive results. In this paper we consider various factors that may impact the effectiveness of PW, including instructor engagement, usage requirements and subject-matter issues. We also report several positive results from other STEM courses at the same institution, discuss methodological issues pertaining to our recent studies and propose approaches for further investigation.

Mastering syntax is one of the earliest challenges facing the novice programmer. Problem solving and algorithms are the focus of many first year programming classes, leaving students to learn syntax on their own while they practice writing code. In this paper we investigate the frequency with which students encounter syntax errors during a drill and practice activity. We find that students struggle with syntax to a greater extent than we anticipated, even when writing short fragments of code.

Although a number of existing systems support student-generated multiple choice questions, such questions tend to focus on lower-order cognitive skills. Free response questions are frequently used to evaluate higher-order thinking, but supporting student-generated free-response questions is challenging. StudySieve is a web-based tool that extends student-generated questions to the free-response domain.

Drill and practice exercises enable students to master skills needed for more sophisticated programming. A barrier to providing such activities is the effort required to set up the programming environment. Testing is an important component to writing good software, but it is difficult to motivate students to write tests. In this paper we describe and evaluate Code Write, a web-based tool that provides drill and practice support for Java programming, and for which testing plays a central role in its use. We describe how we have used Code Write in a CS1 course, and demonstrate its effectiveness in providing good coverage of the language features presented in the course.

We demonstrate a web-based tool called PeerWise that supports students in the collaborative construction of an annotated test bank of multiple choice questions. PeerWise attempts to exploit the familiarity that students have with social software and the phenomenon of Web 2.0, allowing them to not only construct and answer one another's questions, but also to rate and discuss questions, to subscribe to individual question authors, and to accumulate points as a measure of their contributions.

Students in a large introductory programming course were asked twice to predict their scores on the final exam: once at the beginning of a six-week module, and once at the end. In between, students in only one of the two lecture streams recorded subjective confidence in their answers to individual questions on weekly quizzes. Students' predictions were moderately correlated with their scores. Students who attended more quizzes had not only higher exam scores, but improved their predictions more than those who attended fewer quizzes. Practice recording confidence on individual quiz questions did not yield significantly more improvement in exam predictions. Several findings from previous work are confirmed, including that women were significantly more underconfident than men.

PeerWise is a collaborative web-based system that engages students in the creation and evaluation of a test bank of multiple-choice questions. Previous studies involving two large CS1 courses in New Zealand have provided preliminary evidence that PeerWise usage is positively correlated with exam performance. In addition, it was reported that the student generated assessment questions were mostly free from errors and were clearly written, and the students appeared to positively value the system. Here we report on the first use of PeerWise in a CS1.5 course (second programming course) in the United States. Although the usage model was modified slightly to accommodate pedagogical factors, we highlight similar positive outcomes to those observed in New Zealand. Of particular note, students who were most active using PeerWise improved their rank in class relative to their peers who were less active.

While popularity of peer assessment in Computer Science has increased in recent years, the validity of peer assessed marks remain a significant concern to instructors and source of anxiety to students. We report here on a large-scale study (1,500 students and 10,000 reviews) involving three introductory programming classes which recorded grades and feedback comments for both student and tutor reviews of novice programs. Using a paired analysis, we compare the quantitative marks given by students with those given by tutors, for both functional and non-functional aspects of the program. We also report on an analysis of the lexical sophistication of feedback comments.

A recent approach to engaging students in deep learning involves an online tool, PeerWise, through which students contribute multiple-choice questions to a shared question bank. Earlier work demonstrated a strong correlation between the use of PeerWise and student performance. In this study we investigate the quality of the MCQ repository created by students in an introductory programming course by analysing the range of topics on which students chose to write questions (i.e.~the repository coverage) without guidance from an instructor. We assess the repository coverage by comparing it with a common list of typical introductory programming topics, and by looking at its extent. We find that, despite having freedom to choose any topic, students created a repository that covered all the major topics in the curriculum.

Computer games are widely used in programming assignments to motivate students. Traditionally, these have been text-based games such as hangman, but as Java has become widespread, the use of graphics and graphical games has increased correspondingly. We report on a framework used as scaffolding to help students understand how to design and implement a variety of interactive games. We share our experiences using this framework for assignments over a number of years.

PeerWise is a system in which students create multiple choice questions and answer those created by their peers. In this paper, we report on some quantitative results which suggest that students who use PeerWise actively perform better in final examinations than students who are not active. We note a significant correlation between performance in written (not just multiple choice) questions and PeerWise activity, suggesting that active use of the system may contribute to deep (and not just drill-and-practise) learning.

Common exam practice centres around two question types: code tracing (reading) and code writing. It is commonly believed that code tracing is easier than code writing, but it seems obvious that different skills are needed for each. These problems also differ in their value on an exam. Pedagogically, code tracing on paper is an authentic task whereas code writing on paper is less so. Yet, few instructors are willing to forgo the code writing question on an exam. Is there another way, easier to grade, that captures the "problem solving through code creation process" we wish to examine? In this paper we propose Parson's puzzle-style problems for this purpose. We explore their potential both qualitatively, through interviews, and quantitatively through a set of CS1 exams. We find notable correlation between Parsons scores and code writing scores. We find low correlation between code writing and tracing and between Parsons and tracing. We also make the case that marks from a Parsons problem make clear what students don't know (specifically, in both syntax and logic) much less ambiguously than marks from a code writing problem. We make recommendations on the design of Parsons problems for the exam setting, discuss their potential uses and urge further investigations of Parsons problems for assessment of CS1 students.

PeerWise is a web-based system that supports the creation of student-generated test banks of multiple choice questions. Students contribute question stems and answers, provide explanations, answer questions contributed by other students, rate questions for difficulty and quality, and participate in on-line discussions of all these activities.