Background. Large language models are increasingly deployed as tutors in introductory programming courses, yet evidence that they actually improve learning remains thin, and their tendency to shortcut productive struggle raises concerns about pedagogical harm. Self-regulated learning (SRL) and cognitive engagement (CE) frameworks offer a principled way to address this, but whether embedding them in system prompts actually changes how students learn is an open question.

Code-generating Artificial Intelligence has gained popularity within both professional and educational programming settings over the past several years. While research and pedagogy are beginning to cope with this change, computing students are left to bear the unforeseen consequences of AI amidst a dearth of empirical evidence about its effects. Though pair programming between students is well studied and known to be beneficial to self-efficacy and academic achievement, it remains underutilized and further threatened by the proposition that AI can replace a human programming partner. In this paper, we present a controlled pair programming study with 22 participants who wrote Python code under time pressure in teams of two and individually with GitHub Copilot for 20 minutes each. They were incentivized by bonus compensation to balance performance with understanding and were retested individually on the programming tasks after a retention interval of one week. Subjective measures of workload and emotion as well as objective measures of performance and learning (retest performance) were collected. When first programming with AI, participants clearly performed better and had an easier time. On retest, they were not dramatically worse in raw score from the human-human condition, but human-AI pairs lost more of their initial advantage. Additionally, the emotional effect of the human teammate was significantly more positive and arousing as compared to working with Copilot. We recommend that educators strongly consider revisiting pair programming as an educational tool in addition to embracing modern AI.

Learning to communicate with code-generating AI models is an emerging skill for novice programmers. One recent pedagogical approach, Prompt Problems, has students solve computational tasks by writing natural-language prompts for code-generating AI models. However, little is known about the specific prompt-level mistakes novice programmers make, the kinds of computational details they fail to communicate, and what strategies they use to recover when generated code is incorrect. In a CS1 course, we studied attempts by more than 900 students to solve dialogue-based Prompt Problems. We analyzed student reflections, unsuccessful prompts, and reported debugging strategies. Compared to traditional coding tasks, students generally found prompting easier, more enjoyable, and better targeted at developing problem-solving skills. The most common mistakes are related to the omission of key details, suggesting both a failure to acknowledge their importance and over-reliance on AI to infer them. When prompts failed, students focused more on clarifying their intent and reflecting on the provided problem details than on tracing generated code or examining test cases.

The rise of generative artificial intelligence (GenAI) has sparked a rapid change in computing curricula and teaching approaches. GenAI coding tools can accurately complete assignments, answer test questions, and perform other tasks traditionally associated with learning programming, especially at the introductory level. Because GenAI is still so new, researchers investigating student usage of GenAI have used informal rubrics and questionnaires. To advance, the field needs validated instruments that measure student perception and use of GenAI. This paper presents the development and initial validation of an instrument to measure self-efficacy while using GenAI to learn programming. Self-efficacy is an important construct in education research because it robustly correlates with student success, across disciplines and ages, including undergraduate computing education. Computing education researchers have presented several validated self-efficacy instruments, most recently by Steinhorst et al. in 2020. Critically, this instrument was created before the rise of GenAI’s popularity in 2022. To complement this instrument, we created a GenAI scale similar in style to the Steinhorst self-efficacy instrument, consisting originally of 11 items and revised to 5 items. We report two important findings in this paper. First, we found strong support for the validity of the existing Steinhorst instrument in a new context, specifically an introductory programming course that fully integrates GenAI. Second, the new GenAI scale shows strong internal reliability, discriminant validity with items in the Steinhorst subscales, and criterion validity with students’ GenAI usage patterns. Based on statistical analysis and cognitive probing interviews, we argue for the validity of the five-item scale to measure students’ GenAI self-efficacy in the context of programming.

Modern programming tools use generative AI (GenAI) to suggest code to the user as they type, interrupting their problem-solving behavior and undermining the development of their programming critical thinking skills. In this paper, we present a scaffolded programming exercise designed to support student differentiation between good and bad GenAI code suggestions based on negative expertise–that identifying why an answer is wrong is part of developing conceptual knowledge. We compare a version of the tool that showed one suggestion (correct or not), to a version that showed three suggestions (one of which was correct). We present results on performance and error rates as well as qualitative findings centered on Pintrich and DeGroot’s theory of self-regulation. Students reported that the single suggestion version better aligned with industry tools and presented a lower cognitive load. Students also reported that the multiple suggestion version caused them to slow down and think critically about the line under consideration, the overall purpose of the code, and the benefits of planning.

Novice CS students can have a very wide range of starting knowledge. Bringing students to a professional level of practice requires an active process for students to develop their skills and knowledge, individually and collaboratively. Through the use of social constructivism, a theory of learning based on the benefits of collaborative social interactions between learners, these interactions allow learners to develop their knowledge by making sense of the interactions. This can take many forms, including discussing concepts, working on collaborative projects, and receiving timely and accurate feedback from their peer group and supervising instructors. The gap between what a learner is capable of doing without support and what the learner could perform with the assistance of the more knowledgeable other (MKO)---whether peer or teacher---is defined as the zone of proximal development (ZPD). Recent advances in chatbot technologies, especially those based on large language models (LLMs), and various institutional incentives naturally guide focus towards AI agents' potential as participants in collaborative social interactions. In this paper, we identify the characteristics and affordances of LLMs in how they might successfully support a computer science student through the ZPD, drawing on existing theory across key disciplines and knowledge of the characteristics of LLMs. We propose a group of models for constructive uses of this new technology, which will support social constructivism well, and provide more general guidelines for the use of LLMs.

Publisher Copyright: © 2026 Owner/Author.

Large language models (LLMs) are increasingly integrated into computing education, yet nearly all prior research has focused on text-based interactions.As voice-enabled interfaces become more capable and more common, there is growing interest in understanding how voice input might shape students' use of LLM-powered tools.In this exploratory study, we investigated how introductory programming students interact with Prompt Problems, which are programming tasks that require crafting natural-language prompts to generate correct code.Students (N = 919) solved a series of Prompt Problems with the freedom to select or switch between text and voice input modalities.We collected their prompt submissions as well as post-activity survey responses, then analysed differences in prompt accuracy, persistence, and perspectives by modality.For two of the three problems, we found that students who typed their prompts using text were more likely to have those prompts succeed on the first attempt than students who submitted unedited voice prompts.There was no difference in success rate if students edited their transcribed voice prompts before submission.Across the problems, we found evidence that students who tried voice prompting varied in their usage of modality -perhaps indicating a complementary, or non-preferential approach.However, most students only tried and reported preferring text.Our qualitative analysis revealed how students' perceived the roles of voice and text input in shaping their problem-solving process, as well as the reported drawbacks and advantages of each modality.We discuss implications for future multimodal tools and instructional design in computing education.

Adaptive programming practice often relies on fixed libraries of worked examples and practice problems, which require substantial authoring effort and may not correspond well to the logical errors and partial solutions students produce while writing code. As a result, students may receive learning content that does not directly address the concepts they are working to understand, while instructors must either invest additional effort in expanding content libraries or accept a coarse level of personalization. We present an approach for knowledge-component (KC) guided educational content generation using pattern-based KCs extracted from student code. Given a problem statement and student submissions, our pipeline extracts recurring structural KC patterns from students' code through AST-based analysis and uses them to condition a generative model. In this study, we apply this approach to worked example generation, and compare baseline and KC-conditioned outputs through expert evaluation. Results suggest that KC-conditioned generation improves topical focus and relevance to students' underlying logical errors, providing evidence that KC-based steering of generative models can support personalized learning at scale.

Large Language Models (LLMs) have shown the potential to generate code explanations that surpass those of peers in quality, offering promising opportunities for computer science education. Inspired by this, we explore whether combining multiple diverse explanations, each emphasizing distinct aspects (e.g., function, concept, goal), can enhance students' understanding of programming exercises compared to generic explanations that do not emphasize distinct conceptual aspects. Insights from other fields, such as computational creativity, suggest that diverse ideas may be more beneficial than relying solely on a single, high-quality option. Variation Theory holds that learners grasp a concept when they see systematic variation that exposes its critical features, helping them distinguish it from related ideas. In creative domains, uniform or homogeneous exemplars can lead to design fixation, whereas varied inputs support more flexible reasoning. In a study with 971 first-year computing students, participants were randomly assigned either diverse or generic LLM-generated explanations for two programming exercises. Students completed multiple-choice (MCQ) and open-ended (OE) questions for each exercise to assess understanding, followed by Likert-scale questions and OE reflections to understand preferences and perception. Across participants, performance was consistently 7.7% higher when students received diverse explanations, and there was no difference in perceived cognitive load. Performance on the closed-form multiple-choice questions was similar for diverse and generic explanations.

We present a reproducible pipeline that aligns CS1 textbook sections with problems from a public dataset via a Knowledge Component (KC) -a single conceptual skill required for problem solving- ontology. It assigns KCs to sections and problems, respects the prerequisite order to avoid inserting problems too early, and generates tips for not-yet-taught concepts. We evaluate three KC assignment strategies: embedding-only, embedding with a Large Language Model (LLM) tie-breaker, and direct LLM assignment. We find direct assignment matches or exceeds human annotators. Our results show that constrained LLMs can enrich CS1 textbooks with curriculum-aware practice problems.

Cryptic compiler error messages continue to present a significant barrier for novice programmers, especially in foundational languages like C. Although large language models (LLMs) can generate accurate and comprehensible error explanations, their computational requirements, propensity for over-assistance, and privacy concerns constrain their suitability for widespread adoption in educational tools. This work investigates how Supervised Fine-Tuning (SFT) can enhance the performance of smaller, open-source models when explaining C compiler errors to students in introductory programming courses (CS1/2). We derive a training dataset of 40,000 input-output pairs from CS1/2 student C compiler errors to fine-tune three open-source models: Qwen3-4B, Llama-3.1-8B, and Qwen3-32B. Model performance was assessed through a dual evaluation framework involving expert human reviewers and a large-scale automated analysis of 8,000 responses using an ensemble of models as judges. Our results indicate that SFT significantly improves both expert and LLM-as-judge ratings in smaller open-source models, with reduced gains in the larger model. We analyse the trade-offs between model size and quality, and validate LLM-as-judge by demonstrating inter-rater agreement with experts. Our findings demonstrate that fine-tuning smaller models on high-quality data is a viable strategy for creating specialised pedagogical tools. We provide a replicable methodology for enabling broader access to advanced AI capabilities within educational contexts, especially with smaller, economical models.

Large language models (LLMs) have demonstrated strong capabilities in language understanding and generation, and their potential in educational contexts is increasingly being explored. One promising area is learnersourcing, where students engage in creating their own educational content, such as multiple-choice questions. A critical step in this process is generating effective explanations for the solutions to these questions, as such explanations aid in peer understanding and promote deeper conceptual learning. However, students often find it difficult to craft high-quality explanations due to limited understanding or gaps in their subject knowledge. To support this task, we introduce ``ILearner-LLM,'' a framework that uses iterative enhancement with LLMs to improve generated explanations. The framework combines an explanation generation model and an explanation evaluation model fine-tuned using student preferences for quality, where feedback from the evaluation model is fed back into the generation model to refine the output. Our experiments with LLaMA2-13B and GPT-4 using five large datasets from the PeerWise MCQ platform show that ILearner-LLM produces explanations of higher quality that closely align with those written by students. Our findings represent a promising approach for enriching the learnersourcing experience for students and for leveraging the capabilities of large language models for educational applications.

Introductory programming courses often rely on small code-writing exercises that have clearly specified problem statements. This limits opportunities for students to practice how to clarify ambiguous requirements -- a critical skill in real-world programming. In addition, the emerging capabilities of large language models (LLMs) to produce code from well-defined specifications may harm student engagement with traditional programming exercises. This study explores the use of ``Probeable Problems'', automatically gradable tasks that have deliberately vague or incomplete specifications. Such problems require students to submit test inputs, or `probes', to clarify requirements before implementation. Through analysis of over 40,000 probes in an introductory course, we identify patterns linking probing behaviors to task success. Systematic strategies, such as thoroughly exploring expected behavior before coding, resulted in fewer incorrect code submissions and correlated with course success. Feedback from nearly 1,000 participants highlighted the challenges and real-world relevance of these tasks, as well as benefits to critical thinking and metacognitive skills. Probeable Problems are easy to set up and deploy at scale, and help students recognize and resolve uncertainties in programming problems.

Collaboration is a crucial part of computing education. The increase in AI capabilities over the last couple of years is bound to profoundly affect all aspects of systems and software engineering, including collaboration. In this position paper, we consider a scenario where AI agents would be able to take on any role in collaborative processes in computing education. We outline these roles, the activities and group dynamics that software development currently include, and discuss if and in what way AI could facilitate these roles and activities. The goal of our work is to envision and critically examine potential futures. We present scenarios suggesting how AI can be integrated into existing collaborations. These are contrasted by design fictions that help demonstrate the new possibilities and challenges for computing education in the AI era.

Publisher Copyright: © 2025 Copyright held by the owner/author(s).

Most of the attention on GenAI in computing education has focused on programming-centric tasks, such as code generation, giving feedback on code, or providing synthetic programming partners. Yet in advanced software engineering and project courses, interpersonal skills such as team meetings or customer interviews are equally important but difficult and instructor-intensive to teach realistically. GenAI presents the possibility of scaffolding the teaching of some of these practices by enabling exercises in which students develop the ability to investigate a topic by iteratively asking questions to find a solution. The goal is to create scenarios in which students train to interact with humans in real-world situations, simulating these interactions in a controlled, guided environment. These simulations could help students practice and refine ''soft skills,'' such as teamwork and interviewing, by mimicking the types of exchanges and problem-solving they would encounter in professional environments. This approach allows learners to engage in realistic communication exercises, improving their ability to handle complex, interpersonal tasks through repeated practice with AI-guided feedback. As an example, we envision examples that include requirements elicitation with customers, development team meetings, and discussion with potential investors, to name just a few.

Code reading and comprehension skills are essential for novices learning programming, and explain-in-plain-English tasks (EiPE) are a well-established approach for assessing these skills. However, manual grading of EiPE tasks is time-consuming and this has limited their use in practice. To address this, we explore an approach where students explain code samples to a large language model (LLM) which generates code based on their explanations. This generated code is then evaluated using test suites, and shown to students along with the test results. We are interested in understanding how automated formative feedback from an LLM guides students' subsequent prompts towards solving EiPE tasks. We analyzed 177 unique attempts on four EiPE exercises from 21 students, looking at what kinds of mistakes they made and how they fixed them. We found that when students made mistakes, they identified and corrected them using either a combination of the LLM-generated code and test case results, or they switched from describing the purpose of the code to describing the sample code line-by-line until the LLM-generated code exactly matched the obfuscated sample code. Our findings suggest both optimism and caution with the use of LLMs for unmonitored formative feedback. We identified false positive and negative cases, helpful variable naming, and clues of direct code recitation by students. For most students, this approach represents an efficient way to demonstrate and assess their code comprehension skills. However, we also found evidence of misconceptions being reinforced, suggesting the need for further work to identify and guide students more effectively.

The emergence of large language models (LLMs) has transformed research and practice across a wide range of domains. Within the computing education research (CER) domain, LLMs have garnered significant attention, particularly in the context of learning programming. Much of the work on LLMs in CER, however, has focused on applying and evaluating proprietary models. In this article, we evaluate the efficiency of open-source LLMs in generating high-quality feedback for programming assignments and judging the quality of programming feedback, contrasting the results with proprietary models. Our evaluations on a dataset of students' submissions to introductory Python programming exercises suggest that state-of-the-art open-source LLMs are nearly on par with proprietary models in both generating and assessing programming feedback. Additionally, we demonstrate the efficiency of smaller LLMs in these tasks and highlight the wide range of LLMs accessible, even for free, to educators and practitioners.

Programming skills are typically developed through completing various hands-on exercises. Such programming problems can be contextualized to students’ interests and cultural backgrounds. Prior research in educational psychology has demonstrated that context personalization of exercises stimulates learners’ situational interests and positively affects their engagement. However, creating a varied and comprehensive set of programming exercises for students to practice on is a time-consuming and laborious task for computer science educators. Previous studies have shown that large language models can generate conceptually and contextually relevant programming exercises. Thus, they offer a possibility to automatically produce personalized programming problems to fit students’ interests and needs. This article reports on a user study conducted in an elective introductory programming course that included contextually personalized programming exercises created with GPT-4. The quality of the exercises was evaluated by both the students and the authors. Additionally, this work investigated student attitudes towards the created exercises and their engagement with the system. The results demonstrate that the quality of exercises generated with GPT-4 was generally high. What is more, the course participants found them engaging and useful. This suggests that AI-generated programming problems can be a worthwhile addition to introductory programming courses, as they provide students with a practically unlimited pool of practice material tailored to their personal interests and educational needs.

Novice programmers often struggle through programming problem solving due to a lack of metacognitive awareness and strategies. Previous research has shown that novices can encounter multiple metacognitive difficulties while programming, such as forming incorrect conceptual models of the problem or having a false sense of progress after testing their solution. Novices are typically unaware of how these difficulties are hindering their progress. Meanwhile, many novices are now programming with generative AI (GenAI), which can provide complete solutions to most introductory programming problems, code suggestions, hints for next steps when stuck, and explain cryptic error messages. Its impact on novice metacognition has only started to be explored. Here we replicate a previous study that examined novice programming problem solving behavior and extend it by incorporating GenAI tools. Through 21 lab sessions consisting of participant observation, interview, and eye tracking, we explore how novices are coding with GenAI tools. Although 20 of 21 students completed the assigned programming problem, our findings show an unfortunate divide in the use of GenAI tools between students who did and did not struggle. Some students who did not struggle were able to use GenAI to accelerate, creating code they already intended to make, and were able to ignore unhelpful or incorrect inline code suggestions. But for students who struggled, our findings indicate that previously known metacognitive difficulties persist, and that GenAI unfortunately can compound them and even introduce new metacognitive difficulties. Furthermore, struggling students often expressed cognitive dissonance about their problem solving ability, thought they performed better than they did, and finished with an illusion of competence. Based on our observations from both groups, we propose ways to scaffold the novice GenAI experience and make suggestions for future work.

Publisher Copyright: © 2024 Owner/Author.

Grasping complex computing concepts often poses a challenge for students who struggle to anchor these new ideas to familiar experiences and understandings. To help with this, a good analogy can bridge the gap between unfamiliar concepts and familiar ones, providing an engaging way to aid understanding. However, creating effective educational analogies is difficult even for experienced instructors. We investigate to what extent large language models (LLMs), specifically ChatGPT, can provide access to personally relevant analogies on demand. Focusing on recursion, a challenging threshold concept, we conducted an investigation analyzing the analogies generated by more than 350 first-year computing students. They were provided with a code snippet and tasked to generate their own recursion-based analogies using ChatGPT, optionally including personally relevant topics in their prompts. We observed a great deal of diversity in the analogies produced with student-prescribed topics, in contrast to the otherwise generic analogies, highlighting the value of student creativity when working with LLMs. Not only did students enjoy the activity and report an improved understanding of recursion, but they described more easily remembering analogies that were personally and culturally relevant.

Introducing students to new concepts in computer science can often be challenging, as these concepts may differ significantly from their existing knowledge and conceptual understanding. To address this, we employed analogies to help students connect new concepts to familiar ideas. Specifically, we generated analogies using large language models (LLMs), namely ChatGPT, and used them to help students make the necessary connections. In this poster, we present the results of our survey, in which students were provided with two analogies relating to different computing concepts, and were asked to describe the extent to which they were accurate, interesting, and useful. This data was used to determine how effective LLM-generated analogies can be for teaching computer science concepts, as well as how responsive students are to this approach.

Large language models (LLMs) have shown great potential for the automatic generation of feedback in a wide range of computing contexts. However, concerns have been voiced around the privacy and ethical implications of sending student work to proprietary models. This has sparked considerable interest in the use of open source LLMs in education, but the quality of the feedback that such open models can produce remains understudied. This is a concern as providing flawed or misleading generated feedback could be detrimental to student learning. Inspired by recent work that has utilised very powerful LLMs, such as GPT-4, to evaluate the outputs produced by less powerful models, we conduct an automated analysis of the quality of the feedback produced by several open source models using a dataset from an introductory programming course. First, we investigate the viability of employing GPT-4 as an automated evaluator by comparing its evaluations with those of a human expert. We observe that GPT-4 demonstrates a bias toward positively rating feedback while exhibiting moderate agreement with human raters, showcasing its potential as a feedback evaluator. Second, we explore the quality of feedback generated by several leading open-source LLMs by using GPT-4 to evaluate the feedback. We find that some models offer competitive performance with popular proprietary LLMs, such as ChatGPT, indicating opportunities for their responsible use in educational settings.

We explored how undergraduate introductory programming students naturalistically used generative AI to solve programming problems. We focused on the relationship between their use of AI to their self-regulation strategies, self-efficacy, and fear of failure in programming. In this repeated-measures, mixed-methods research, we examined students' patterns of using generative AI with qualitative student reflections and their self-regulation, self-efficacy, and fear of failure with quantitative instruments at multiple times throughout the semester. We also explored the relationships among these variables to learner characteristics, perceived usefulness of AI, and performance. Overall, our results suggest that student factors affect their baseline use of AI. In particular, students with higher self-efficacy, lower fear of failure, or higher prior grades tended to use AI less or later in the problem-solving process and rated it as less useful than others. Interestingly, we found no relationship between students' self-regulation strategies and their use of AI. Students who used AI less or later in problem-solving also had higher grades in the course, but this is most likely due to prior characteristics as our data do not suggest that this is a causal relationship.

Generative AI (GenAI) has seen great advancements in the past two years and the conversation around adoption is increasing. Widely available GenAI tools are disrupting classroom practices as they can write and explain code with minimal student prompting. While most acknowledge that there is no way to stop students from using such tools, a consensus has yet to form on how students should use them if they choose to do so. At the same time, researchers have begun to introduce new pedagogical tools that integrate GenAI into computing curricula. These new tools offer students personalized help or attempt to teach prompting skills without undercutting code comprehension. This working group aims to detail the current landscape of education-focused GenAI tools and teaching approaches, present gaps where new tools or approaches could appear, identify good practice-examples, and provide a guide for instructors to utilize GenAI as they continue to adapt to this new era.

Folk-wisdom in software engineering suggests that small functions that adhere to the principle of single-responsibility have several advantages over longer, monolithic functions, including improvement in code comprehension. Despite this widespread view, empirical research on the impact of functional decomposition on understanding code is sparse, yet it is central to software development practices.

Large language models (LLMs) are revolutionizing the field of computing education with their powerful code-generating capabilities. Traditional pedagogical practices have focused on code writing tasks, but there is now a shift in importance towards reading, comprehending and evaluating LLM-generated code. Alongside this shift, an important new skill is emerging -- the ability to solve programming tasks by constructing good prompts for code-generating models. In this work we introduce a new type of programming exercise to hone this nascent skill: 'Prompt Problems'. Prompt Problems are designed to help students learn how to write effective prompts for AI code generators. A student solves a Prompt Problem by crafting a natural language prompt which, when provided as input to an LLM, outputs code that successfully solves a specified programming task. We also present a new web-based tool called Promptly which hosts a repository of Prompt Problems and supports the automated evaluation of prompt-generated code. We deploy Promptly in one CS1 and one CS2 course and describe our experiences, which include student perceptions of this new type of activity and their interactions with the tool. We find that students are enthusiastic about Prompt Problems, and appreciate how the problems engage their computational thinking skills and expose them to new programming constructs. We discuss ideas for the future development of new variations of Prompt Problems, and the need to carefully study their integration into classroom practice.

In the past year, large language models (LLMs) have taken the world by storm, demonstrating their potential as a transformative force in many domains including computing education. Computing education researchers have found that LLMs can solve most assessments in introductory programming courses, including both traditional code writing tasks and other popular tasks such as Parsons problems. As more and more students start to make use of LLMs, the question instructors might ask themselves is "what can I do?". We propose that one promising way forward is to integrate LLMs into teaching practice, providing all students with an equal opportunity to learn how to interact productively with LLMs as well as encounter and understand their limitations. In this workshop, we first present state-of-the-art research results on how to utilize LLMs in computing education practice, after which participants will take part in hands-on activities using LLMs. We end the workshop by brainstorming ideas with participants around adapting their classrooms to most effectively integrate LLMs while avoiding some common pitfalls.

The panel comprises a diverse set of Computing educators working on AI in education. The panelists will address four areas of AI in Computing education: 1) AI for introductory CS classrooms, 2) Investigating opportunities presented by LLMs, 3) LLM-based tool development, and 4) Ethics and inclusion in AI curriculum. The panel will share experiences and discuss opportunities and challenges in AI education with the community.

The emergence of publicly accessible large language models (LLMs) such as ChatGPT poses unprecedented risks of new types of plagiarism and cheating where students use LLMs to solve exercises for them. Detecting this behavior will be a necessary component in introductory computer science (CS1) courses, and educators should be well-equipped with detection tools when the need arises. However, ChatGPT generates code non-deterministically, and thus, traditional similarity detectors might not suffice to detect AI-created code. In this work, we explore the affordances of Machine Learning (ML) models for the detection task. We used an openly available dataset of student programs for CS1 assignments and had ChatGPT generate code for the same assignments, and then evaluated the performance of both traditional machine learning models and Abstract Syntax Tree-based (AST-based) deep learning models in detecting ChatGPT code from student code submissions. Our results suggest that both traditional machine learning models and AST-based deep learning models are effective in identifying ChatGPT-generated code with accuracy above 90%. Since the deployment of such models requires ML knowledge and resources that are not always accessible to instructors, we also explore the patterns detected by deep learning models that indicate possible ChatGPT code signatures, which instructors could possibly use to detect LLM-based cheating manually. We also explore whether explicitly asking ChatGPT to impersonate a novice programmer affects the code produced. We further discuss the potential applications of our proposed models for enhancing introductory computer science instruction.

In a previous Birds of a Feather discussion, we delved into the nascent applications of generative AI, contemplating its potential and speculating on future trajectories. Since then, the landscape has continued to evolve revealing the capabilities and limitations of these models. Despite this progress, the computing education research community still faces uncertainty around pivotal aspects such as (1) academic integrity and assessments, (2) curricular adaptations, (3) pedagogical strategies, and (4) the competencies students require to instill responsible use of these tools. The goal of this Birds of a Feather discussion is to unravel these pressing and persistent issues with computing educators and researchers, fostering a collaborative exploration of strategies to navigate the educational implications of advancing generative AI technologies. Aligned with this goal of building an inclusive learning community, our BoF is led by globally distributed leaders to facilitate multiple coordinated discussions that can lead to a broader conversation about the role of LLMs in CS education.

Large language models (LLMs) have recently taken many fields, including computer science, by storm. Most recent work on LLMs in computing education has shown that they are capable of solving most introductory programming (CS1) exercises, exam questions, Parsons problems, and several other types of exercises and questions. Some work has investigated the ability of LLMs to solve CS2 problems as well. However, it remains unclear how well LLMs fare against more advanced upper-division coursework, such as proofs in algorithms courses. After all, while known to be proficient in many programming tasks, LLMs have been shown to have more difficulties in forming mathematical proofs.

Much of the recent work investigating large language models and AI Code Generation tools in computing education has focused on assessing their capabilities for solving typical programming problems and for generating resources such as code explanations and exercises. If progress is to be made toward the inevitable lasting pedagogical change, there is a need for research that explores the instructor voice, seeking to understand how instructors with a range of experiences plan to adapt. In this paper, we report the results of an interview study involving 12 instructors from Australia, Finland and New Zealand, in which we investigate educators' current practices, concerns, and planned adaptations relating to these tools. Through this empirical study, our goal is to prompt dialogue between researchers and educators to inform new pedagogical strategies in response to the rapidly evolving landscape of AI code generation tools.

Recent developments in deep learning have resulted in code-generation models that produce source code from natural language and code-based prompts with high accuracy. This is likely to have profound effects in the classroom, where novices learning to code can now use free tools to automatically suggest solutions to programming exercises and assignments. However, little is currently known about how novices interact with these tools in practice. We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical introductory programming (CS1) assignment. Through observations and interviews we explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students. We consider design implications of these findings, specifically in terms of how tools like Copilot can better support and scaffold the novice programming experience.

Background and Context: Over the past year, large language models (LLMs) have taken the world by storm. In computing education, like in other walks of life, many opportunities and threats have emerged as a consequence.

Background and Context: Struggling with programming assignments while learning to program is a common phenomenon in programming courses around the world. Supporting struggling students is a common theme in Computing Education Research (CER), where a wide variety of support methods have been created and evaluated. An important stream of research here focuses on program repair, where methods for automatically fixing erroneous code are used for supporting students as they debug their code. Work in this area has so far assessed the performance of the methods by evaluating the closeness of the proposed fixes to the original erroneous code. The evaluations have mainly relied on the use of edit distance measures such as the sequence edit distance and there is a lack of research on which distance measure is the most appropriate.

Reasoning about code and explaining its purpose are fundamental skills for computer scientists. There has been extensive research in the field of computing education on the relationship between a student's ability to explain code and other skills such as writing and tracing code. In particular, the ability to describe at a high-level of abstraction how code will behave over all possible inputs correlates strongly with code writing skills. However, developing the expertise to comprehend and explain code accurately and succinctly is a challenge for many students. Existing pedagogical approaches that scaffold the ability to explain code, such as producing exemplar code explanations on demand, do not currently scale well to large classrooms. The recent emergence of powerful large language models (LLMs) may offer a solution. In this paper, we explore the potential of LLMs in generating explanations that can serve as examples to scaffold students' ability to understand and explain code. To evaluate LLM-created explanations, we compare them with explanations created by students in a large course (n ≈ 1000) with respect to accuracy, understandability and length. We find that LLM-created explanations, which can be produced automatically on demand, are rated as being significantly easier to understand and more accurate summaries of code than student-created explanations. We discuss the significance of this finding, and suggest how such models can be incorporated into introductory programming education.

In this article, we report results from a randomized controlled trial where novice programmers completed code mimicking exercises -- writing and modifying code shown to them -- designed to help learn the basics of how variables work. Using a tailored code writing system with feedback on program correctness, we conducted a two-group design study where only one of the groups could see the program output and feedback on the correctness of the program they wrote, while the other group just saw feedback on correctness. Learning gain was measured using a code-reading multiple choice questionnaire as both a pretest and a posttest. Our data suggests that being able to see program output leads to higher learning gains for novices, when compared to just being able to see feedback on the correctness of the code. For more experienced students, we observed benefits from code mimicking in both groups, without a strong distinction between being able to see the output and not being able to see the output. Based on our experiment, we recommend that environments used by novices for learning programming should encourage -- or even require -- running the code before allowing submitting the program for assessment.

Recent breakthroughs in deep learning have led to the emergence of generative AI models that exhibit extraordinary performance at producing human-like outputs. Using only simple input prompts, it is possible to generate novel text, images, video, music, and source code, as well as tackle tasks such as answering questions and translating and summarising text.

The recent advent of highly accurate and scalable large language models (LLMs) has taken the world by storm. From art to essays to computer code, LLMs are producing novel content that until recently was thought only humans could produce. Recent work in computing education has sought to understand the capabilities of LLMs for solving tasks such as writing code, explaining code, creating novel coding assignments, interpreting programming error messages, and more. However, these technologies continue to evolve at an astonishing rate leaving educators little time to adapt. This working group seeks to document the state-of-the-art for code generation LLMs, detail current opportunities and challenges related to their use, and present actionable approaches to integrating them into computing curricula.

The recent emergence of code generation tools powered by large language models has attracted wide attention. Models such as OpenAI Codex can take natural language problem descriptions as input and generate highly accurate source code solutions, with potentially significant implications for computing education. Given the many complexities that students face when learning to write code, they may quickly become reliant on such tools without properly understanding the underlying concepts. One popular approach for scaffolding the code writing process is to use Parsons problems, which present solution lines of code in a scrambled order. These remove the complexities of low-level syntax, and allow students to focus on algorithmic and design-level problem solving. It is unclear how well code generation models can be applied to solve Parsons problems, given the mechanics of these models and prior evidence that they underperform when problems include specific restrictions. In this paper, we explore the performance of the Codex model for solving Parsons problems over various prompt variations. Using a corpus of Parsons problems we sourced from the computing education literature, we find that Codex successfully reorders the problem blocks about half of the time, a much lower rate of success when compared to prior work on more free-form programming tasks. Regarding prompts, we find that small variations in prompting have a noticeable effect on model performance, although the effect is not as pronounced as between different problems.

A key part of learning to program is learning to understand programming error messages. They can be hard to interpret and identifying the cause of errors can be time-consuming. One factor in this challenge is that the messages are typically intended for an audience that already knows how to program, or even for programming environments that then use the information to highlight areas in code. Researchers have been working on making these errors more novice friendly since the 1960s, however progress has been slow. The present work contributes to this stream of research by using large language models to enhance programming error messages with explanations of the errors and suggestions on how to fix them. Large language models can be used to create useful and novice-friendly enhancements to programming error messages that sometimes surpass the original programming error messages in interpretability and actionability. These results provide further evidence of the benefits of large language models for computing educators, highlighting their use in areas known to be challenging for students. We further discuss the benefits and downsides of large language models and highlight future streams of research for enhancing programming error messages.

The computing education (CEd) research community's growth is fueled in part by the growing number of CEd Ph.D. graduates, who are also increasingly entering postdoctoral positions and fellowships. While prior work has shown that postdoctoral researchers have positive effects on research labs, provide essential support to graduate students, and serve critical roles in the research ecosystem, there are no existing support structures for postdoctoral researchers within CEd. As postdocs navigate the challenges of their new positions alongside the demands of research and job searches, they would benefit from a community of peers wherein they could share their experiences and learn from others. This Birds-of-a-Feather (BoF) session is an organized space for CEd postdoctoral researchers, including those interested in postdoctoral positions, to build community and share postdoc experiences. Building on prior experience with running a similar BoF space for postdocs in SIGCSE 2022, the BoF discussion leaders is composed of current CEd postdocs, previous CEd postdocs who have taken on academic and research careers, and a CEd researcher from a non-academic space. This combination of diverse experiences, career contexts, and expertise will provide valuable perspectives that will aid in discussions of various postdoc experiences and research and career opportunities. BoF participants will have opportunities to discuss research goals, areas, and activities; exchange advice on navigating career paths and job searches; discuss approaches for mentorship within and outside research labs; and share best practices on the postdoc experience.

The introduction of Large Language Models (LLMs) has generated a significant amount of excitement both in industry and among researchers. Recently, tools that leverage LLMs have made their way into the classroom where they help students generate code and help instructors generate learning materials. There are likely many more uses of these tools -- both beneficial to learning and possibly detrimental to learning. To help ensure that these tools are used to enhance learning, educators need to not only be familiar with these tools, but with their use and potential misuse. The goal of this BoF is to raise awareness about LLMs and to build a learning community around their use in computing education. Aligned with this goal of building an inclusive learning community, our BoF is led by globally distributed discussion leaders, including undergraduate researchers, to facilitate multiple coordinated discussions that can lead to a broader conversation about the role of LLMs in CS education.

Recent breakthroughs in Large Language Models (LLMs), such as GPT-3 and Codex, now enable software developers to generate code based on a natural language prompt. Within computer science education, researchers are exploring the potential for LLMs to generate code explanations and programming assignments using carefully crafted prompts. These advances may enable students to interact with code in new ways while helping instructors scale their learning materials. However, LLMs also introduce new implications for academic integrity, curriculum design, and software engineering careers. This workshop will demonstrate the capabilities of LLMs to help attendees evaluate whether and how LLMs might be integrated into their pedagogy and research. We will also engage attendees in brainstorming to consider how LLMs will impact our field.

Advances in natural language processing have resulted in large language models (LLMs) that can generate code and code explanations. In this paper, we report on our experiences generating multiple code explanation types using LLMs and integrating them into an interactive e-book on web software development. Three different types of explanations -- a line-by-line explanation, a list of important concepts, and a high-level summary of the code -- were created. Students could view explanations by clicking a button next to code snippets, which showed the explanation and asked about its utility. Our results show that all explanation types were viewed by students and that the majority of students perceived the code explanations as helpful to them. However, student engagement varied by code snippet complexity, explanation type, and code snippet length. Drawing on our experiences, we discuss future directions for integrating explanations generated by LLMs into CS classrooms.

Student performance prediction aims to build models to help educators identify struggling students so they can be better supported. However, prior work in the space frequently evaluates features and models on data collected from a single semester, of a single course, taught at a single university. Without evaluating these methods in a broader context there is an open question of whether or not performance prediction methods are capable of generalising to new data. We test three methods for evaluating student performance models on data from introductory programming courses from two universities with a total of 3,323 students. Our results suggest that using cross-validation on one semester is insufficient for gauging model performance in the real world. Instead, we suggest that where possible future work in student performance prediction collects data from multiple semesters and uses one or more as a distinct hold-out set. Failing this, bootstrapped cross-validation should be used to improve confidence in models' performance. By recommending stronger methods for evaluating performance prediction models, we hope to bring them closer to practical use and assist teachers to understand struggling students in novice programming courses.

Full research paper—We describe the results of a pilot study wherein we applied simple natural language generation methods to produce automated feedback for students of an online course based on student high-level progress data. Experimenting with both personalized and non-personalized feedback, we show that such feedback can be easily produced given access to even rudimentary data regarding student assignment submissions and their correctness. Our results suggest that students perceive automatically generated feedback generally positively and believe it to be useful. Our results also indicate that minor personalization and stylistic alterations in the feedback can have meaningful effects on how the feedback is interacted with and perceived. In particular, we observe that personalized feedback is perceived as being slightly easier to understand and as being better aligned with their progress. Students also felt better about the personalized feedback in comparison to non-personalized feedback. We conclude that the automated generation of personalized textual feedback shows promise as a low-threshold way of increasing student satisfaction. Further research is needed to assess the effect of different types of automated personalized feedback on student performance and behavior.

This article explores the natural language generation capabilities of large language models with application to the production of two types of learning resources common in programming courses. Using OpenAI Codex as the large language model, we create programming exercises (including sample solutions and test cases) and code explanations, assessing these qualitatively and quantitatively. Our results suggest that the majority of the automatically generated content is both novel and sensible, and in some cases ready to use as is. When creating exercises we find that it is remarkably easy to influence both the programming concepts and the contextual themes they contain, simply by supplying keywords as input to the model. Our analysis suggests that there is significant value in massive generative machine learning models as a tool for instructors, although there remains a need for some oversight to ensure the quality of the generated content before it is delivered to students. We further discuss the implications of OpenAI Codex and similar tools for introductory programming education and highlight future research streams that have the potential to improve the quality of the educational experience for both teachers and students alike.

Programming is a complex task that requires the development of many skills including knowledge of syntax, problem decomposition, algorithm development, and debugging. Code-writing activities are commonly used to help students develop these skills, but the difficulty of writing code from a blank page can overwhelm many novices. Parsons problems offer a simpler alternative to writing code by providing scrambled code blocks that must be placed in the correct order to solve a problem. The extensive literature on Parsons problems documents numerous benefits to using them as both formative and summative assessments. These include more efficient learning, the possibility to dynamically adapt to learner needs, and more reliable grading. Despite these positive findings, further research is needed in order to draw broader inferences. Most work has been conducted at single institutions under unique conditions that are not easily replicated, and some prior studies have been inconclusive or had limitations that affected data validity. To address this, we propose a multi-institutional and multi-national study of the effectiveness of Parsons problems for novice programmers. We will focus on introductory programming courses (CS0/1/2) that use Java, Python, and C/C++ as these are the most common teaching languages. The working group will collaborate to refine the scope, methodology and research questions, and contribute to data collection and analysis.

Having students peer review each other's exercises is a common task in modern computing classrooms. In large classes, peer review might even partly replace traditional instructor-led review -- and prior work has found some indications that the quality of peer reviews can be close to that of instructor reviews. In this work, we explore the difference between instructor and peer reviews of student-created programming exercises. One task in an introductory programming course was to have students design their own programming exercises -- including an exercise description, model solution, and test cases -- which were then reviewed by peers. After the course, we had two instructors review the same student-created exercises. We compare the scores given by the instructors and the students to analyze potential differences. Our results suggest that agreement between instructors and students as measured by inter-rater reliability is low, although differences between instructor and student review score distributions are not statistically significant. Additionally, instructors have more fluctuation in their reviews compared to students. Due to the rising popularity of peer reviews, more research is needed to examine to what extent they could complement traditional instructor-led review of exercises.

Although computing education research quite often targets within-university courses, an important role of universities is educating the public through open online lifelong learning offerings. Compared to within-university courses, in lifelong learning, the student population is often more diverse. For example, participants often have more varied motivations and aspirations as well as more varied educational backgrounds. In this work, we explore what kinds of learners attend open online lifelong learning programming courses and what characteristics of learners lead to completing courses and proceeding to subsequent courses. We examine student-related factors collected through surveys in our online course environment. These factors include motivation, previous experience, and demographics. Our results show that motivations, previous experience, and demographics by themselves only explain a small amount of the variance in completing courses or continuing to a subsequent course. At the same time, we identify individual factors that are more likely to lead to learners dropping out (or continuing) in the courses. Our study provides further evidence that lifelong learning benefits most the already educated part of the population with prior knowledge and high motivation. This calls for further studies that seek to identify means to engage and support participants less likely to continue in such courses.

An indubitable way to put learning at scale in practice is to implement Massive Open Online Courses, or MOOCs. When a wide-enough portfolio of them is available, new applications arise. For instance, university admissions in Finland, where this study was conducted, have traditionally been based on students' grades in high school studies, an entrance examination, or a combination of both. A minority of students have been accepted through an open university admission path where students can get a study right if they complete enough university course credits with a high enough grade in a given time frame. In this work, we report results from a multi-university project in which the open university admission path has been expanded. All the universities in the Digital Education For All (DEFA) project remarkably expanded their portfolio of MOOCs that were offered both openly and for free, and the new admission path was simultaneously actively marketed and modified. In our analysis, we focus on examining whether the project increased computer science enrolments in the participating universities and how students accepted through the project perform in their studies compared to their peers accepted through other, traditional intake paths.

\emphTime-on-task is one key contributor to learning. However, how time-on-task is measured often varies, and is limited by the available data. In this work, we study two different time-on-task metrics---derived from programming process data---for predicting performance in an introductory programming course. The first metric, coarse-grained time-on-task, is based on students' submissions to programming assignments; the second, fine-grained time-on-task, is based on the keystrokes that students take while constructing their programs. Both types of time-on-task metrics have been used in prior work, and are supposedly designed to measure the same underlying feature: time-on-task. However, previous work has found that the correlation between these two metrics is not as high as one might expect. We build on that work by analyzing how well the two metrics work for predicting students' performance in an introductory programming course. Our results suggest that the correlation between the fine-grained time-on-task metric and both weekly exercise points and exam points is higher than the correlation between the coarse-grained time-on-task metric and weekly exercise points and exam points. Furthermore, we show that the fine-grained time-on-task metric is a better predictor of students' future success in the course exam than the coarse-grained time-on-task metric. We thus propose that future work utilizing time-on-task as a predictor of performance should use as fine-grained data as possible to measure time-on-task if such data is available.

How students are assessed has a powerful effect on their strategies for studying and their learning. When designing assessments, instructors should consider how different approaches for providing feedback to students could encourage positive learning behaviours. One such design is the use of interim deadlines that enable students to receive and respond to feedback. This is used to encourage students to start early and thus reduce the negative effects of procrastination. If multiple submissions are allowed, penalty schemes can be included to encourage students to reflect deeply on the feedback they receive, rather than developing an over-reliance on autograders. In this work we describe two approaches to feedback used over two consecutive semesters for a final project in a large introductory programming course. In both semesters, the complexity and structure of the final project was similar and students received identical instruction. In the first instance of the course students could submit their work prior to two scheduled interim deadlines, after which they would receive automated feedback, before meeting a final third deadline. In the second instance, students received automated feedback immediately upon submission but with increasing penalties to discourage excessive submissions. In both cases, the ability to receive automated feedback -- both scheduled and immediate -- was designed to encourage early participation with the project. Under the two feedback schemes, we observed different patterns of behaviour -- particularly for the lower performing students. We explore the benefits and drawbacks of the two schemes and consider implications for future project grading.

Teachers often preach for their students to start working on assignments early. There is even a fair amount of scientific evidence that starting early is beneficial for learning. In this work, we investigate students’ time management behavior in a second-year project-based software engineering course. In the course, students work on a software project in small groups of four to six students. We study time management from multiple angles. Firstly, we conduct an exploratory factor analysis and study how different time management related behavioral metrics are related to one another, for example, whether individual students’ time management practices in the second-year group project-based course are similar to their earlier time management practices in first-year courses where students work on assignments individually. Understanding how students’ previous time management behavior is manifested in later project-based courses would be beneficial when designing project-based education. Secondly, we study whether students’ time management practices affect the peer-review scores they get from their group members. Lastly, we explore how time management affects course performance. Our findings suggest that time management behavior, even from courses taken in the past, can be used to predict how students perform in future courses.

Intuitively, it seems plausible that students who start their work earlier and work on more days than their peers should perform better in any course. But does the early bird really catch the worm? In this article, we examine introductory programming students' time management behavior as evidenced by data collected from a programming environment. We analyze: 1) the earliness of students' work, i.e. when they start working on their course assignments, 2) the number of days students work on course assignments, and 3) the relationship between earliness, the number of days worked, and course outcomes. Our results provide further support for the notion that, on average, students who start working on course assignments early perform slightly better in the course. At the same time, we found that starting early does not necessarily mean that students work on more days, and that starting early and working on many days does not necessarily mean that students get better grades. In addition, some students who start working early on the assignments in the first weeks of the course seem to start delaying when they begin working on assignments as the course progresses, while other students seem to be able to continue starting early throughout the course.

The Digital Education For All project (DEFA) is a joint collaboration of five universities to open first-year computer science courses online and for free to audiences outside of universities. Additionally, students who complete enough courses through the project can apply for a study right at any of the participating universities.

Gamification has been used in introductory programming courses, for example, to increase engagement with study materials, reduce procrastination, and increase attendance to practice sessions. Indeed, with the rapidly growing adoption of digital tools in such courses, the use of various game elements and mechanics to drive participation is increasing. Previous studies on gamification in computing have examined the effects over the whole student population. Prior work in other disciplines has found that the benefits associated with gamification may only be realized for some students, while others may even experience reduced motivation. The Hexad user types survey attempts to tackle this problem by grouping users into six different types for whom gamification should have different effects. The goal is to personalize the game elements for different user types, thus creating gamified experiences more suitable for individual learners. In this work, we study whether the Hexad survey could be used to guide the personalization of gamification in an introductory programming course. Specifically, we examine the quality of students' answers to the Hexad survey and explore whether they can be used to predict students' preferences for enabling gamification in the platform where they complete assignments. In our specific computing education context, we find that classifying students using the Hexad survey does not appear to be an effective approach for the automatic personalization of gamification.

Analyzing keystroke data from students working on essay and programming tasks, we study to what extent the difference in task context influences performance in typing. Using data from two introductory programming courses offered at two separate institutions, we compare and contrast typing speed between programming and natural language tasks. We observe that students tend to be faster at typing (the same) character pairs when writing natural language text than when learning to write code. We show that students improve on typing character pairs that appear in frequently used words in programming languages, and that typing programming constructs also improves. We find that students are faster at detecting and erasing their mistakes when typing natural language text than when programming. Our results support theories regarding contextual memory, procedural memory, and practice, and have implications for course curriculum and pedagogy design.

Crowdsourcing refers to the act of using the crowd to create content or to collect feedback on some particular tasks or ideas. Within computer science education, crowdsourcing has been used -- for example -- to create rehearsal questions and programming assignments. As a part of their computer science education, students often learn relational databases as well as working with the databases using SQL statements. In this article, we describe a system for practicing SQL statements. The system uses teacher-provided topics and assignments, augmented with crowdsourced assignments and reviews. We study how students use the system, what sort of feedback students provide to the teacher-generated and crowdsourced assignments, and how practice affects the feedback. Our results suggest that students rate assignments highly, and there are only minor differences between assignments generated by students and assignments generated by the instructor.

When student programs are compared for similarity, certain segments of code are always sure to be similar. Some of these segments are boilerplate code -- public static void main String [] args and the like -- and some will be code that was provided to students as part of the assessment specification. The purpose of this working group is to explore what other code is expected to be reasonably common in student assessments, and should therefore be excluded from similarity checking. The answers will clearly vary with programming language, and perhaps with level of assessment item. Working group members will collect assessment submissions from their own or their colleagues' students, and it is hoped that these submissions will together encompass a wide variety of assessment tasks in a wide variety of programming languages. The working group aims to deliver clear guidelines as to what code can reasonably be excluded from automatic code similarity detection in various circumstances. It also aims to deliver a summary of what sort of code lecturers tend to provide for students when setting an assigned task, and why they provide that code.

We study programming process data from two introductory programming courses. Between the course contexts, the programming languages differ, the teaching approaches differ, and the spoken languages differ. In both courses, students' keystroke data -- timestamps and the pressed keys -- are recorded as students work on programming assignments. We study how the keystroke data differs between the contexts, and whether research on predicting course outcomes using keystroke latencies generalizes to other contexts. Our results show that there are differences between the contexts in terms of frequently used keys, which can be partially explained by the differences between the spoken languages and the programming languages. Further, our results suggest that programming process data that can be collected non-intrusive in-situ can be used for predicting course outcomes in multiple contexts. The predictive power, however, varies between contexts possibly because the frequently used keys differ between programming languages and spoken languages. Thus, context-specific fine-tuning of predictive models may be needed.

Since 2012, part of computer science student body at the University of Helsinki has been selected by using a massively open online version of the same introductory programming course that our freshmen take. In this multi-year study, we compare study success between students accepted through the online course (MOOC intake) and students accepted through the traditional entrance exam and high school matriculation exam based intake (normal intake). Our findings indicate that the MOOC intake perform better in computer science studies when looking at completed credits and grade point average, but there is no difference when considering other courses. Retention among the MOOC intake is better than among the normal intake. Additionally, students in the MOOC intake are more likely to complete their capstone project and Bachelor's thesis in the studied time-frame. However, the MOOC intake makes the already skewed gender balance more pronounced.

In this article, we report an experiment where students in an introductory programming course were given the opportunity to view model solutions to programming assignments whenever they wished, without the need to complete the assignments beforehand or to wait for the deadline to pass. Our experiment was motivated by the observation that some students may spend hours stuck with an assignment, leading to non-productive study time. At the same time, we considered the possibility of students using the sample solutions as worked examples, which could help students to improve the design of their own programs. Our experiment suggests that many of the students use the model solutions sensibly, indicating that they can control their own work. At the same time, a minority of students used the model solutions as a way to proceed in the course, leading to poor exam performance.

Vast numbers of publications in computing education begin with the premise that programming is hard to learn and hard to teach. Many papers note that failure rates in computing courses, and particularly in introductory programming courses, are higher than their institutions would like. Two highly distinct research projects have established that average success rates in introductory programming courses world-wide are in the region of 67%. However, there is little published work comparing pass rates in computing courses with those in other STEM disciplines. As institutions continually ask computing educators to justify the atypical failure rates in their courses, a thoroughly researched comparison of this sort could prove useful in demonstrating whether the phenomenon is real, and, if so, whether it extends somewhat beyond the boundaries of individual institutions. This working group will gather information on pass rates in computing courses, particularly introductory programming courses, and in courses at comparable levels in other STEM disciplines. Members of the group will be required to gather the information from their own institutions, and further data will be gathered by way of a broad survey. The data will be analysed to see whether global patterns can be established, and the group will survey the literature to gather and summarise postulated explanations for any difference between pass rates in computing and in other STEM disciplines.

When learning programming, students learn the syntax of a programming language, the semantics underlying the syntax, and practice applying the language in solving programming problems. Research has suggested that simply the syntax may be hard to learn. In this article, we study difficulty of learning the syntax of a programming language. We have constructed a tool that provides students code that they write character-by-character. When writing, the tool automatically highlights each character in code that is incorrectly typed, and through the highlight-based feedback directs students into writing correct syntax. We conducted a randomized controlled trial in an introductory programming course organized in Java. One half of the population had the tool in the course material immediately before programming exercises where the practiced syntax was used, while the other half of the course population did not have the tool, thus approaching the exercises in a traditional way. Our results imply that isolated syntax writing practice may not be a meaningful addition to the arsenal used for teaching programming, at least when the programming course utilizes a large set of small programming exercises. We encourage researchers to replicate our work in contexts where syntax seems to be an issue.

Since computing education began, we have sought to learn why students struggle in computer science and how to identify these at-risk students as early as possible. Due to the increasing availability of instrumented coding tools in introductory CS courses, the amount of direct observational data of student working patterns has increased significantly in the past decade, leading to a flurry of attempts to identify at-risk students using data mining techniques on code artifacts. The goal of this work is to produce a systematic literature review to describe the breadth of work being done on the identification of at-risk students in computing courses. In addition to the review itself, which will summarize key areas of work being completed in the field, we will present a taxonomy (based on data sources, methods, and contexts) to classify work in the area.

Small automatically assessed programming assignments are an often used resource for learning programming. Creating sufficiently large amounts of such assignments is, however, time consuming. As a consequence, offering large quantities of practice assignments to students is not always possible. CrowdSorcerer is an embeddable open-source system that students and teachers alike can use for creating and evaluating small automatically assessed programming assignments. While creating programming assignments, the students also write simple input-output -tests, and are gently introduced to the basics of testing. Students can also evaluate the assignments of others and provide feedback on them, which exposes them to code written by others early in their education. In this article we both describe the CrowdSorcerer system and our experiences in using the system in a large undergraduate programming course. Moreover, we discuss the motivation for crowdsourcing course assignments and present some usage statistics.

We explore educational pair programming in a university context with high student autonomy and individual responsibility. The data comes from two separate introductory programming courses with optional pair programming assignments. We analyze lab attendance and course outcomes to determine whether students' previous programming experience or gender influence attendance. We further compare these statistics to self-reported data on study motivation, study strategies, and student enjoyment of pair programming. The influence of grading systems on pair programming behavior and course outcomes is also examined. Our results suggest that gender and previous programming experience correlate with participation in pair programming labs. At the same time, there are no significant differences in self-reported enjoyment of pair programming between any of the groups, and the results from commonly used study motivation and strategy questionnaires provide little insight into students/ actual behavior.

In this article, we study how visualizations could be used to support students' self-regulation in online learning. We conducted a randomized controlled trial with three groups: one control group without visualization, one treatment group with textual visualization, and one treatment with graphical visualization with information on peers' average achievement. We studied how different visualizations affect students' academic performance and behavior. We focused on four factors; starting, scheduling, earliness and exercise points, where the first three are related to time management and self-regulation. The last factor measures course performance in terms of completed exercises. Our results suggest that the lowest performing students can benefit from a visualization, whereas the highest performing students are not affected by the presence or absence of a visualization. We also found that visualizations that do not provide the means to compare your own performance with others may even be harmful to performance oriented students.

I am studying the use of fine-grained programming process data for student modeling. The initial plan is to construct different types of program state representations such as Abstract Syntax Trees (ASTs) from the data. These program state representations could be used for both automatically inferring knowledge components that the students are trying to learn as well as for modeling students' knowledge on those specific components.

Research on the indicators of student performance in introductory programming courses has traditionally focused on individual metrics and specific behaviors. These metrics include the amount of time and the quantity of steps such as code compilations, the number of completed assignments, and metrics that one cannot acquire from a programming environment. However, the differences in the predictive powers of different metrics and the cross-metric correlations are unclear, and thus there is no generally preferred metric of choice for examining time on task or effort in programming. In this work, we contribute to the stream of research on student time on task indicators through the analysis of a multi-source dataset that contains information about students' use of a programming environment, their use of the learning material as well as self-reported data on the amount of time that the students invested in the course and per-assignment perceptions on workload, educational value and difficulty. We compare and contrast metrics from the dataset with course performance. Our results indicate that traditionally used metrics from the same data source tend to form clusters that are highly correlated with each other, but correlate poorly with metrics from other data sources. Thus, researchers should utilize multiple data sources to gain a more accurate picture of students' learning.

Due to the increased enrollments in Computer Science education programs, institutions have sought ways to automate and streamline parts of course assessment in order to be able to invest more time in guiding students' work.

Large-scale courses such as Massive Online Open Courses (MOOCs) can be a great data source for researchers. Ideally, the data gathered on such courses should be openly available to all researchers. Studies could be easily replicated and novel studies on existing data could be conducted. However, very fine-grained data such as source code snapshots can contain hidden identifiers. For example, distinct typing patterns that identify individuals can be extracted from such data. Hence, simply removing explicit identifiers such as names and student numbers is not sufficient to protect the privacy of the users who have supplied the data. At the same time, removing all keystroke information would decrease the value of the shared data significantly.

In traditional programming courses, students have usually been at least partly graded using pen and paper exams. One of the problems related to such exams is that they only partially connect to the practice conducted within such courses. Testing students in a more practical environment has been constrained due to the limited resources that are needed, for example, for authentication.

Studies on retention and success in introductory programming course have suggested that previous programming experience contributes to students' course outcomes. If such background information could be automatically distilled from students' working process, additional guidance and support mechanisms could be provided even to those, who do not wish to disclose such information. In this study, we explore methods for automatically distinguishing novice programmers from more experienced programmers using fine-grained source code snapshot data. We approach the issue by partially replicating a previous study that used students' keystroke latencies as a proxy to introductory programming course outcomes, and follow this by an exploration of machine learning methods to separate those students with little to no previous programming experience from those with more experience. Our results confirm that students' keystroke latencies can be used as a metric for measuring course outcomes. At the same time, our results show that students programming experience can be identified to some extent from keystroke latency data, which means that such data has potential as a source of information for customizing the students' learning experience.