As machine learning (ML) becomes integral in more disciplines, introductory courses in the field are attracting increasingly diverse audiences. Design of these introductory ML courses needs to be theoretically sound, but also intuitive, engaging, and accessible to a range of students. Effective teaching of ML must go beyond teaching the theoretical or practical mechanics of algorithms. In this paper, we synthesize effective teaching strategies from 6 experienced ML instructors across 5 institutions to help students define appropriate ML problems, build intuition, develop reasoning skills, and apply models responsibly. We organize these strategies into eight thematic areas: preparing students for success, motivating learners through real-world relevance, integrating ethics and societal impact, avoiding common methodological pitfalls in model evaluation, guiding students on design decisions, adapting effective classroom practices, assessing student learning, and preparing for the future. Each section offers practical examples of classroom-tested activities (or references to existing resources), and in many cases, reflections on our experiences with the strategies. Our aim is for this paper to be a starting point for instructors aiming to improve learning in introductory ML courses. We hope this is a resource-rich guide for teaching ML to diverse learners, grounded in both pedagogy and practice.

Background. When AI tools can easily complete programming assignments, students face a motivational question: why invest effort in completing them independently? While prior work has examined instructor policies and usage patterns, we focus on how students themselves experience and respond to AI availability, a perspective important for designing courses that sustain engagement with programming practice.

This pilot study investigated students' perceptions of visual progress bar and on-demand hints in a CS1 course. Students valued both feedback systems, with more confident students preferring on-demand hints for direct problem-solving support. Progress bars were consistently perceived as beneficial across varying confidence levels.

Adaptive Experimentation is one of the most promising approaches to support complex decision-making in learning experience design and delivery. This paper reports on our experience with a real-world, multi-experimental evaluation of an adaptive experimentation platform within the XPRIZE Digital Learning Challenge framework, and summarizes data-driven lessons learned and best practices for Adaptive Experimentation in education. We outline key scenarios of the applicability of platform-supported experiments and reflect on lessons learned from this two-year project, focusing on implications relevant to platform developers, researchers, practitioners, and policy stakeholders to integrate Adaptive Experiments in real-world courses.

The rapid evolution of artificial intelligence (AI), specifically large language models (LLMs), has opened opportunities for various educational applications. This paper explored the feasibility of utilizing ChatGPT, one of the most popular LLMs, for automating feedback for Java programming assignments in an introductory computer science (CS1) class. Specifically, this study focused on three questions: 1) To what extent do students view LLM-generated feedback as formative? 2) How do students see the comparative affordances of feedback prompts that include their code, vs. those that exclude it? 3) What enhancements do students suggest for improving LLM-generated feedback? To address these questions, we generated automated feedback using the ChatGPT API for four lab assignments in a CS1 class. The survey results revealed that students perceived the feedback as aligning well with formative feedback guidelines established by Shute. Additionally, students showed a clear preference for feedback generated by including the students' code as part of the LLM prompt, and our thematic study indicated that the preference was mainly attributed to the specificity, clarity, and corrective nature of the feedback. Moreover, this study found that students generally expected specific and corrective feedback with sufficient code examples, but had diverged opinions on the tone of the feedback. This study demonstrated that ChatGPT could generate Java programming assignment feedback that students perceived as formative. It also offered insights into the specific improvements that would make the ChatGPT-generated feedback useful for students.

Students often run into trouble when learning and practicing software testing. Recent prior studies demonstrate that a lightweight testing checklist that contains testing strategies and tutorial information could assist students in writing higher-quality tests. Prior studies also suggest that students with lower prior knowledge in unit testing may benefit more from the checklists. However, insights on the potential benefits and costs of the testing checklists in a classroom setting are lacking. To address this, we conducted an operational replication study in a CS2 course with 342 students (171 from Fall 2023 and 171 from Spring 2024) who had no prior experience in unit testing.

External help resources are frequently used by novice programmers solving classwork in undergraduate computing courses. Traditionally, these tools consisted of web resources such as tutorial websites and Q&A forums. With the rise of Generative AI (GenAI), there has been increasing concern and research about how external resources should be used in the classroom. However, little work has directly contrasted student beliefs and perceptions of web resources with GenAI, has grounded these beliefs in prior psychological theory, and has investigated how demographic factors and student backgrounds influence these beliefs and intentions. We administered a vignette-style survey across two courses required for a CS major at an R1 University, a freshman (n = 152) and senior capstone course (n = 44). Students responded to likert questions aiming to measure behavioral factors related to these tools, such as intention to use, perceived attitudes, peer perceptions, and their own perceived tool competency. We primarily investigate the results of an introductory course, finding that novices have a wide range of opinions on both resources, but overall find them slightly useful and have a tendency to prefer web-search. We compare this with seniors, who have more positive perceptions of these tools, and discuss possible reasons and implications for this difference. We constructed two path models to investigate which factors strongly influence novices’ intention to use resources and find the primary factor to be their general attitudes in how these tools will result in a positive or negative outcome (e.g. perceived benefits, justifiability). We also measure the effects of student background on intention to use these resources. Finally, we discuss implications and suggestions on how instructors can use this information to approach, address, and influence resource usage in their classrooms.

Background and Context. Research software in the Computing Education Research (CER) domain frequently encounters issues with scalability and sustained adoption, which limits its educational impact. Despite the development of numerous CER programming (CER-P) tools designed to enhance learning and instruction, many fail to see widespread use or remain relevant over time. Previous research has primarily examined the challenges educators face in adopting and reusing CER tools, with few focusing on understanding the barriers to scaling and adoption practices from the tool developers’ perspective.

The emergence of publicly accessible large language models (LLMs) such as ChatGPT poses unprecedented risks of new types of plagiarism and cheating where students use LLMs to solve exercises for them. Detecting this behavior will be a necessary component in introductory computer science (CS1) courses, and educators should be well-equipped with detection tools when the need arises. However, ChatGPT generates code non-deterministically, and thus, traditional similarity detectors might not suffice to detect AI-created code. In this work, we explore the affordances of Machine Learning (ML) models for the detection task. We used an openly available dataset of student programs for CS1 assignments and had ChatGPT generate code for the same assignments, and then evaluated the performance of both traditional machine learning models and Abstract Syntax Tree-based (AST-based) deep learning models in detecting ChatGPT code from student code submissions. Our results suggest that both traditional machine learning models and AST-based deep learning models are effective in identifying ChatGPT-generated code with accuracy above 90%. Since the deployment of such models requires ML knowledge and resources that are not always accessible to instructors, we also explore the patterns detected by deep learning models that indicate possible ChatGPT code signatures, which instructors could possibly use to detect LLM-based cheating manually. We also explore whether explicitly asking ChatGPT to impersonate a novice programmer affects the code produced. We further discuss the potential applications of our proposed models for enhancing introductory computer science instruction.

Affective state, referring to an individual's feeling, can impact students' confidence and retention in CS, particularly for novice programmers. However, little research has been conducted to examine how moments that occur during programming impact students' affective states in real-time. In this pilot study, seven undergraduate students in an introductory block-based programming course completed a programming assignment and were surveyed and interviewed about their experience and self-efficacy as programmers. While programming, students periodically recorded their affective states via a popup in the programming environment. We performed retrospective think-aloud interviews with students afterward, asking them to watch and reflect on recordings of their programming. We subsequently analyzed student interviews using thematic analysis to derive 206 codes. These codes were grouped into three areas that impacted affect: the environment, objective progress, and perceptions during programming. To explore why students responded as they did to moment occurrence, we further categorized students based on four dimensions: programming experience, assignment completion, confidence, and the impact of the programming session on self-efficacy. Our initial results suggest that while certain moments elicit similar affective states among students, the interaction of the aforementioned four dimensions may have a higher impact on novices' affective states during programming. We conclude with recommendations for educators to improve students' affective states during and after programming.

Many SIGCSE attendees are either developing or using online educational tools, and all will benefit from better interoperability among these tools and better analysis of the clickstream data coming from those tools. New tools for analyzing big data leveraged by AI (e.g., deep learning for assessment) in turn improve both content and pedagogy, thus setting up a virtuous cycle fueling learning discoveries and leveraging innovation in AI: Online technologies → big data analysis → better online technologies. This NSF-supported workshop is the latest in a series of SPLICE workshops, and is a continuation of our event at SIGCSE 2023, where the SPLICE-Portal, a dedicated socio-technical research infrastructure for Computing Education Research, was presented. This year, we continue the work with several new SPLICE community working groups, including those on Dashboards, Large Language Models, Parsons Problems, and Smart Learning Content Protocols. We continue to build upon our existing collaborations developed over the course of the project to engage more members of the community in tasks that will advance the project agenda.

External help resources are frequently used by novice programmers solving classwork in undergraduate computing courses. Traditionally, these tools consisted of web-based resources such as tutorial websites and Q&A forums. With the rise of AI code-generation and explanation tools, understanding how students use external resources and their roles in classroom have become especially relevant. Despite this, little research has directly investigated the extent to which students intent to use these tools and what factors influence their beliefs. It is unknown when students think it is appropriate to use these tools and what features they find valuable. Understanding these beliefs would allow instructors and researchers to better focus their efforts on what aspects of pedagogy and tool usage should be addressed. We administered a pilot vignette-style survey to introductory programming classes at an R1 University (n=45), giving students scenarios of external resource usage while questioning their attitudes, subjective norms, and their perceived behavioral control on using these external resources. We share preliminary findings on free response data, showcasing the variety of beliefs and opinions that novice programming students have on when and how much external resource usage is acceptable in the classroom. Some students felt that AI tools can provide more exact solutions than searching for help online, but also expressed that this exactness could be detrimental to their learning. Others expressed awareness that professionals use these resources, and expressed a desire to learn how to use them in a way to help their educational and career goals.

In computing classrooms, building an open-ended programming project engages students in the process of designing and implementing an idea of their own choice. An explicit planning process has been shown to help students build more complex and ambitious open-ended projects. However, novices encounter difficulties in exploring and creatively expressing ideas during planning. We present Idea Builder, a storyboarding-based planning system to help novices visually express their ideas. Idea Builder includes three features: 1) storyboards to help students express a variety of ideas that map easily to programming code, 2) animated example mechanics with example actors to help students explore the space of possible ideas supported by the programming environments, and 3) synthesized starter code to help students easily transition from planning to programming. Through two studies with high school coding workshops, we found that students self-reported as feeling creative and feeling easy to communicate ideas; having access to animated example mechanics of an actor help students to build those actors in their plans and projects; and that most students perceived the synthesized starter code from Idea Builder as helpful and time-saving.

Many students struggle with decomposition and planning despite the necessity of these skills in computing education. Hence, more tools are needed to scaffold these processes. In this paper, we present Jigsaw, a standalone visual planning tool to help students practice decomposition and planning before writing code. Jigsaw allows students to compose a solution to a new problem based on previously seen “patterns,” such as the accumulator pattern for summing values or the filter pattern for conditional input selection. Students can connect these patterns together to see how data flows between them and define a solution plan. Jigsaw’s goal is to scaffold students’ planning processes by presenting relevant patterns for a given problem. Using a within-subjects design, we evaluated Jigsaw by observing 17 undergraduate students as they planned for and implemented two programming assignments. The experimental task included Jigsaw, and the control task did not. This design aimed to understand how the tool impacted students’ planning and programming process. Subsequently, we conducted interviews with these students regarding their planning and programming experiences with and without Jigsaw. Many students explicitly mentioned they would employ Jigsaw for planning and appreciated the scaffolding it provided. Students also admired the Jigsaw’s novelty in visualizing programming problems. We conclude with our design takeaways and recommendations for future work.

Background and Context: Open-ended programming projects encourage novice students to choose and pursue projects based on their own ideas and interests, and are widely used in many introductory programming courses. However, novice programmers encounter challenges exploring and discovering new ideas, implementing their ideas, and applying unfamiliar programming concepts and APIs. Code examples are one of the primary resources students use to apply code usage patterns and learn API knowledge, but little work has investigated the effect of having access to examples on students’ open-ended programming experience.

Software testing is a critical skill for computing students, but learning and practicing testing can be challenging, particularly for beginners. A recent study suggests that a lightweight testing checklist that contains testing strategies and tutorial information could assist students in writing quality tests. However, students expressed a desire for more support in knowing how to test the code/scenario. Moreover, the potential costs and benefits of the testing checklist are not yet examined in a classroom setting. To that end, we improved the checklist by integrating explicit testing strategies to it (ETS Checklist), which provide step-by-step guidance on how to transfer semantic information from instructions to the possible testing scenarios. In this paper, we report our experiences in designing explicit strategies in unit testing, as well as adapting the ETS Checklist as optional tool support in a CS1.5 course. With the quantitative and qualitative analysis of the survey responses and lab assignment submissions generated by students, we discuss students' engagement with the ETS Checklists. Our results suggest that students who used the checklist intervention had significantly higher quality in their student-authored test code, in terms of code coverage, compared to those who did not, especially for assignments earlier in the course. We also observed students' unawareness of their need for help in writing high-quality tests.

Many students rely on examples when learning to program, but they often face barriers when incorporating these examples into their own code and learning the concepts they present. As a step towards designing effective example interfaces that can support student learning, we investigate novices' needs and strategies when using examples to write code. We conducted a study with 12 pairs of high school students working on open-ended game design projects, using a system that allows students to browse examples based on their functionality, and to view and copy the example code. We analyzed interviews, screen recordings, and log data, identifying 5 moments when novices request examples, and 4 strategies that arise when students use examples. We synthesize these findings into principles that can inform the design of future example systems to better support students.

A key goal of many computer science education efforts is to increase the number and diversity of students who persist in the field of computer science and into computing careers. Many interventions have been developed in computer science designed to increase students' persistence in computing. However, it is often difficult to measure the efficacy of such interventions, as measuring actual persistence by tracking student enrollments and career placements after an intervention is difficult and time-consuming, and sometimes even impossible. In the social sciences, attitudinal research is often used to solve this problem, as attitudes can be collected in survey form around the same time that interventions are introduced and are predictive of behavior. This can allow researchers to assess the potential efficacy of an intervention before devoting the time and energy to conduct a longitudinal analysis. In this paper, we develop and validate a scale to measure intentions to persist in computing, and demonstrate its use in predicting actual persistence as defined by enrolling in another computer science course within two semesters. We conduct two analyses to do this: First, we develop a computing persistence index and test whether our scale has high alpha reliability and whether our scale predicts actual persistence in computing using students' course enrollments. Second, we conduct analyses to reduce the number of items in the scale, to make the scale easy for others to include in their own research. This paper contributes to research on computing education by developing and validating a novel measure of intentions to persist in computing, which can be used by computer science educators to evaluate potential interventions. This paper also creates a short version of the index, to ease implementation.

Web-based help-seeking -- finding and utilizing websites to solve a problem -- is a critical skill during programming in both professional and academic settings. However, little work has explored how students, especially novices, engage in web-based help-seeking during programming, or what strategies they use and barriers they face. This study begins to investigate these questions through analysis of students' web-search behaviors during programming. We collected think-aloud, screen recording, and log data as students completed a challenging programming task. Students were encouraged to use the web for help when needed, as if in an internship. We then qualitatively analyzed the data to address three research questions: 1) What events motivate students to use web search? 2) What strategies do students employ to search for, select, and learn from web pages? 3) What barriers do students face in web search, and when do they arise? Our results suggest that that novices use a variety of web-search strategies -- some quite unexpected -- with varying degrees of success, suggesting that web search can be a challenging skill for novice programmers. We discuss how these results inform future research and pedagogy focused on how to support students in effective web search.

Many students struggle when they are first learning to program. Without help, these students can lose confidence and negatively assess their programming ability, which can ultimately lead to dropouts. However, detecting the exact moment of student struggle is still an open question in computing education. In this work, we conducted a think-aloud study with five high-school students to investigate the automatic detection of progressing and struggling moments using a detector algorithm (SPD). SPD classifies student trace logs into moments of struggle and progress based on their similarity to prior students' correct solutions. We explored the extent to which the SPD-identified moments of struggle aligned with expert-identified moments based on novices' verbalized thoughts and programming actions. Our analysis results suggest that SPD can catch students' struggling and progressing moments with a 72.5% F1-score, but room remains for improvement in detecting struggle. Moreover, we conducted an in-depth examination to discover why discrepancies arose between expert-identified and detector-identified struggle moments. We conclude with recommendations for future data-driven struggle detection systems.

Student modeling sits at the epicenter of adaptive learning technology. In contrast to the voluminous work on student modeling for well-defined domains such as algebra, there has been little research on student modeling in programming (SMP) due to data scarcity caused by the unbounded solution spaces of open-ended programming exercises. In this work, we focus on two essential SMP tasks: program classification and early prediction of student success and propose a Cross-Lingual Adversarial Domain Adaptation (CrossLing) framework that can leverage a large programming dataset to learn features that can improve SMP's build using a much smaller dataset in a different programming language. Our framework maintains one globally invariant latent representation across both datasets via an adversarial learning process, as well as allocating domain-specific models for each dataset to extract local latent representations that cannot and should not be united. By separating globally-shared representations from domain-specific representations, our framework outperforms existing state-of-the-art methods for both SMP tasks.

Are women less likely to persist in computer science because of gender differences in self-assessed computing ability? And why do gender differences exist in self-assessments among women and men who earn the same grades? We use a mixed-method research design to answer these questions, utilizing both quantitative survey data (n = 764) and qualitative interview data (n = 59) from students in introductory computing courses at a large U.S. state university. Quantitatively, we find that women self-assess their computing ability significantly lower than men who earn the same grades, and that these lower self-assessments reduce the likelihood that women enroll in future CS courses (relative to men who earn equivalent grades). Qualitatively, we explore how women and men perceive their own computing ability to understand why women self-assess their ability lower than men. Our interviews revealed that women were much less likely than men to make favorable comparative judgements about their ability relative to their classmates. Women also had higher personal performance standards than men. Lastly, women were more likely than men to experience disrespectful treatment, with an undertone of presumed incompetence, from their TAs and classmates. In sum, this research furthers our understanding of why gender differences exist in self-assessments of computing ability and how these differences can contribute to gender disparities in computing persistence. It also draws attention to the importance of feedback in computing courses and suggests that improving course feedback may reduce gender disparities in computing.

Research has shown that high self-assessment of ability, sense of belonging, and professional role confidence are crucial for students' persistence in computing. As grades in introductory computer science courses tend to be lower than other courses, it is essential to provide students with contextualized feedback about their performance in these courses. Giving students unambiguous and con- textualized feedback is especially important during COVID when many classes have moved online and instructors and students have fewer opportunities to interact. In this study, we investigate the effect of a lightweight, scalable intervention where students received personalized, contextualized feedback from their instructors after two major assignments during the semester. After each intervention, we collected survey data to assess students' self-assessment of computing ability, sense of belonging, intentions to persist in computing, professional role confidence, and the likelihood of stating intention to pursue a major in computer science. To analyze the effectiveness of our intervention, we conducted linear regression and mediation analysis on student survey responses. Our results have shown that providing students with personalized feedback can significantly improve their self-assessment of computing ability, which will significantly improve their intentions to persist in computing. Furthermore, our results have demonstrated that our intervention can significantly improve students' sense of belonging, professional role confidence, and the likelihood of stating an intention to pursue a major in computer science.

Software testing is an essential skill for computer science students. Prior work reports that students desire support in determining what code to test and which scenarios should be tested. In response to this, we present a lightweight testing checklist that contains both tutorial information and testing strategies to guide students in what and how to test. To assess the impact of the testing checklist, we conducted an experimental, controlled A/B study with 32 undergraduate and graduate students. The study task was writing a test suite for an existing program. Students were given either the testing checklist (the experimental group) or a tutorial on a standard coverage tool with which they were already familiar (the control group). By analyzing the combination of student-written tests and survey responses, we found students with the checklist performed as well as or better than the coverage tool group, suggesting a potential positive impact of the checklist (or at minimum, a non-negative impact). This is particularly noteworthy given the control condition of the coverage tool is the state of the practice. These findings suggest that the testing tool support does not need to be sophisticated to be effective.

Our researchers seek to support students in building block-based programming projects that are motivating and engaging as well as valuable practice in learning to code. A difficult part of the programming process is planning. In this research, we explore how novice programmers used a custom-built planning tool, PlanIT, contrasted against how they used storyboarding when planning games. In a three-part study, we engaged novices in planning and programming three games: a maze game, a break-out game, and a mashup of the two. In a set of five case studies, we show how five pairs of students approached the planning and programming of these three games, illustrating that students felt more creative when storyboarding rather than using PlanIT. We end with a discussion on the implications of this work for designing supports for novices to plan open-ended projects.

We have found that giving top-performing students in CS1 courses personalized feedback increases their intentions to persist in computing, especially among students who are women. This personalized feedback also appears to improve students' course experience and increases the likelihood that women apply to be CS1 TAs. Yet despite these benefits, giving personalized feedback may seem too impractical and time-intensive for faculty members to adopt in their own classrooms. In this workshop, we will reduce the burden of giving students personalized feedback by: 1) giving instructors empirically validated email templates to use in their own courses, and 2) guiding faculty how to send emails at-scale. We will also discuss how self-assessments influence students' career choices, how gender stereotypes bias self-assessments, and what faculty can do to counteract biased self-assessments of computing ability.

Machine learning (ML) is one of the fastest growing subfields in Computer Science, and it is important to identify ways to improve ML education. A key way to do so is by understanding the common errors that students make when writing ML programs, so they can be addressed. Prior work investigating ML errors has focused on an instructor perspective, but has not looked at student programming artifacts, such as projects and code submissions to understand how these errors occur and which are most common. To address this, we qualitatively coded over 2,500 cells of code from 19 final team projects (63 students) in an upper-division machine learning course. By isolating and codifying common errors and misconceptions across projects, we can identify what ML errors students struggle with. In our results, we found that library usage, hyperparameter tuning, and misusing test data were among the most common errors, and we give examples of how and when they occur. We then provide suggestions on why these misconceptions may occur, and how instructors and software designers can possibly mitigate these errors.

Open-ended programming engages students by connecting computing with their real-world experience and personal interest. However, such open-ended programming tasks can be challenging, as they require students to implement features that they may be unfamiliar with. Code examples help students to generate ideas and implement program features, but students also encounter many learning barriers when using them. We explore how to design code examples to support novices' effective example use by presenting our experience of building and deploying Example Helper, a system that supports students with a gallery of code examples during open-ended programming. We deployed Example Helper in an undergraduate CS0 classroom to investigate students' example usage experience, finding that students used different strategies to browse, understand, experiment with, and integrate code examples, and that students who make more sophisticated plans also used more examples in their projects.

Block-based programming environments, such as Scratch and Snap!, engage users to create programming artifacts such as games and stories, and share them in an online community. Many Snap! users start programming by reusing and modifying an example project, but encounter many barriers when searching and identifying the relevant parts of the program to learn and reuse. We present Pinpoint, a system that helps Snap! programmers understand and reuse an existing program by isolating the code responsible for specific events during program execution. Specifically, a user can record an execution of the program (including user inputs and graphical output), replay the output, and select a specific time interval where the event of interest occurred, to view code that is relevant to this event. We conducted a small-scale user study to compare users’ program comprehension experience with and without Pinpoint, and found suggestive evidence that Pinpoint helps users understand and reuse a complex program more efficiently.

Open-ended programming increases students' motivation by allowing them to solve authentic problems and connect programming to their own interests. However, such open-ended projects are also challenging, as they often encourage students to explore new programming features and attempt tasks that they have not learned before. Code examples are effective learning materials for students and are well-suited to supporting open-ended programming. However, there is little work to understand how novices learn with examples during open-ended programming, and few real-world deployments of such tools. In this paper, we explore novices' learning barriers when interacting with code examples during open-ended programming. We deployed Example Helper, a tool that offers galleries of code examples to search and use, with 44 novice students in an introductory programming classroom, working on an open-ended project in Snap. We found three high-level barriers that novices encountered when using examples: decision, search, and integration barriers. We discuss how these barriers arise and design opportunities to address them.

Programming environments such as Snap, Scratch, and Processing engage learners by allowing them to create programming artifacts such as apps and games, with visual and interactive output. Learning programming with such a media-focused context has been shown to increase retention and success rate. However, assessing these visual, interactive projects requires time and laborious manual effort, and it is therefore difficult to offer automated or real-time feedback to students as they work. In this paper, we introduce SnapCheck, a dynamic testing framework for Snap that enables instructors to author test cases with Condition-Action templates. The goal of SnapCheck is to allow instructors or researchers to author property-based test cases that can automatically assess students' interactive programs with high accuracy. Our evaluation of SnapCheck on 162 code snapshots from a Pong game assignment in an introductory programming course shows that our automated testing framework achieves at least 98% accuracy over all rubric items, showing potentials to use SnapCheck for auto-grading and providing formative feedback to students.

Understanding students’ misconceptions is important for effective teaching and assessment. However, discovering such misconceptions manually can be time-consuming and laborious. Automated misconception discovery can address these challenges by highlighting patterns in student data, which domain experts can then inspect to identify misconceptions. In this work, we present a novel method for the semi-automated discovery of problem-specific misconceptions from students’ program code in computing courses, using a state-of-the-art code classification model. We trained the model on a block-based programming dataset and used the learned embedding to cluster incorrect student submissions. We found these clusters correspond to specific misconceptions about the problem and would not have been easily discovered with existing approaches. We also discuss potential applications of our approach and how these misconceptions inform domain-specific insights into students’ learning processes.

Students often struggle during programming homework and may need help getting started or localizing errors. One promising and scalable solution is to provide automated programming hints, generated from prior student data, which suggest how a student can edit their code to get closer to a solution, but little work has explored how to design these hints for large-scale, real-world classroom settings, or evaluated such designs. In this paper, we present CodeChecker, a system which generates hints automatically using student data, and incorporates them into an existing CS1 online homework environment, used by over 1000 students per semester. We present insights from survey and interview data, about student and instructor perceptions of the system. Our results highlight affordances and limitations of automated hints, and suggest how specific design choices may have impacted their effectiveness.

Instructors have limited time and resources to help struggling students, and these resources should be directed to the students who most need them. To address this, researchers have constructed models that can predict students' final course performance early in a semester. However, many predictive models are limited to static and generic student features (e.g. demographics, GPA), rather than computing-specific evidence that assesses a student's progress in class. Many programming environments now capture complete time-stamped records of students' actions during programming. In this work, we leverage this rich, fine-grained log data to build a model to predict student course outcomes. From the log data, we extract patterns of behaviors that are predictive of students' success using an approach called differential sequence mining. We evaluate our approach on a dataset from 106 students in a block-based, introductory programming course. The patterns extracted from our approach can predict final programming performance with 79% accuracy using only the first programming assignment, outperforming two baseline methods. In addition, we show that the patterns are interpretable and correspond to concrete, effective -- and ineffective -- novice programming behaviors. We also discuss these patterns and their implications for classroom instruction.

Project-based learning can encourage and motivate students to learn through exploring their own interests, but introduces special challenges for novice programmers. Recent research has shown that novice students perceive themselves to be "bad at programming, especially when they do not know how to start writing a program, or need to create a plan before getting started. In this paper, we present PlanIT, a guided planning tool integrated with the Snap! programming environment designed to help novices plan and program their open-ended projects. Within PlanIT, students can add a description for their project, use a to do list to help break down the steps of implementation, plan important elements of their program including actors, variables, and events, and view related example projects. We report findings from a pilot study of high school students using PlanIT, showing that students who used the tool learned to make more specific and actionable plans. Results from student interviews show they appreciate the guidance that PlanIT provides, as well as the affordances it offers to more quickly create program elements.

One approach to teaching game design to students with a wide variety of disciplinary backgrounds is through team game projects that span multiple weeks, up to an entire term. However, open-ended, creative projects introduce a gamut of challenges to novice programmers. Our goal is to assist game design students with the planning stage of their projects. This paper describes our data collection process through three course interventions and student interviews, and subsequent analysis in which we learned students had difficulty expressing their creative vision and connecting the game mechanics to the intended player experience. We present these results as a step towards the goal of scaffolding the planning process for student game projects, supporting more creative ideas, clearer communication among team members, and a stronger understanding of human-centered design in software development.

Students working on programming homework do not receive the same level of support as in the classroom, relying primarily on automated feedback from test cases. One low-effort way to provide more support is by prompting students to compare their solution to an instructor's solution, but it is unclear the best way to design such prompts to support learning. We designed and deployed a randomized controlled trial during online programming homework, where we provided students with an instructor's solution, and randomized whether they were prompted to compare their solution to the instructor's, to fill in the blanks for a written explanation of the instructor's solution, to do both, or neither. Our results suggest that these prompts can effectively engage students in reflecting on instructor solutions, although the results point to design trade-offs between the amount of effort that different prompts require from students and instructors, and their relative impact on learning.

Prior work suggests that novice programmers are greatly impacted by the feedback provided by their programming environments. While some research has examined the impact of feedback on student learning in programming, there is no work (to our knowledge) that examines the impact of adaptive immediate feedback within programming environments on students' desire to persist in computer science (CS). In this paper, we integrate an adaptive immediate feedback (AIF) system into a block-based programming environment. Our AIF system is novel because it provides personalized positive and corrective feedback to students in real time as they work. In a controlled pilot study with novice high-school programmers, we show that our AIF system significantly increased students' intentions to persist in CS, and that students using AIF had greater engagement (as measured by their lower idle time) compared to students in the control condition. Further, we found evidence that the AIF system may improve student learning, as measured by student performance in a subsequent task without AIF. In interviews, students found the system fun and helpful, and reported feeling more focused and engaged. We hope this paper spurs more research on adaptive immediate feedback and the impact of programming environments on students' intentions to persist in CS.

While programming, novices often lack the ability to effectively seek help, such as when to ask for a hint or feedback. Students may avoid help when they need it, or abuse help to avoid putting in effort, and both behaviors can impede learning. In this paper we present two main contributions. First, we investigated log data from students working in a programming environment that offers automated hints, and we propose a taxonomy of unproductive help-seeking behaviors in programming. Second, we used these findings to design a novel user interface for hints that subtly encourages students to seek help with the right frequency, estimated with a data-driven algorithm. We conducted a pilot study to evaluate our data-driven (DD) hint display, compared to a traditional interface, where students request hints on-demand as desired. We found students with the DD display were less than half as likely to engage in unproductive help-seeking, and we found suggestive evidence that this may improve their learning.

In this paper, we introduce ProgSnap2, a standardized format for logging programming process data. ProgSnap2 is a tool for computing education researchers, with the goal of enabling collaboration by helping them to collect and share data, analysis code, and data-driven tools to support students. We give an overview of the format, including how events, event attributes, metadata, code snapshots and external resources are represented. We also present a case study to evaluate how ProgSnap2 can facilitate collaborative research. We investigated three metrics designed to quantify students' difficulty with compiler errors - the Error Quotient, Repeated Error Density and Watwin score - and compared their distributions and ability to predict students' performance. We analyzed five different ProgSnap2 datasets, spanning a variety of contexts and programming languages. We found that each error metric is mildly predictive of students' performance. We reflect on how the common data format allowed us to more easily investigate our research questions.

Students often get stuck when programming independently, and need help to progress. Existing, automated feedback can help students progress, but it is unclear whether it ultimately leads to learning. We present Step Tutor, which helps struggling students during programming by presenting them with relevant, step-by-step examples. The goal of Step Tutor is to help students progress, and engage them in comparison, reflection, and learning. When a student requests help, Step Tutor adaptively selects an example to demonstrate the next meaningful step in the solution. It engages the student in comparing "before" and "after" code snapshots, and their corresponding visual output, and guides them to reflect on the changes. Step Tutor is a novel form of help that combines effective aspects of existing support features, such as hints and Worked Examples, to help students both progress and learn. To understand how students use Step Tutor, we asked nine undergraduate students to complete two programming tasks, with its help, and interviewed them about their experience. We present our qualitative analysis of students' experience, which shows us why and how they seek help from Step Tutor, and Step Tutor's affordances. These initial results suggest that students perceived that Step Tutor accomplished its goals of helping them to progress and learn.

The goal of this workshop is to bring together the existing community of researchers working on Infrastructure Design for Data-Intensive Research in Computer Science Education and a community of Learning at Scale researchers focused on Computer Science Education. While both communities share many similar goals and could greatly benefit from each other work, the interaction between the communities is small. We hope that the proposed workshop will be instrumental in bringing together like-minded researchers from different communities, establishing collaboration, and expanding the scope of infrastructure project to address critical scaling issues.

The analysis of data plays an increasingly critical role in computing education research, enabled by more and larger datasets, more powerful analysis techniques and better infrastructure for sharing. This panel brings together four panellists at various stages of work involving the collection and analysis of large datasets in different fields of computing education. The panellists will each discuss the current state of their work, the unique aspects of their data, and how that data fits into the larger landscape of computing education and research. Panellists will be asked to explain how they are employing AI and data mining techniques to learn about learners, the research methods they have used to make this happen, and any significant key findings they have discovered through this processes. The panel will discuss emerging topics, including: going beyond log data, handling global-scale datasets, efficiently collaborating with cross-dataset analysis, and ethical and privacy considerations. After the panelists present (5 minutes each), the moderator will pose follow-up questions and invite the audience to pose additional questions or provide other feedback. Key takeaways will include how data mining and artificial intelligence can contribute to improved insight and learning gains and how the larger computer education community can participate in data collection or analysis.

This paper introduces Crescendo, a self-paced programming practice environment that combines the block-based and visual, interactive programming of Snap!, with the structured practices commonly found in Drill-and-Practice Environments. Crescendo supports students with Parsons problems to reduce problem complexity, Use-Modify-Create task progressions to gradually introduce new programming concepts, and automated feedback and assessment to support learning. In this work, we report on our experience deploying Crescendo in a programming camp for middle school students, as well as in an introductory university course for non-majors. Our initial results from field observations and log data suggest that the support features in Crescendo kept students engaged and allowed them to progress through programming concepts quickly. However, some students still struggled even with these highly-structured problems, requiring additional assistance, suggesting that even strong scaffolding may be insufficient to allow students to progress independently through the tasks.

A growing body of work has explored how to automatically generate hints for novice programmers, and many programming environments now employ these hints. However, few studies have investigated the efficacy of automated programming hints for improving performance and learning, how and when novices find these hints beneficial, and the tradeoffs that exist between different types of hints. In this work, we explored the efficacy of next-step code hints with 2 complementary features: textual explanations and self-explanation prompts. We conducted two studies in which novices completed two programming tasks in a block-based programming environment with automated hints. In Study 1, 10 undergraduate students completed 2 programming tasks with a variety of hint types, and we interviewed them to understand their perceptions of the affordances of each hint type. For Study 2, we recruited a convenience sample of participants without programming experience from Amazon Mechanical Turk. We conducted a randomized experiment comparing the effects of hints' types on learners' performance and performance on a subsequent task without hints. We found that code hints with textual explanations significantly improved immediate programming performance. However, these hints only improved performance in a subsequent post-test task with similar objectives, when they were combined with self-explanation prompts. These results provide design insights into how automatically generated code hints can be improved with textual explanations and prompts to self-explain, and provide evidence about when and how these hints can improve programming performance and learning.

Parsons problems are program puzzles, where students piece together code fragments to construct a program. Similar to block-based programming environments, Parsons problems eliminate the need to learn syntax. Parsons problems have been shown to improve learning efficiency when compared to writing code or fixing incorrect code in lab studies, or as part of a larger curriculum. In this study, we directly compared Parsons problems with block-based programming assignments in classroom settings. We hypothesized that Parsons problems would improve students' programming efficiency on the lab assignments where they were used, without impacting performance on the subsequent, related homework or the later programming project. Our results confirmed our hypothesis, showing that on average Parsons problems took students about half as much time to complete compared to equivalent programming problems. At the same time, we found no evidence to suggest that students performed worse on subsequent assignments, as measured by performance and time on task. The results indicate that the effectiveness of Parsons problems is not simply based on helping students avoid syntax errors. We believe this is because Parsons problems dramatically reduce the programming solution space, letting students focus on solving the problem rather than having to solve the combined problem of devising a solution, searching for needed components, and composing them together.

Automated hints, a powerful feature of many programming environments, have been shown to improve students' performance and learning. New methods for generating these hints use historical data, allowing them to scale easily to new classrooms and contexts. These scalable methods often generate next-step, code hints that suggest a single edit for the student to make to their code. However, while these code hints tell the student what to do, they do not explain why, which can make these hints hard to interpret and decrease students' trust in their helpfulness. In this work, we augmented code hints by adding adaptive, textual explanations in a block-based, novice programming environment. We evaluated their impact in two controlled studies with novice learners to investigate how our results generalize to different populations. We measured the impact of textual explanations on novices' programming performance. We also used quantitative analysis of log data, self-explanation prompts, and frequent feedback surveys to evaluate novices' understanding and perception of the hints throughout the learning process. Our results showed that novices perceived hints with explanations as significantly more relevant and interpretable than those without explanations, and were also better able to connect these hints to their code and the assignment. However, we found little difference in novices' performance. Our results suggest that explanations have the potential to make code hints more useful, but it is unclear whether this translates into better overall performance and learning.

Tinkering has been shown to have a positive influence on students in open-ended making activities. Open-ended programming assignments in block-based programming resemble making activities in that both of them encourage students to tinker with tools to create their own solutions to achieve a goal. However, previous studies of tinkering in programming discussed tinkering as a broad, ambiguous term, and investigated only self-reported data. To our knowledge, no research has studied student tinkering behaviors while solving problems in block-based programming environments. In this position paper, we propose a definition for tinkering in block-based programming environments as a kind of behavior that students exhibit when testing, exploring, and struggling during problem-solving. We introduce three general categories of tinkering behaviors (test-based, prototype-based, and construction-based tinkering) derived from student data, and use case studies to demonstrate how students exhibited these behaviors in problem-solving. We created the definitions using a mixed-methods research design combining a literature review with data-driven insights from submissions of two open-ended programming assignments in iSnap, a block-based programming environment. We discuss the implication of each type of tinkering behavior for learning. Our study and results are the first in this domain to define tinkering based on student behaviors in a block-based programming environment.

Research in a variety of domains has shown that viewing worked examples (WEs) can be a more efficient way to learn than solving equivalent problems. We designed a Peer Code Helper system to display WEs, along with scaffolded self-explanation prompts, in a block-based, novice programming environment called \snap. We evaluated our system during a high school summer camp with 22 students. Participants completed three programming problems with access to WEs on either the first or second problem. We found that WEs did not significantly impact students' learning, but may have impacted students' intrinsic cognitive load, suggesting that our WEs with scaffolded prompts may be an inherently different learning task. Our results show that WEs saved students time on initial tasks compared to writing code, but some of the time saved was lost in subsequent programming tasks. Overall, students with WEs completed more tasks within a fixed time period, but not significantly more. WEs may improve students' learning efficiency when programming, but these effects are nuanced and merit further study.

iSnap is a block-based programming environment that supports struggling students with on-demand hints and error-checking feedback. iSnap is an extension of Snap!, a creative and novice-friendly programming environment, used in the Beauty and Joy of Computing (BJC) AP CS Principles curriculum. iSnap is designed to support the open-ended, exploratory programming problems of BJC, while adapting to many possible student solutions. When students ask iSnap for help, it highlights possible errors in their code and suggests next steps they can make. Hints are presented visually, right alongside students/ code, making them easy to interpret and implement. iSnap/s hints are generated automatically from student data, so no teacher input is required to create them, making iSnap appropriate for both new and experienced instructors. The demonstration will showcase iSnap/s hints on a variety of assignments and explain how the algorithm is working behind the scenes to generate data-driven hints. It will also include an overview of the results from two years of research with iSnap on how students seek and use programming help. A key objective of this demonstration is to solicit feedback from SIGCSE attendees on the design of iSnap as we work to make the system ready for deployment in classrooms. More information on iSnap can be found at http://go.ncsu.edu/isnap.

Instructional supports (Supports) help students learn more effectively in intelligent tutoring systems and gamified educational environments. However, the implementation and success of Supports vary by environment. We explored Support design in an educational programming game, BOTS, implementing three different strategies: instructional text (Text), worked examples (Examples) and buggy code (Bugs). These strategies are adapted from promising Supports in other domains and motivated by established educational theory. We evaluated our Supports through a pilot study with middle school students. Our results suggest Bugs may be a promising strategy, as demonstrated by the lower completion time and solution code length in assessment puzzles. We end reflecting on our design decisions providing recommendations for future iterations. Our motivations, design process, and study's results provide insight into the design of Supports for programming games.

When novice students encounter difficulty when learning to program, some can seek help from instructors or teaching assistants. This one-on-one tutoring is highly effective at fostering learning, but busy instructors and large class sizes can make expert help a scarce resource. Increasingly, programming environments attempt to imitate this human support by providing students with hints and feedback. In order to design effective, computer-based help, it is important to understand how and why students seek and avoid help when programming, and how this process differs when the help is provided by a human or a computer. We explore these questions through a qualitative analysis of 15 students' interviews, in which they reflect on solving two programming problems with human and computer help. We discuss implications for help design and present hypotheses on students' help-seeking behavior.

As more programming environments add logging features and programming data becomes more accessible, it is important to have a conversation about how we share and use this data. Uses of programming log data range from big-picture analyses to dashboards for instant teacher feedback, to intelligent, data-driven learning environments. The goal of this BOF is to talk about what data is important to collect, where it can be gathered and shared, what general data formats make sense, how to handle privacy and anonymization, and what ultimately we want to see the data used for. The BOF welcomes both producers of programming log data and current or potential consumers, interested in how it could be applied in their classrooms or research. One hopeful outcome of this BOF is a commitment to documenting and sharing existing programming data in an accessible location and format.

Programming environments intentionally designed to support novices have become increasingly popular, and growing research supports their efficacy. While these environments offer features to engage students and reduce the burden of syntax errors, they currently offer little support to students who get stuck and need expert assistance. Intelligent Tutoring Systems (ITSs) are computer systems designed to play this role, helping and guiding students to achieve better learning outcomes. We present iSnap, an extension to the Snap programming environment which adds some key features of ITSs, including detailed logging and automatically generated hints. We share results from a pilot study of iSnap, indicating that students are generally willing to use hints and that hints can create positive outcomes. We also highlight some key challenges encountered in the pilot study and discuss their implications for future work.

This showpiece will present iSnap, an extension of the block-based, novice programming environment Snap!, which supports struggling students by providing on-demand hints and feedback that help them complete programming assignments. iSnap extends the existing syntactic scaffolding offered by block-based programming to additionally support the implementation of programming tasks. Research on iSnap has explored questions of how visual programming environments can better support learners, the impact of this support, and how learners seek and use computer-based help. The showpiece will consist of an interactive demonstration of iSnap, including the user interface experienced by students and the data-driven algorithm used to automatically generate the programming feedback.

Frame-based editing is a novel way to edit programs, which claims to combine the benefits of textual and block-based programming. It combines structured `frames' of preformatted code, designed to reduce the burden of syntax, with `slots' that allow for efficient textual entry of expressions. We present an empirical evaluation of Stride, a frame-based language used in the Greenfoot IDE. We compare two groups of middle school students who worked on a short programming activity in Greenfoot, one using the original Java editor, and one using the Stride editor. We found that the two groups reported similarly low levels of frustration and high levels of satisfaction, but students using Stride progressed through the activity more quickly and completed more objectives. The Stride group also spent significantly less time making purely syntactic edits to their code and significantly less time with non-compilable code.

Clashroom is a web-based fantasy game that was created to enhance the motivation and learning of a traditional classroom. It was created to address several drawbacks of other educational games by being broadly applicable to any course, requiring no class time, and allowing for optional rather than mandatory game play. Each player in the game is a dragon trainer, hoping to train a champion pet dragon. To gain skills and magical items for the dragon to use, players complete educational quests set up by the teacher. Players compete in weekly tournaments, where the quest rewards are vital to success. Because the gameplay and the learning material are decoupled, the game can be used for any subject at any level. After initial testing, students report that the game is fun and that the intrinsic rewards of a fun game are motivation enough for extra learning.

Computer Science Principles (CSP) will become an Advanced Placement course during the 2016-17 school year, and there is an immediate need to train new teachers to be leaders in computing classrooms. From 2012-2015, the Beauty and Joy of Computing team offered professional development (PD) to 133 teachers, resulting in 89 BJC CSP courses taught in high schools. Our data show that the PD improved teachers' confidence in our four core content categories and met its primary goal of training teachers in equitable, inquiry-based instruction. In this paper, we present the evolution of the BJC PD, its challenges and lessons that we learned while continually adapting to teachers' needs and contexts.

Block Programming Environments (BPEs) are becoming popular tools for introducing novices to programming, due in part to their connection with students' interests in games, apps and stories. This has led to increasing use of BPEs outside of classroom settings, where knowledgeable instructors are not always available. Intelligent Tutoring Systems (ITSs) can keep students on track in the absence of instructors by providing hints and warnings to students in need of help. Further, data-driven techniques can generate this feedback automatically from previous students' attempts at a problem. This research focuses on the integration of this data-driven, ITS-style feedback into a modern BPE and the evaluation of its impact.

Visual, block-based programming environments present an alternative way of teaching programming to novices and have proven successful in classrooms and informal learning settings. However, few studies have been able to attribute this success to specific features of the environment. In this study, we isolate the most fundamental feature of these environments, the block interface, and compare it directly to its textual counterpart. We present analysis from a study of two groups of novice programmers, one assigned to each interface, as they completed a simple programming activity. We found that while the interface did not seem to affect users' attitudes or perceived difficulty, students using the block interface spent less time off task and completed more of the activity's goals in less time.