Background and Context: Replication studies play an important role in Computing Education Research (CER) by supporting the development of consistent and reliable scientific knowledge. However, prior research indicates that the CER community tends to prioritise novel contributions over replication. A 2019 Systematic Literature Review (SLR) identified only 54 (2.38%) replication studies among 2,269 papers published between 2009 and 2018 across five major CER venues. In response, the Computer Science Education journal released a special issue dedicated to replication studies to encourage greater adoption of this research design. Objectives: This study aims to examine how the landscape of replication research in CER has evolved since 2019. Specifically, we investigate whether the prevalence of replication studies has increased and explore current perceptions and experiences of CER researchers regarding replication. Method: We replicated two prior studies. First, we conducted an updated SLR to identify replication studies published between 2019 and 2025 in the same five CER venues. Second, we replicated a survey of Computing Education researchers to better understand their perceptions, experiences, and challenges related to conducting and publishing replication studies. Findings: Our SLR identified 63 (2.50%) replication studies among 2,516 published papers. While the proportion of replication studies has increased slightly, overall growth remains limited. We observed a shift toward more published replication studies in journals and an increase in authors replicating their own prior work. Survey results indicate that although many researchers engage in replication within their teaching and research practice, they encounter significant challenges when attempting to publish replication studies. Implications: Despite increased discourse around open science and research rigour, the adoption of replication studies in CER has not substantially grown. Our findings offer opportunities for future research to promote replication in CER and to explore how the CER community can encourage researchers to publish replication studies.

Motivation: Program visualizations are widely used to support novice programmers, yet students often ignore or resist well-designed visual scaffolds. Research on multiple external representations (MERs) suggests cognitive design principles for coordinating views, but says little about what determines whether learners actually engage with the representations available to them.

Researchers use replication to confirm, strengthen, and advance Computing Education Research (CER). However, prior research shows that replication is infrequently used in CER, even though the community encourages its use. Previous research suggests that the CER community uses different terms to describe replication, which we aim to confirm in this Working Group (WG). We will conduct a Systematic Literature Review (SLR) across influential international CER venues to understand how researchers present and conduct replication studies, possibly identifying venues receptive to papers applying this research design. In addition, we will interview Computing Education researchers, conference leaders, and journal editors to understand their experiences and perceptions with replication. We expect to collect suggestions and recommendations on how CER can encourage more replication in future studies. Our work will highlight and confirm the terms the CER community uses to present replication studies, enabling researchers and educators to better identify these studies.

With the rise of generative AI performing tasks traditionally allocated to entry-level roles, and the current economic slowdown, recent graduates need to demonstrate non-technical skills and behaviours to stand out in the job market. Our study collects perspectives from academic and industry professionals on how graduates can demonstrate essential non-technical skills, also known as soft skills, and workplace behaviours. We used a mixed methods approach to analyse 168 survey responses and 34 interviews. Our findings indicate that communication and collaboration are crucial for entry-level professionals, whereas mentorship and a purpose-driven disposition are less important at this stage of their careers. Our findings suggest that asking questions and reflective thinking showcase non-technical skills and behaviours in recent graduates; these same approaches are also used by industry to assess such attributes, especially during interviews. We conclude with implications and practical advice for educators and researchers to integrate and advance authentic practice, helping students learn to apply and demonstrate non-technical skills and behaviours applicable in the workplace.

Skills-based evaluation is an alternative evaluation model that is a variation of both mastery and specifications grading. Students are evaluated on the set of skills they have demonstrably acquired over the duration of a course and the level at which they are able to demonstrate those skills. This is in opposition to traditional models which evaluate performance at fixed time points. A growing body of research suggests that such alternative evaluation models are more equitable, motivating, and efficacious for students. However, adoption remains limited in part due to concerns about student acceptance, perceived lack of rigour, and the potential increase in workload due to repeated assessments.

Generative Artificial Intelligence (GenAI) tools are increasingly used by computing students, yet their effects on learning outcomes remain mixed. Prior work found that while GenAI use may improve performance on assignments, it can negatively relate to overall course performance. We aim to replicate and extend this work across four computing courses. Using self-reported GenAI usage from assignments and study preferences alongside course performance data, we examine how these relationships vary by course, and compared to the previous study. Our results show that students who used GenAI tools to solve the assignment performed equally or better than those who did not report using it, however, they received lower final grades in the course. We observe no major difference between students who used GenAI to study for the midterm test compared to those who did not. These findings suggest that the impact of GenAI use is present in various contexts, highlighting the need for instructional guidance on how students should use GenAI as a learning aid, and insights for other instructors that wish to integrate GenAI tools into computing curricula.

As computing education becomes increasingly globalized, many students learn computer science through a second language. Prior work has documented cognitive and performance challenges faced by non-native English-speaking (NNES) students, yet less is known about how language background shapes their motivational experiences and intentions to persist. Drawing on Expectancy-Value Theory, we analyzed matched pre- and post-term survey data from 374 students (198 NNES, 176 NES) in a CS1 course at a large North American university. We measured programming self-efficacy, implicit theories of intelligence, need for cognition, sense of belonging, motivation and learning strategies, and intentions to major in computing. NNES students began the course believing intelligence is fixed and had lower self-efficacy on language-dependent tasks, gaps that persisted throughout the term. However, despite lower confidence, NNES students were more willing to choose challenging assignments and consistently used more strategic learning approaches. While NNES and native English-speaking (NES) students reported similar overall belonging, NNES students experienced greater belonging uncertainty, specifically when encountering difficulties, and were more likely to want to fade into the background within the CS community. These findings reveal language background as a persistent motivational cost in introductory computing and underscore the need for instructional designs that explicitly support belonging, self-efficacy, and adaptive strategy use for linguistically diverse learners.

Motivation: Database courses focus on SQL querying (DQL) while treating schema definition (DDL) and data manipulation (DML) as side topics, even though real-world database work begins with understanding schema design and data updates. This misalignment leaves students underprepared for authentic data management practice. Method: We integrated scaffolded DDL and DML exercises as a core concept in a third-year data course across three offerings (2023-2025). Students completed structured weekly tasks in an LMS that provides immediate feedback and unlimited attempts, encouraging low-stakes, iterative practice. We analyzed student interaction data (number of attempts and performance) to examine learning patterns across DDL/DML and DQL. We analyzed 9,071 total exercise submissions from 669 students, examining both the number of LMS exercise attempts and assignment performance across DDL/DML and DQL. Results: Students required fewer attempts on DDL/DML tasks than on traditional DQL tasks, indicating strong receptiveness when these topics were properly scaffolded. Early performance on schema-definition tasks was moderately correlated with later SQL performance, suggesting that schema competence supports subsequent query learning. Implications: We encourage database educators to teach schema design and data manipulation as core topics to strengthen students' conceptual foundations, as our results suggest these skills are learnable with scaffolding and may support subsequent query learning.

Systematic Literature Reviews (SLRs) are fundamental to scientific progress, yet the process is hindered by a fragmented tool ecosystem that imposes a high cognitive load. This friction suppresses the iterative, exploratory nature of scholarly work. To investigate these challenges, we conducted an exploratory design study with 20 experienced researchers. This study identified key friction points: 1) the high cognitive load of managing iterative query refinement across multiple databases, 2) the overwhelming scale and pace of publication of modern literature, and 3) the tension between automation and scholarly agency.

This pilot study investigated students' perceptions of visual progress bar and on-demand hints in a CS1 course. Students valued both feedback systems, with more confident students preferring on-demand hints for direct problem-solving support. Progress bars were consistently perceived as beneficial across varying confidence levels.

Access to STEM education is limited in many low-resource areas due to inadequate access to computers, the internet, trained educators, and reliable electricity. However, according to the GSM Association, over 54% of the global population owns a smartphone, presenting an opportunity for mobile-based learning solutions. To address this gap, we developed ShapeCreator Mobile, a learning tool designed to deliver our Algebraic Thinking curriculum to children with limited access to laptops but who own smartphones. This curriculum strengthens foundational math skills through coding, preparing students for high school algebra, a gateway to STEM careers. This approach bridges the digital divide and promotes inclusivity in STEM education in low-resource environments.

Self-explanation promotes active learning by having students articulate their conceptual understanding in their own words. This study investigates whether the timing of self-explanations (before vs. after solving an exercise) relates to performance in a flipped, second-year computer organization course. Although students who self-explained before the exercises achieved higher course marks, the difference was not significant. Still, these findings suggest that early self-explanation may better prepare students for problem-solving when learning new concepts.

Self-explanation deepens understanding by giving learners an opportunity to reflect on what they are learning in a structured way. However, many students struggle to engage in it effectively. We investigate whether large language models (LLMs) can scaffold self-explanations in a flipped computer organization course. In an A/B test, one group used a fixed prompt to compare their explanations with an expert's, while another engaged in an interactive dialogue with an LLM to identify gaps. Although the overall quality of the explanation did not differ significantly between conditions, some students (non-native English speakers and women) reported greater comfort and perceived value when using the LLM.

There is a wealth of evidence that involving undergraduate students in research has positive impacts in a variety of areas, from representation and retention to outcomes and self-efficacy. However, developing and growing an undergraduate research program can be daunting, especially for institutions that do not have a large existing research enterprise. In this work, we detail a program that revolves around student-developed literature maps to help students gain the ability to read and assess research papers in a way that is accessible, robust, and requires relatively little faculty overhead. We further detail how this program has been run through 4 iterations, with a total of 47 students producing 5 posters or short papers, and 3 full papers. In this work, we provide our experiences using literature mapping projects to boot-strap an undergraduate research program and provide quantitative and qualitative analysis of the students who have participated. All of the materials, including sample spreadsheets, and scripts to generate LaTeX tables and figures are included for anyone wishing to undertake a literature mapping project of their own.

Generative Artificial Intelligence (GenAI) and Large Language Models (LLMs) have led to changes in educational practices by creating opportunities for personalized learning and immediate support. Computer science student perceptions and behaviors towards GenAI tools have been studied, but the effects of such tools on student learning have yet to be determined conclusively. We investigate the impact of GenAI tools on computing students' performance in a database course and aim to understand why students use GenAI tools in assignments. Our mixed-methods study (N=226) asked students to self-report whether they used a GenAI tool to complete a part of an assignment and why. Our results reveal that students utilizing GenAI tools performed better on the assignment part in which LLMs were permitted but did worse in other parts of the assignment and in the course overall. Also, those who did not use GenAI tools viewed more discussion board posts and participated more than those who used ChatGPT. This suggests that using GenAI tools may not lead to better skill development or mental models, at least not if the use of such tools is unsupervised, and that engagement with official course help supports may be affected. Further, our thematic analysis of reasons for using or not using GenAI tools, helps understand why students are drawn to these tools. Shedding light into such aspects empowers instructors to be proactive in how to encourage, supervise, and handle the use or integration of GenAI into courses, fostering good learning habits.

Creating a supportive community in introductory programming courses is vital to student success, yet forums meant to facilitate this can cause stress due to social comparison. According to social identity theory, students are more likely to engage and feel a sense of belonging when they perceive connections with their peers. This study investigates whether peer-modeled posts that simulate students exhibiting desirable engagement behavior can reduce feelings of isolation and foster social connection among students. We introduced curated posts modeling expected student behavior -- covering content, providing emotional support, and offering study tips -- into Q&A forums for two introductory computing courses. These posts were inserted using different student accounts. Surveys and forum data were analyzed to measure the impact on students' feelings of isolation. Students responded positively to the seeded posts, reporting a significant reduction in feelings of isolation. Notably, women reported feeling less isolated after seeing the posts more than men, and many students reported feeling relieved that other students had the same worries and concerns as them. Seeding peer-modeled posts can significantly reduce student isolation and foster a greater sense of belonging in competitive academic contexts. However, future work may explore alternative delivery mechanisms, such as instructor posts framed as ''questions from last year,'' to determine if they can achieve similar effects.

Although engagement in self-directed wellness exercises typically declines over time, integrating social support such as coaching can sustain it. However, traditional forms of support are often inaccessible due to the high costs and complex coordination. Large Language Models (LLMs) show promise in providing human-like dialogues that could emulate social support. Yet, in-depth, in situ investigations of LLMs to support behavior change remain underexplored. We conducted two randomized experiments to assess the impact of LLM agents on user engagement with mindfulness exercises. First, a single-session study, involved 502 crowdworkers; second, a three-week study, included 54 participants. We explored two types of LLM agents: one providing information and another facilitating self-reflection. Both agents enhanced users' intentions to practice mindfulness. However, only the information-providing LLM agent, featuring a friendly persona, significantly improved engagement with the exercises. Our findings suggest that specific LLM agents may bridge the social support gap in digital health interventions.

This research evaluates voice-based self-explanations as a pedagogical tool in preparation for lectures, assesses user preferences between voice and text, and derives design insights. We report two studies: Study 1, a quasi-experimental field study, with 247 participants divided into voice-based (N = 83), text-based (N = 81), and choice (N = 83) conditions. Study 2 uses semi-structured interviews (N = 16) to explore perceptions of the interaction paradigms in-depth. Results from the first study revealed a general preference for text, though voice users produced longer responses and more topic-related keywords. Over time, the preference for voice increased among students, from 10% to 46%, when given a choice. Study 2 suggested that factors like social presence contribute to hesitance toward voice-based explanations, with a cognitive load, self-confidence, and performance anxiety also influencing medium preferences. Our findings highlight design recommendations and demonstrate the potential of voice-based self-explanations in educational settings, indicating that mixed interfaces might better meet diverse needs.

Motivation: Q&A forums are a critical resource for supporting students in large educational environments, yet students often perceive these forums as stressful and report discomfort in participating visibly, especially in classes that are large and have students with varying levels of prior programming experience (PE). Method: We divided students in a CS1 Q&A forum into smaller, homogenous groups based on their PE. We use a mixed-methods approach to compare data from this experience to data from a setting where all students shared a single, large Q&A forum (a “mixed” setting). We quantitatively analyze measures of student engagement and use an open-ended qualitative approach to examine responses about student experience on the forums. This approach helps us identify the motivation behind student decisions to participate in visible or non-visible ways and to evaluate their alignment with theoretical frameworks. Results: In the mixed setting, students frequently use anonymity, with students without PE using anonymity more than students with PE and women using anonymity more than men. In contrast, in the homogenous groups, novices used anonymity less than novices in the mixed setting, while the students in higher-experience groups tended to use it more. We also observe a reduced anonymity usage among women in the homogenous experience groups, suggesting that PE plays a critical role in the observed gender disparities in forum participation. The qualitative analysis provides additional evidence that social status issues and confidence may explain these behavioral patterns. Conclusion: This study highlights the potential benefits and consequences of grouping students by experience. Homogenous PE groups foster increased student comfort and engagement within the Q&A forum for students with less experience, but students with more experience are exposed to more perceived status threats. We discuss how these results align with the theories we used to design the homogenous group setting. This exploration contributes to a deeper understanding of the underlying dynamics shaping student behavior in online learning communities. Educators and platform designers can use these lessons to more effectively create inclusive environments that accommodate diverse student needs and preferences.

The ITiCSE '23 final keynote raised teaching soft skills, or professional dispositions, to help students face challenges in modern programming. This project addresses helping computing students develop professional dispositions through collaborative learning (CL) since some in the industry observe entry-level engineers struggling due to their fragile professional dispositions. We are motivated to understand professional expectations from entry-level engineers and present the academia-industry gap to support practitioners and researchers in advancing CL in Computing Education, encouraging positive curricula and policy changes that promote DEIA. We will present CL practices alongside their supported professional dispositions to assist practitioners in adoption. We will present the academia-industry gap in CL for future research opportunities, helping researchers advance CL practices to integrate professional dispositions the industry expects from entry-level engineers.

In computing courses, instructor involvement and social comfort are vital for resilience and belonging. We examine engagement with instructor emails aimed at strengthening the connection with students. We sent weekly emails from instructors to first- and upper-year computing students. These emails included reminders for the assignments due each week. Half of the students received reminders embedded in an informal message that contained approachable wording and relevant current course events, while the rest received a list of precise deadlines. This text had no emotional engagement from the instructor. We collected and analyzed email access and link click rates, along with student survey responses about email preferences and engagement. We found that first-year students had lower email access and link click rates than upper-year students. While we did not find differences in first-year engagement based on the type of email, upper-year students appeared to be more engaged when receiving the intentionally informal version of the email. Understanding the message preferences of computing students can enhance instructor messaging and improve engagement. Strategies should be explored to boost first-year student engagement, while the higher engagement among upper-year students underscores the importance of instructor support in advanced courses.

Asking for help (help-seeking) is a recognized and effective problem-solving strategy. This study investigates students' interaction with on-demand hints (automated hints requested by students) and assesses their impact on learning progress. We conducted an A/B experiment in a third-year computer science database course, offering hints for selected SQL problems with different hint designs. We collected data on students' code submissions, grades, and hint requests, and we administered a survey to gather feedback and gauge student perception of the hints. Many students accessed hints immediately without attempting the problem first, often requesting multiple hints in quick succession. While students perceived the hints to be valuable, we did not detect an impact on student problem-solving. These insights could inform future studies on the possible impact of students' attitudes toward hints, and how different types of hints might impact uptake and perception of hints.

This study explores links between attributes of computing students, such as prior programming experience (PE) and gender, with expectations for success and the perception of challenges. Using Expectancy-Value Theory (EVT), we investigate their major intentions and the impact of these factors post-CS1. Data was gathered using surveys at the beginning and end of an introductory programming course, focusing on demographics, expectations of success, and perceptions of challenges. Application status for the computing major was also recorded. Our results revealed that men and students with PE generally perceived greater potential for success and reported facing fewer challenges. In contrast, women and students without PE more often indicated concerns about intellectual ability and perceived challenges less positively. Notably, while gender appears in the preceding results, an intersectional analysis indicates that PE is the central factor. PE is also linked to persistence in the field of computing. Our results further highlight the importance of providing students with opportunities to develop experience, as it can help shape their expectations, perceived challenges, and retention in computing.

Stress reappraisal interventions try to shift students’ negative perceptions towards eustress, stress that can be beneficial, and help them perform better. However, it is less clear how to present them to users as online interventions that are brief, voluntary, and scale well in real-world contexts. We explore the design of online exam eustress interventions by generating six design factors (D1-6) that reinforce a core reappraisal message (D0), and evaluate them through: (i) user interviews (N = 20) revealing six findings (F1-6) on the importance of elaboration, layout, modality, and source of intervention content; (ii) a field experiment (N = 1283) showing a significant positive effect on exam scores (p = 0.003). Subgroup analyses indicate a significant effect for first-year but not for upper-year students, and no detectable gender differences. Our work offers insight into how students interact with online mindset interventions and design considerations for incorporating them into large courses.

Recent work has confirmed that computing students experience the Imposter Phenomenon (IP) at higher rates than reported in other disciplines. However, no work has examined what aspects of the university computing experience might lead to a higher rate of IP experiences. We aim to illustrate the IP experiences students have, identify common sources of these experiences, and document the effects of these experiences and how students respond to them. We asked undergraduate students to share recent experiences that illustrate their experiences with the IP. We conducted an inductive thematic analysis on these open-ended responses, resulting in a set of inter-connected themes. A significant fraction of students related stories about making comparisons with peers or observing peer behaviour that made them question their abilities. Students also spoke about holding unrealistic expectations learned from their peers or imposed by the environment. These experiences may be particularly acute for minority-affiliated students who may come to feel they do not belong. Ultimately, these IP experiences can lead to a loss of motivation or a cycle of failure that leads students to leave computing. The central role social comparisons play in IP experiences suggests that it is particularly important to foster communities where opportunities for comparison are reduced and where realistic expectations are explicitly set.

Generating self-explanations has been identified as a successful strategy in helping learners engage with course content and organize what they learn in a structured format. While typing an explanation may allow more structure and formality, explaining by voice can be more natural and help free cognitive resources to focus on learning goals and understanding concepts. As we investigated the effects and students' perceptions of using voice or text to self-explain new course concepts, we failed to find a tool that would meet our needs. We present our work in designing and developing VoiceEx, a submission courseware that allows text and voice input to collect data in both mediums. VoiceEx was created to support a self-explanations intervention for computer science students; however, given its features and the advantages of being able to collect spoken responses, it can be used in a variety of environments. Future refinement of this tool includes artificial intelligence features to better guide students' submissions.

In this poster, we present a pilot study investigating the impact of the medium used for self-explanation on students' performance outcomes in a databases course. We did not see notable differences in student performance based on the medium they used for self-explanation. We also note that while most students prefer using text to submit self-explanations, their preferences may differ when they use voice.

Many novice programmers struggle to write code from scratch and get frustrated when their code does not work. Parsons problems can reduce the difficulty of a coding problem by providing mixed-up blocks that the learner assembles in the correct order. Parsons problems can also include distractor blocks that are not needed in a correct solution, but which may help students learn to recognize and fix errors. Evidence indicates that students find Parsons problems engaging, easier than writing code from scratch, useful for learning patterns, and typically faster to solve than writing code from scratch with equivalent learning gains. This working group leverages the work of the 2022 ITiCSE working group which published an extensive literature review of Parsons problems and designed and piloted several studies based on the gaps identified by the literature review. The 2023 working group is revising, conducting, and creating new studies. We will analyze the data from these multi-institutional and multi-national studies and publish the results as well as recommendations for future working groups.

Q&A forums are widely used in large classes to provide scalable support. In addition to offering students a space to ask questions, these forums aim to create a community and promote engagement. Prior literature suggests that the way students participate in Q&A forums varies and that most students do not actively post questions or engage in discussions. Students may display different participation behaviours depending on their comfort levels in the class. This paper investigates students' use of a Q&A forum in a CS1 course. We also analyze student opinions about the forum to explain the observed behaviour, focusing on students' lack of visible participation (lurking, anonymity, private posting). We analyzed forum data collected in a CS1 course across two consecutive years and invited students to complete a survey about perspectives on their forum usage. Despite a small cohort of highly engaged students, we confirmed that most students do not actively read or post on the forum. We discuss students' reasons for the low level of engagement and barriers to participating visibly. Common reasons include fearing a lack of knowledge and repercussions from being visible to the student community.

Previous work has reported on the advantageous effects of prior experience in CS1, but it remains unclear whether these effects fade over a sequence of introductory programming courses. Furthermore, while student perceptions suggest that prior experience remains important, studies have reported that a student's expectation of their performance is a more accurate predictor of outcome. We aim to confirm if prior experience (formal or informal) provides short-term and long-term advantages in computing courses or if the advantage fades. Furthermore, we explore whether the expectation of performance is a more accurate predictor of student success than informal and formal prior experience. To explore these questions, we deployed surveys in a CS1 course to gauge students' level of prior experience in programming, prediction of final exam grades, and self-efficacy to succeed in university. Grades from CS1 and CS2 were also collected. We observed a persistent (1-letter grade) gap between the performance of students with no prior experience and those with any experience, but we did not observe a noteworthy gap when comparing student performance based on formal or informal experience. We also observed differences in self-efficacy and retention rates between different levels of prior experience. Lastly, we confirm that success in CS1 can be better reflected and predicted by some controllable factors, such as students' perceptions of ability.

Many students are first exposed to computing in a programming course such as CS1. This course affects their understanding of computing and may affect their intention to major in the program. We investigate the intention to major in computing in relation to demographic factors and factors related to academic success. We deployed surveys at the beginning and end of a CS1 course to gauge students' level of prior experience in programming, elicit demographic factors such as gender and parental education level, and identify their intention to major in computing. Grades from CS1 and CS2 were also collected. Our results suggest that most students do not change their intention to major in computing after taking CS1. Students who were more likely to intend to major in programming at the beginning of the course were those with prior experience, those who identified as men, or students who had a parent with a bachelor's or post-grad degree. We also find that students' grades correlate to their change in program intention. This reinforces the need to change perceptions about computing early, prior to CS1.

Many studies have shown the efficacy of pair programming for students learning to program. However, most of these studies have taken place in an in-person environment, where the driver and navigator are physically sharing a keyboard and screen and can communicate verbally and non-verbally. With the increase in online learning, especially during the COVID-19 pandemic. It is important to know whether these results generalize to an online environment.

Drawing inspiration from machine learning and experimentation in product development at leading technology companies, we explore how adaptive experimentation might help in continuous course improvement. In adaptive experiments, as different arms/conditions are deployed to students, data is analyzed and used to change the experience for future students. We discuss an example side-by-side comparison of traditional and adaptive experimentation of self-explanation prompts in online homework problems in a CS1 course. This provides the first step in exploring the future of how this approach can help bridge research and practice in continuous course improvement.

Instructors often prefer to use email for course communication. The use of emails has been widely discussed in the fields of marketing and behavioural design, but the prevalence of email in education makes it important for instructors to collect metrics on emails to see how students engage with them. One component of emails are the subject lines, which constitute as one of the first things a receiver sees before deciding to open an email. This poster discusses a case study at deploying an email intervention in an online CS1 course. We investigate how the length of subject lines impact the rate at which students open emails of a particular type that prompts them to start their homework early. We aim to share key results to inform instructors how to design their emails to better reach students. Further, we highlight the potential benefits for instructors when collecting and analyzing email engagement data.

Digital experiments can be used in CSedu to test hypotheses about interventions and conditions' efficacy (or inefficacy). This workshop will discuss and deconstruct the design process and analysis for various experiments conducted in CS1. E.g., experiments testing which explanations students find helpful, which emails get them to start homework early, or which webpages effectively encourage and motivate students. This workshop teaches participants how to conduct, interpret, and analyze adaptive field experiments. These adaptive experiments employ machine learning algorithms to analyze experiments during deployment and dynamically shift the allocation of arms/conditions to give future students better conditions more rapidly. Adaptive field experiments can accelerate scientific discovery by enabling more complex experimental designs and increasing statistical power by phasing conditions in and out more efficiently. The workshop is supported by a 5-year NSF grant to build software tools and a digital community, gathering instructors, domain scientists and methodologists to teach them how to run adaptive experiments. The methodological focus includes understanding: (1) which algorithms are best for adaptive experiments that meet domain scientists' needs in specific experimental designs and data sets; (2) which hypothesis tests and Bayesian analyses to choose. Software companies use these innovative methodologies extensively to continuously improve product design. This workshop demonstrates how the same methods can be used in CSedu to improve research rigor and accelerate educational research implementation, ultimately improving student outcomes.

Email communication between instructors and students is ubiquitous, and it could be valuable to explore ways of testing out how to make email messages more impactful. This paper explores the design space of using emails to get students to plan and reflect on starting weekly homework earlier. We deployed a series of email reminders using randomized A/B comparisons to test alternative factors in the design of these emails, providing examples of an experimental paradigm and metrics for a broader range of interventions. We also surveyed and interviewed instructors and students to compare their predictions about the effectiveness of the reminders with their actual impact. We present our results on which seemingly obvious predictions about effective emails are not borne out, despite there being evidence for further exploring these interventions, as they can sometimes motivate students to attempt their homework more often. We also present qualitative evidence about student opinions and behaviours after receiving the emails, to guide further interventions. These findings provide insight into how to use randomized A/B comparisons in everyday channels such as emails, to provide empirical evidence to test our beliefs about the effectiveness of alternative design choices.

Motivation Despite the widespread belief that computing practitioners frequently experience the Imposter Phenomenon (IP), little formal work has measured the prevalence of IP in the computing community despite its negative effect on achievement.

With the widespread usage of mobile devices, users can now choose to provide input through voice or text. As researchers frequently ask students open-ended questions, we want to explore a natural mode to obtain better feedback in surveys. This study details a preliminary study demonstrating the importance of allowing students to choose between voice or text input to respond to surveys. A survey with several open-ended questions was deployed in a CS1 course. Correlations between the gender of the respondent and their method of responding were evaluated. We found that voice responses tended to be longer and preferred more by females relative to male students.

Procrastination by students may lead to adverse outcomes such as a focus on completion rather than learning or even a failure to complete learning tasks. One common method for motivating students and reducing procrastination is to send reminders with hints and study strategies, but it's not clear if these messages are effective or when is the best time to send them. Randomized A/B comparisons could be used to try different reminders or alternative ideas about how best to get students to start work earlier and, crucially, to measure the impact of these interventions on behaviour. This paper describes an A/B comparison of reminder emails set in a large CS1 course at a research-focused North American university. We found evidence that the email interventions caused a higher proportion of students to attempt the online homework but did not see evidence that these particular emails got students to start early, irrespective of changes to the timing of the reminder. More broadly, these findings illustrate how to use A/B comparisons in educational settings to test ideas about how to help students, and demonstrate the value of using randomized A/B comparisons, even when evaluating actions that seem obviously beneficial, such as reminder emails.

Several studies have shown mixed results when presenting enhanced (simplified or extended) error messages to introductory programming students. In this work, we detail a pilot study using a tool specifically designed to capture data about students presented with different error messages. Initial data indicates that the tool is capable of capturing relevant data, and that future studies may show an impact of modifying error messages.

Many research papers pull data from student surveys. But are those surveys well designed? Are the questions used validated? Are the results comparable across studies? What exactly are we asking our students? In this work, we performed a systematic literature map of the past 15 years of papers in the three main conferences sponsored by the ACM Special Interest Group on Computer Science Education: International Computing Education Research (ICER), Innovation and Technology in Computer Science Education (ITiCSE), and the Special Interest Group on Computer Science Education Technical Symposium (SIGCSE). We search for all papers referring to student surveys or questionnaires. Out of 1313 papers analyzed, 42 papers referred to surveys containing general questions applicable to many or all computer science students. Our analysis showed that many papers were using surveys to extract similar types of information, such as demographics, prior experience or motivation to study computer science. However, the questions were being asked in different ways, using different scales, thus making it difficult or impossible to compare survey results between studies. We further found that while some studies based their questions on well-validated surveys, or at least shared their questions for possible later validation, approximately half of the papers found neither validated their questions, nor shared them to allow for post-hoc validation.

Many CS education research papers pull data from student surveys. But are those surveys well designed? Are the questions used validated? Are the results comparable across studies? What exactly are we asking our students? In this work, we performed a systematic literature review of the past 15 years of papers in the three main conferences sponsored by the ACM Special Interest Group on Computer Science Education. Out of 1313 papers analyzed, 44 papers referred to general questions applicable to many or all computer science students. Our analysis showed that many papers were using surveys to extract similar types of information, such as demographics, prior experience or motivation to study computer science. However, the questions were being asked in different ways, using different scales, thus making it difficult or impossible to compare survey results between studies. We further found that while some studies based their questions on well-validated surveys, or shared their questions for possible later validation, approximately half of the papers retrieved neither validated their questions nor shared them to allow for post-hoc validation.