Reflection activities in introductory programming courses help students assess their learning experience, but prior work overlooks how they translate struggles into improvement plans. We analyzed 906 open-ended reflections from two introductory programming courses on coding struggles (code design, debugging, concepts, timeliness) vs. improvement plans (timeliness, debugging, implementation, collaboration) aggregating these into themes. Some themes aligned strongly; others did not, revealing what students perceive as challenging yet actionable in their learning.

Researchers use replication to confirm, strengthen, and advance Computing Education Research (CER). However, prior research shows that replication is infrequently used in CER, even though the community encourages its use. Previous research suggests that the CER community uses different terms to describe replication, which we aim to confirm in this Working Group (WG). We will conduct a Systematic Literature Review (SLR) across influential international CER venues to understand how researchers present and conduct replication studies, possibly identifying venues receptive to papers applying this research design. In addition, we will interview Computing Education researchers, conference leaders, and journal editors to understand their experiences and perceptions with replication. We expect to collect suggestions and recommendations on how CER can encourage more replication in future studies. Our work will highlight and confirm the terms the CER community uses to present replication studies, enabling researchers and educators to better identify these studies.

In recent years, computer science (CS) education has undergone rapid change, first due to the shift to online learning during the COVID-19 pandemic, and more recently with the rise of generative AI (GenAI) tools. How these changes impact students' learning habits and problem-solving attitude remains unknown. This paper presents a long-term analysis of attitudinal survey data from a CS2 course across eight semesters. Drawing on the Computing Attitudes Survey (CAS v4) and the Computer Science Attitudes (CSA) survey, we measure changes in seven constructs, including confidence, mindset, strategies, and motivation. We observe stable or positive shifts in earlier semesters, but sharp negative trends in Fall 2023 and Spring 2024 semesters coinciding with the widespread availability of Generative AI (GenAI) tools. These results highlight the urgent need for instructors to consider how new technologies shape not just students' learning outcomes but their beliefs about their own ability to succeed in computing.

When students leave office hours, what do they do next? This study introduces preliminary work on a behavior-based measure of help-seeking success by exploring the relationship between office hour (OH) interactions and subsequent coding progress. We ask: Can students' code changes following help-seeking interactions serve as evidence of forward progress?

Interest in teaching-track faculty positions has been steadily increasing as enrollments in computer science degree programs continue to trend upward. While departments have welcomed these new teaching-track faculty members, senior faculty, department chairs, and university committees often struggle with how to best evaluate these faculty members during the promotion process. In our experience, some universities try to use a "watered-down" version of the tenure-track promotion standards with the intent of uniformity. Other universities have created whole new processes, which may be better at capturing the differences in teaching-track positions, but also can create a "second-class citizen" status for the teaching-track faculty members.

Office hours are a core feature of CS courses. They provide a crucial vector for personalized instruction and individual support. Despite their importance, we the impact of these personal events has not been well assessed each office hour interaction is often hidden from instructors. We do not know if students actually understand the solution or the advice given during the interaction. This lack of knowledge presents a challenge to efforts geared at improving instructional practices and student outcomes. In this poster we report on the results of a survey analysis of office hour participants. Our results show that students and instructional staff had divergent assessments of the interactions with teaching staff overestimating the benefits more than 18.9% of the time. This work highlights detailed analyses of these results along with implications for teaching strategies.

As Computer Science undergraduate programs continue to increase in response to the still-unmet student demand, one growing trend is to engage current students as instructors of record, allowing for increases in course offerings and/or enrollments and giving these students extraordinary experiences in teaching. Despite these benefits, there are significant concerns that could be raised: do these student instructors have the maturity and training to handle the additional job requirements such as mentoring junior teaching staff, resolving difficult interpersonal situations, addressing diversity, equity, inclusion, and accessibility (DEIA) issues, and garnering respect from students who might be a similar age? The goal of this panel is not to claim this solution is novel, but to share lessons learned, including the student instructor perspective, from faculty whose four institutions have been doing this successfully for years, each taking different approaches to engagement and pre-service professional development.

Figure 1: EclipseMonitor workflow diagram.It shows how the student's coding session data from Eclipse development environment has been collected through the EclipseMonitor.

Background and Context. Existing works in computing students' help-seeking and resource selection identified an expanding set of important dimensions that students consider when choosing a help resource. However, most works either assume a predefined list of help resources or focus on one specific help resource, while the landscape of help resources evolve at a faster speed.

BACKGROUND. This full research paper focuses on understanding research foci (what) and student participants (who) in primary and secondary (K-12) computer science (CS) education research (CER). With the growing interest in CS education at primary and secondary schools, there is a growing body of research. Unfortunately, there is limited analysis of research against the broader education ecosystem. Given the significant contributions in CER in the U.S. and the United Kingdom, a comparative analysis of research between the two regions can identify strengths and opportunities for each. RESEARCH QUESTIONS. Our research questions for this study were: When comparing using the CAPE framework, what similarities and differences across areas of study exist among research studies with student participants from the UK and the U.S. ?and What similarities and differences exist in reporting of participant demographics between studies with student participants from the UK and the U.S.? METHODOLOGY. We conducted a systematic mapping review of K-12 CS education research publications (2019–2022). We used the Capacity, Access, Participation, and Experience (CAPE) framework to group publications that studies similar topic areas. We then ran descriptive statistics (counts and percentages across number of codes and also number of publications) to identify commonalities and differences between the regions with respect to areas of focus and student participant demographics, FINDINGS AND IMPLICATIONS. Our results suggest that dominant-norming likely exists in our corpus of CER that needs mitigating. Studies with UK participants reported prior experi-ence and gender of participants significantly more, while those in the US were twice as likely to report participant race/ethnicity significantly more. Research focused on pedagogy, the topics of student engagement, and content knowledge were studied more frequently than other areas in both regions. There is limited research related to funding for computing education, school-based extracurricular activities, and social-familial influences, all of which are critical for understanding their impacts on students.

Promptly and properly addressing students' help requests during office hours is a critical challenge for large CS courses. With a large number of help requests, the queue gets longer and students have to endure long wait times. To address this problem, we try to quantify students' patience in the queue through survival analysis. Our results show that half of the students are willing to stay in the queue after waiting for 142.5 minutes. Moreover, we find that female students, morning requests, returning students, and requests about test failures are more likely to stay in the queue for a longer time.

While there are multiple standards bodies that define characteristics of high-quality, there are limited guidelines on conducting equity-enabling research, particularly in the context of high quality and in computing education. As part of an ACM ITiCSE Working Group in 2023, we engaged in a concept analysis and structured literature review to identify high-impact practices for conducting both high-quality and equity-enabling education research. As a result of this work, we produced a set of guidelines across each major phase of research that integrates characteristics of high-quality education research with those that are necessary for producing research that is designed to honor and meet the needs of various subgroups of learners. Special emphasis is given to the role that the researcher plays in shaping the research based upon how the researcher's lived experiences, perspectives, and training influences their work. During this special session, we will review each set of guidelines and engage attendees in reflection and discussion of them and how they can use the guidelines to enhance their education research.

To promote good coding practices, we need to understand what students do when they are on their own. In this research study, we explore students' testing behavior and response to persistent errors to better understand their coding patterns. We investigate how those patterns change when they struggle, and how help-seeking might influence their coding behaviors. We define struggle during coding as failing the same unit test case consecutively for more than four submission events, considering only unit test cases created by the instructors. To analyze the students' coding data, we use progress indicators, student test implementation indicators, and both student-generated and instructor-generated unit test results from each student submission event. In addition, we use office hours attendance records and amount of assignment-related posts created on the course forum. Results show that students tend not to follow test-driven development practices, even when explicitly directed to, and tend to create unit tests only to earn assignment credit rather than to guide their software development. Students also tend not to modify their own unit tests once they have earned the related credits, even when facing coding struggle; they tend to modify their unit tests only after they have been facing coding struggle for an extended number of submission events.

This experience report presents results from a quasi-experiment comparing course performance and student-reported survey constructs between two groups of students. One group took a Data Structures course with traditional, in-person modality. The second group took the same course with flexible, online modality. The work was motivated by the rapid adjustments computer science instructors made due to remote learning during the COVID-19 pandemic. In a response to these forced changes, this study was set up to investigate the differences between pre- and post- pandemic modalities. There are 212 students in the study, which took place in Fall 2021 at an R1, minority serving institution in the Midwestern United States. The study found that students in both groups performed similarly on common course components like projects, labs, homework, and a final Data Structures assessment. There were not significant differences in their self-reported ease of learning, enjoyment, belongingness, attitude, mindset, and self-efficacy. However, when taking into consideration gender, we found that women's performance was lower than men's in the traditional modality course. Women's performance in the flexible modality course was on par with men. Lastly, we present some feedback from students relating to assessments and modality.

With the rise in enrollment in computing programs, online learning offers the flexibility to scale up enrollment. To help students succeed in an online course, evidence-based engagement strategies need to be incorporated and evaluated for their effectiveness in increasing student engagement, which is a key determinant of student learning outcomes. This also addresses a concern with online learning, where students often feel detached and disengaged. In this innovative practice full paper, we describe our implementation of a study group strategy that we employed to increase students' engagement across different online formats of an upper-level computing course. Study groups are intended to allow students to discuss course concepts, foster social-emotional engagement, and provide positive learning experiences rather than being dependent on group members to complete a task or receive a grade. Although the strategy has been implemented in face-to-face computing courses, it has not been implemented and evaluated in different online formats of a computing course. The primary contribution of this study is the empirical evidence of the impact of study groups on students' learning outcome and engagement across the online formats of the course. The context of this study is an upper-level web-development course offered in different online formats, including online asynchronous in Spring 2020, online asynchronous and online hybrid in Fall 2020, and online synchronous in Spring 2021, where the study group strategy was employed. Three semesters of data collection were conducted using questionnaires administered at the beginning and end of the courses. Students learning outcomes and engagement were not significantly different between the different online course formats, allowing us to compare across the courses. Findings indicate that a study group strategy has a positive relationship with participation engagement in all of the online course formats. However, study groups may not directly influence learning outcomes. A study group strategy may improve learning outcomes in an online environment if the instructor facilitates the study group's work. This study has implications for utilizing study groups in different online formats of upper-level computing courses.

Software testing is a critical skill for computing students, but learning and practicing testing can be challenging, particularly for beginners. A recent study suggests that a lightweight testing checklist that contains testing strategies and tutorial information could assist students in writing quality tests. However, students expressed a desire for more support in knowing how to test the code/scenario. Moreover, the potential costs and benefits of the testing checklist are not yet examined in a classroom setting. To that end, we improved the checklist by integrating explicit testing strategies to it (ETS Checklist), which provide step-by-step guidance on how to transfer semantic information from instructions to the possible testing scenarios. In this paper, we report our experiences in designing explicit strategies in unit testing, as well as adapting the ETS Checklist as optional tool support in a CS1.5 course. With the quantitative and qualitative analysis of the survey responses and lab assignment submissions generated by students, we discuss students' engagement with the ETS Checklists. Our results suggest that students who used the checklist intervention had significantly higher quality in their student-authored test code, in terms of code coverage, compared to those who did not, especially for assignments earlier in the course. We also observed students' unawareness of their need for help in writing high-quality tests.

To investigate and identify promising practices in equitable K-12 computer science (CS) education, the capacity for education researchers to conduct this research must be rapidly built globally. Simultaneously, concerns have arisen over the last few years about the quality of research that is being conducted and the lack of equity-focused research.

Students can have widely varying experiences while working on CS2 coding projects. Challenging experiences can lead to lower motivation and less success in completing these assignments. In this paper, we identify the common struggles CS2 students face while working on course projects and examine whether or not there is evidence of improvement in these areas of struggle between projects. While previous work has been conducted on understanding the importance of self-regulated learning to student success, it has not been fully investigated in the scope of CS2 coursework. We share our observations on investigating student struggles while working on coding projects through their self-reported response to a project reflection form. We apply emergent coding to identify student struggles at three points during the course and compare them against student actions in the course, such as project start times and office hours participation, to identify if students were overcoming these struggles. Through our coding and analysis we have found that while a majority of students encounter struggles with time management and debugging of failing tests, students tend to emphasize wanting to improve their time management skills in future coding assignments.

Research Problem. The CAPE Framework has been used in multiple studies to situate capacity-building efforts within schools to offer equitable student access to and participation in K-12 computer science (CS) education. CAPE defines four major components of capacity, access, participation and experience. However, to define what each of the CAPE components can entail, well-defined subcomponents are needed.

The help-seeking interactions faculty encounter will vary depending upon the course structure and the students enrolled. While the course structure tends to remain the same, the students enrolled change each semester, presenting a new set of students who seek help in many different ways. We share our observations in investigating student behavior when using course resources, including office hours and online discussion forums, in two introductory computer science courses. Our goal is to explore differences in help-seeking behavior to construct Student "Help-Seeking" Personas. Preliminary analysis has shown that, for these two introductory CS courses, there are no well-defined personas that emerge from the grouping of help-seeking behaviors. The demographic of students exhibiting various help-seeking behaviors tends to be a near-proportionate subset of the overall course demographic. Thus, no distinct personas emerge from the students' help-seeking behaviors in introductory CS courses that faculty can utilize to better understand their students.

Motivation. There are countless factors that contribute to academic achievement [3, 6, 7, 8]. For K-12 computer science (CS) education, these building blocks of student success are broadly captured in the four components of the CAPE framework, which considers the capacity for, access to, participation in, and experiences of students learning CS through a lens of diversity, equity, and inclusion [4]. However, there is currently little published research that identifies factors within each component that influence the education ecosystem that students rely on to participate and have positive learning experiences in CS [10].

Teaming is a core component in practically all professional software engineering careers, and as such, is a key skill taught in many undergraduate Computer Science programs. However, not all teams manage to work together effectively, and in education, this can deprive some students of successful teaming experiences. In this work, we seek to gain insights into the characteristics of successful and unsuccessful undergraduate student teams in a software engineering course. We conduct semi-structured interviews with 18 students who have recently completed a team-based software engineering course to understand how they worked together, what challenges they faced, and how they tried to overcome these challenges. Our results show that common problems include communicating, setting and holding to deadlines, and effectively identifying tasks and their relative difficulty. Additionally, we find that self-reflection on what is working and not working or external motivators such as grades help some, but not all, teams overcome these challenges. Finally, we conclude with recommendations for educators on successful behaviours to steer teams towards, and recommendations for researchers on future work to better understand challenges that teams face.

The computing education community endeavors to consistently move forward, improving the educational experience of our students. As new innovations in computing education practice are learned and shared, however, these papers may not exhibit the desired qualities that move simple experience reports to true Scholarship of Teaching and Learning (SoTL). We report on our six years of experience in running professional development for computing educators in empirical research methods for social and behavioral studies in the classroom. Our goal is to have a direct impact on instructors who are in the beginning stages of transitioning their educational innovations from anecdotal to empirical results that can be replicated by instructors at other institutions. To achieve this, we created a year-long mentoring experience, beginning with a multi-day workshop on empirical research methods during the summer, followed by regular mentoring sessions with participants, and culminating in a follow-up session at the following year's SIGCSE Technical Symposium. From survey results and as evidenced by eventual research results and publications from participants, we believe that our method of structuring empirical research professional development was successful and could be a model for similar programs in other areas.

As enrollment in CS programs have risen, it has become increasingly difficult for teaching staff to provide timely and detailed guidance on student projects. To address this, instructors use automated assessment tools to evaluate students' code and processes as they work. Even with automation, understanding students' progress, and more importantly, if students are making the 'right' progress toward the solution is challenging at scale. To help students manage their time and learn good software engineering processes, instructors may create intermediate deadlines, or milestones, to support progress. However, student's adherence to these processes is opaque and may hinder student success and instructional support. Better understanding of how students follow process guidance in practice is needed to identify the right assignment structures to support development of high-quality process skills.

In Computer Science (CS) education, instructors use office hours for one-on-one help-seeking. Prior work has shown that traditional in-person office hours may be underutilized. In response many instructors are adding or transitioning to virtual office hours. Our research focuses on comparing in-person and online office hours to investigate differences between performance, interaction time, and the characteristics of the students who utilize in-person and virtual office hours. We analyze a rich dataset covering two semesters of a CS2 course which used in-person office hours in Fall 2019 and virtual office hours in Fall 2020. Our data covers students' use of office hours, the nature of their questions, and the time spent receiving help as well as demographic and attitude data. Our results show no relationship between student's attendance in office hours and class performance. However we found that female students attended office hours more frequently, as did students with a fixed mindset in computing, and those with weaker skills in transferring theory to practice. We also found that students with low confidence in or low enjoyment toward CS were more active in virtual office hours. Finally, we observed a significant correlation between students attending virtual office hours and an increased interest in CS study; while students attending in-person office hours tend to show an increase in their growth mindset.

Classroom dashboards are designed to help instructors effectively orchestrate classrooms by providing summary statistics, activity tracking, and other information. Existing dashboards are generally specific to an LMS or platform and they generally summarize individual work, not group behaviors. However, CS courses typically involve constellations of tools and mix on- and offline collaboration. Thus, cross-platform monitoring of individuals and teams is important to develop a full picture of the class. In this work, we describe our work on Concert, a data integration platform that collects data about student activities from several sources such as Piazza, My Digital Hand, and GitHub and uses it to support classroom monitoring through analysis and visualizations. We discuss team visualizations that we have developed to support effective group management and to help instructors identify teams in need of intervention.

Are you the director of undergraduate programs for your department? Or do you want to be one in the future? In many departments, the role of shepherding the undergraduate program is a "middle management" position held by a single person, with pressures and responsibilities unique to the role. This Birds of a Feather session will be an opportunity to build community and learn from one another: what does it mean to effectively lead an undergraduate computing program? what challenges do we face? and how can this role help position us for other professional opportunities?

Discussion forums are invaluable resources when scaling up undergraduate CS courses to larger class sizes. However, passive incorporation of discussion forums is not a silver bullet, as these platforms tend to devolve into places of shallow engagement. To aid our understanding of the factors that influence the nature of these interactions, we collected data from three CS1/CS2 forums. We obtained survey responses from the course instructors and performed a content analysis of the question-response pairs across all the courses. The results suggest that students' help-seeking patterns are influenced by the course curriculum, mode of delivery, and the existence of other help-seeking avenues. The findings also shed light on common strategies used by instructors to incentivize productive student-teaching staff and student-student interactions (e.g., instructing students to describe their debugging questions in detail, asking teaching staff to respond with hints/questions instead of direct answers). This poster presents a series of takeaways that can inform CS educators' choices around discussion forums.

Teaming is increasingly a core aspect of professional software engineering and most undergraduate computer science curricula. At NC State University, we teach communication and project-management skills explicitly through a junior-level software engineering course. However, some students may have a dysfunctional team experience that imperils their ability to learn these skills. Identifying these teams during a team project is important so the teaching staff can intervene early and hopefully alleviate the issues.

Context. Computing Education Research (CER) is critical to help the computing education community and policy makers support the increasing population of students who need to learn computing skills for future careers. For a community to systematically advance knowledge about a topic, the members must be able to understand published work thoroughly enough to perform replications, conduct meta-analyses, and build theories. There is a need to understand whether published research allows the CER community to systematically advance knowledge and build theories. Objectives. The goal of this study is to characterize the reporting of empiricism in Computing Education Research literature by identifying whether publications include content necessary for researchers to perform replications, meta-analyses, and theory building. We answer three research questions related to this goal: (RQ1) What percentage of papers in CER venues have some form of empirical evaluation? (RQ2) Of the papers that have empirical evaluation, what are the characteristics of the empirical evaluation? (RQ3) Of the papers that have empirical evaluation, do they follow norms (both for inclusion and for labeling of information needed for replication, meta-analysis, and, eventually, theory-building) for reporting empirical work? Methods. We conducted a systematic literature review of the 2014 and 2015 proceedings or issues of five CER venues: Technical Symposium on Computer Science Education (SIGCSE TS), International Symposium on Computing Education Research (ICER), Conference on Innovation and Technology in Computer Science Education (ITiCSE), ACM Transactions on Computing Education (TOCE), and Computer Science Education (CSE). We developed and applied the CER Empiricism Assessment Rubric to the 427 papers accepted and published at these venues over 2014 and 2015. Two people evaluated each paper using the Base Rubric for characterizing the paper. An individual person applied the other rubrics to characterize the norms of reporting, as appropriate for the paper type. Any discrepancies or questions were discussed between multiple reviewers to resolve. Results. We found that over 80% of papers accepted across all five venues had some form of empirical evaluation. Quantitative evaluation methods were the most frequently reported. Papers most frequently reported results on interventions around pedagogical techniques, curriculum, community, or tools. There was a split in papers that had some type of comparison between an intervention and some other dataset or baseline. Most papers reported related work, following the expectations for doing so in the SIGCSE and CER community. However, many papers were lacking properly reported research objectives, goals, research questions, or hypotheses; description of participants; study design; data collection; and threats to validity. These results align with prior surveys of the CER literature. Conclusions. CER authors are contributing empirical results to the literature; however, not all norms for reporting are met. We encourage authors to provide clear, labeled details about their work so readers can use the study methodologies and results for replications and meta-analyses. As our community grows, our reporting of CER should mature to help establish computing education theory to support the next generation of computing learners.

To address the increasing enrollment in Computer Science (CS), face-to-face (F2F) courses are often redesigned to online courses. For the success of a large-enrollment online CS class, it is important to design it with effective strategies that can enhance student learning and engagement. With this aim, we developed an online, third year web-development course, and redesigned its existing F2F version. This study focuses on examining the course strategies utilized in the two versions of of the course, and how students' learning and engagement vary between these two offerings. In Fall 2019, we conducted experimental research with pre- and post-tests to collect data from these two sections. The participants of the study include 51 out of 103 online students, and 38 out of 72 F2F students. We identified course strategies that were more valuable to one group versus the other, and both the sections. In terms of student learning, we found that there was no significant difference between these sections. Behavioral engagement measures depicted how the online and F2F students engaged within their courses, and how it is related to students' learning. We also found how the level of engagement varied in the online course according to the engagement factors that include performance, emotions, skills, and participation. The results of this study will inform online, F2F, and hybrid course design of upper-level CS courses.

Analytics about how students navigate online learning tools throughout the duration of an assignment is scarce. Knowledge about how students use online tools before a course's end could positively impact students' learning outcomes. We introduce PEDI (Piazza Explorer Dashboard for Intervention), a tool which analyzes and presents visualizations of forum activity on Piazza, a question and answer forum, to instructors. We outline the design principles and data-informed recommendations used to design PEDI. Our prior research revealed two critical periods in students' forum engagement over the duration of an assignment. Early engagement in the first half of an assignment duration positively correlates with class average performance. Whereas, extremely high engagement toward the deadline predicted lower class average performance. PEDI uses these findings to detect and flag troubling engagement levels and informs instructors through clear visualizations to promote data-informed interventions. By providing insights to instructors, PEDI may improve class performance and pave the way for a new generation of online tools.

One of the most important, and difficult, aspects of starting an education research project is identifying an interesting, answerable, repeatable, measurable, and appropriately scoped research question. The lack of a valid research question reduces the potential impact of the work and could result in wasted effort. The goal of this workshop is to help educational researchers get off on the right foot by defining such a research question. This workshop is part of the larger Designing Empirical Education Research Studies (DEERS) project, which consists of an ongoing series of workshops in which researcher cohorts work with experienced empirical researchers to design, implement, evaluate, and publish empirical work in computer science education. In addition to instruction on the various aspects of good research questions, DEERS alumni will join us to mentor attendees in development of their own research questions in small group breakout sessions. At the end of the workshop, attendees will leave with a valid research question that can then be the start for designing a research study. Attendees will also receive information on how to apply to attend the full summer workshop, where they can fully flesh out the empirical study design, and join a DEERS research cohort. More information about DEERS can be found at http://empiricalcsed.org.

Situated learning theory supports engaging students with materials and resources that reflect professional standards and best practices. Starting with our introductory courses, we incorporate situated learning to support student engagement in software engineering practices and processes through the use of industrial strength open-source tools in several classes throughout the undergraduate computer science curriculum at NC State University. Additionally, these tools support several logistical and educational needs in computer science classrooms, including assignment submission systems and automated grading. In this tools paper, we present our Canary Framework for supporting software engineering practices through the use of Eclipse for development; GitHub for submission and collaboration; and Jenkins for continuous integration and automated grading. These tools are used in five of ten core courses by more than 3000 students over ten semesters. While the use of these tools in education is not unique, we want to share our model of using professional tools in a classroom setting and our experiences on how this framework can support multiple courses throughout the curriculum and at scale.

Capstone and large projects in computing education are used as a vehicle for giving students as close to a “real-world” experience in software development as possible within the constraints of a computing degree program. This special issue presents four articles that focus on empirical research on capstone or other large-scale projects. These articles discuss areas such as project selection, working with external stakeholders, choosing the appropriate development methodology, incorporating creative activities to support student engagement, and learning.

One of the most important, and difficult, aspects of starting an education research project is identifying an interesting, answerable, repeatable, measurable, and appropriately scoped research question. The lack of a valid research question reduces the potential impact of the work and could result in wasted effort. The goal of this workshop is to help educational researchers get off on the right foot by defining such a research question. This workshop is part of the larger Designing Empirical Education Research Studies (DEERS) project, which consists of an ongoing series of workshops in which researcher cohorts work with experienced empirical researchers to design, implement, evaluate, and publish empirical work in computer science education. In addition to instruction on the various aspects of good research questions, DEERS alumni will join us to mentor attendees in development of their own research questions in small group breakout sessions. At the end of the workshop, attendees will leave with a valid research question that can then be the start for designing a research study. Attendees will also receive information on how to apply to attend the full summer workshop, where they can fully flesh out the empirical study design, and join a DEERS research cohort. More information about DEERS can be found at http://empiricalcsed.org.

Increased enrollments in computer science programs presents a new challenge of quickly accommodating higher enrollment in computer science introductory courses. Because peer teaching scales with enrollment size, it is a promising solution for supporting computer science students in this setting. However, pedagogical and logistical challenges can arise when implementing a large peer teaching program. To study these challenges, we developed a transparent online tool, My Digital Hand, for tracking one-to-one peer teaching interactions. We deployed the tool across three universities in large CS2 computer science courses. The data gathered confirms the pedagogical and logistical challenges that exist at scale and gives insight into ways we might address them. Using this information, we developed the second iteration of My Digital Hand to better support peer teaching. This paper presents the modified tool for use by the computer science education community.

Online discussion forums are widely used and hold great promise for supporting students in learning computer science. Understanding how we can best support students in learning computer science through online discussion forums is an important open question for the CS Ed community. This paper analyzes discussion forum posts from 395 students enrolled in CS2 across two different universities. The results demonstrate that students use the discussion forums often for logistical and relatively shallow questions. However, the largest portion of questions reflect some level of constructive problem-solving activity, and are positively correlated with course grades. Questions that neither describe students' reasoning nor their attempts to solve the problem constitute the smallest percentage of questions, but these questions may be particularly important to attend to because of their relationship to students' prior experience.

The computer science education (CSEd) research community consists of a large group of passionate CS educators who often contribute to other disciplines of CS research. There has been a trend in other disciplines toward more rigorous and empirical evaluation of various hypotheses. Prior investigations of the then-current state of CSEd research showed a distinct lack of rigor in the top research publication venues, with most papers falling in the general category of experience reports. In this paper, we present our examination of the two most recent proceedings of the SIGCSE Technical Symposium, providing a snapshot of the current state of empiricism at the largest CSEd venue. Our goal to categorize the current state of empiricism in the SIGCSE Technical Symposium and identify where the community might benefit from increased empiricism when conducting CSEd research. We found an increase in empirical validation of CSEd research to over 70%; however, our findings suggest that current CSEd research minimizes replication precluding meta-analysis and theory building.

Students learn best in environments where they can meaningfully engage with materials that emulate real-world scenarios. Incorporating software engineering best practices and supporting tools in introductory courses provides students the opportunity to engage in course materials as a novice member of the profession. We support student engagement with industry tools to support software engineering best practices for tutorials, in-class labs, and programming projects. The goal of the research is to improve student learning, engagement in the course and profession, and retention through the use of software engineering practices and tools that introduce students to the software engineering profession. A prior study on the incorporation of in-class laboratories, supported with software engineering best practices, on linear data structures showed an increase in engagement, but did not show a difference on student learning when compared with active learning lectures. We are currently expanding the study by incorporating in-class laboratories across a full semester of a CS1.5 class at NC State University. The poster presents the preliminary results from Fall 2015.

Program analysis tools use notifications to communicate with developers, but previous research suggests that developers encounter challenges that impede this communication. This paper describes a qualitative study that identifies 10 kinds of challenges that cause notifications to miscommunicate with developers. Our resulting notification communication theory reveals that many challenges span multiple tools and multiple levels of developer experience. Our results suggest that, for example, future tools that model developer experience could improve communication and help developers build more accurate mental models.

Active learning increases student learning through collaborative engagement with materials during class time. A CS1.5 course at NC State, CSC216, uses active learning lectures involving short simplified think-pair-share in-class exercises to engage students with course materials. However, students still struggle with the course materials and several students do not successfully complete the course on their first attempt. To increase student learning and engagement, we conducted a quasi-experimental study incorporating in-class labs into two sections of CSC216 during the linear data structures unit in the Fall 2014 semester. Both sections completed in-class labs on the Java Collections Framework and iterators. One section completed in-class labs on array-based lists; the other section completed in-class labs on linked lists, in a counter-balanced study design. The active learning lecture delivery was used for the control section and an Exam was administered between the array-based list and linked list topics. Overall, we found no significant difference in student learning on array-based and linked lists as measured by the final exam. Students displayed half as much disengaged behavior during in-class labs and were five times more likely to ask for help from the teaching staff during in-class labs.

The Computer Science and Software Engineering (CS/SE) profession reports that new college graduates lack the communication skills needed for personal and organizational success. Many CS/SE faculty may omit communication instruction from their courses because they do not want to reduce technical content. We experimented in a software-engineering-intensive second-semester programming course with strategies for improving students' writing of black box test plans that included no instruction on writing the plans beyond the standard lecture on testing. The treatment version of the course used 1) a modified assignment that focused on the plan's readers, 2) a model plan students could consult online, and 3) a modified grading rubric that identified the readers' needs. Three external raters found that students in the treatment sections outperformed students in the control sections on writing for five of nine criteria on rubrics for evaluating the plans and on the raters' holistic impression of the students' technical and communication abilities from the perspectives of a manager and a tester.

Ideally, software engineering courses should adequately reflect real-world software development so that students obtain a better understanding and experience with practices and techniques used in industry. Our objective is to improve software engineering courses by incorporating best practices for automated software engineering and facilitating rapid feedback for students using an open source continuous integration framework for evaluating student software development. The open source Jenkins Continuous Integration Server is the core of our framework, which provides a consistent environment for building student projects, executing automated test cases, calculating code coverage, executing static analysis, and generating reports for students. By using continuous integration, a common tool in real-world software development, we can incorporate software engineering best practices, introduce students to continuous integration in practice, and provide formative feedback to students throughout the software development lifecycle. We found that 76% or more of students in each of the classes that deploy our framework reported that using Jenkins increased their productivity, and that 84% or more of students in each of the classes reported that using Jenkins increased their code quality.

Over the past several years, there has been a marked increase in empirical validation in research papers presented at the SIGCSE Symposium. This birds-of-a-feather session intends to bring together faculty interested in empirical research in computer science education to share methodologies and techniques, while also continuing the conversation about how to move empirical computer science education research forward. The computer science education (CSEd) research community consists of a large group of passionate CS educators who often contribute to other disciplines of CS research. There has been a trend in other disciplines toward more rigorous and empirical evaluation of various hypotheses. Prior meta-analyses of the then-current state of CSEd research showed a distinct lack of rigor in the top CSEd publication venues, with most papers falling in the general category of experience reports. Our goal is to engage the SIGCSE community in a discussion as to how we can move empirical research in CSEd forward. We hope to help researchers establish connections that will allow them to share research methodologies and challenges.

Even though different developers have varying levels of expertise, the tools in one developer's integrated development environment (IDE) behave the same as the tools in every other developers' IDE. In this paper, we propose the idea of automatically customizing development tools by modeling what a developer knows about software concepts. We then sketch three such ``bespoke'' tools and describe how development data can be used to infer what a developer knows about relevant concepts. Finally, we describe our ongoing efforts to make bespoke program analysis tools that customize their notifications to the developer using them.

Over the past two years, second-year Java and software engineering courses have been taught at the University of Virginia and North Carolina State University utilizing the Android OS platform. Instructors taught a variety of traditional second-year topics, including abstraction, design, requirements, and testing, utilizing a variety of Android-based mobile devices. Anecdotal responses from student surveys and evaluations from five course sessions indicate that teaching lower-level courses with more advanced and current technology, even with a steeper learning curve, is beneficial. In this tutorial proposal, we outline our plan for presenting a session that would help educators incorporate the Android OS into their curriculum and how to use the system even if mobile devices are not available.

Automated static analysis can identify potential source code anomalies early in the software process that could lead to field failures. However, only a small portion of static analysis alerts may be important to the developer (actionable). The remainder are false positives (unactionable). We propose a process for building false positive mitigation models to classify static analysis alerts as actionable or unactionable using machine learning techniques. For two open source projects, we identify sets of alert characteristics predictive of actionable and unactionable alerts out of 51 candidate characteristics. From these selected characteristics, we evaluate 15 machine learning algorithms, which build models to classify alerts. We were able to obtain 88-97% average accuracy for both projects in classifying alerts using three to 14 alert characteristics. Additionally, the set of selected alert characteristics and best models differed between the two projects, suggesting that false positive mitigation models should be project-specific.

Benchmarks provide an experimental basis for evaluating software engineering processes or techniques in an objective and repeatable manner. We present the FAULTBENCH v0.1 benchmark, as a contribution to current benchmark materials, for evaluation and comparison of techniques that prioritize and classify alerts generated by static analysis tools. Static analysis tools may generate an overwhelming number of alerts, the majority of which are likely to be false positives (FP). Two FP mitigation techniques, alert prioritization and classification, provide an ordering or classification of alerts, identifying those likely to be anomalies. We evaluate FAULTBENCH using three versions of a FP mitigation technique within the AWARE adaptive prioritization model. Individual FAULTBENCH subjects vary in their optimal FP mitigation techniques. Together, FAULTBENCH subjects provide a precise and general evaluation of FP mitigation techniques.

In this paper, we propose a technique for leveraging historical field failure records in conjunction with automated static analysis alerts to determine which alerts or sets of alerts are predictive of a field failure. Our technique uses singular value decomposition to generate groupings of static analysis alert types, which we call alert signatures, that have been historically linked to field failure-prone files in previous releases of a software system. The signatures can be applied to sets of alerts from a current build of a software system. Files that have a matching alert signature are identified as having similar static analysis alert characteristics to files with known field failures in a previous release of the system. We performed a case study involving an industrial software system at IBM and found three distinct alert signatures that could be applied to the system. We found that 50% of the field failures reported since the last static analysis run could be discovered by examining the 10% of the files and static analysis alerts indicated by these three alert signatures. The remaining failures were either not detected by a signature which could be an indication of a new type of error in the field, or they were on areas of the code where no static analysis alerts were detected.