The call for alternative grading practices has been made both inside and outside the computing education community. Various practices exist to provide assessment and feedback to students that do not rely strictly on points out of one hundred percent, weighted averages, high stakes assignments, and grading for behaviors instead of learning. However, modern classrooms, especially computer science classrooms, rely on a myriad of digital tools to organize and maintain the course structure. Tools like learning management systems, automatic grading systems, submission systems, and practice systems all exist for computing students and faculty to use to help support the learning of programming concepts. By and large, these systems all rely on an underlying mechanism of points and aggregating points for scoring. In the face of such technological choices, adopting alternative grading practices can prove challenging for instructors and confusing for students. In this position paper, we advocate addressing key research problems to make these systems easier to use with alternative grading practices. These include comprehensive support for categorical grading, comprehensive support for rework and resubmission, and improved protocols for communication of scores and feedback. We propose an extension to LTI to support the needs of alternative grading practices, and we provide an initial design for this LTI extension. We discuss current problems and potential solutions and challenge the community to work on these problems and consider the design of future systems to embrace grading approaches that go beyond just points-based scoring.

Alternative grading practices seek to improve learning and make education more equitable. For the past few years, our community has been exploring these techniques and contextualizing them for use in computing courses. In this panel, we will present various grading techniques with a particular attention to the experience implementing them and pitfalls to be avoided. The presenters all have extensive experience exploring alternative grading techniques. Furthermore, we hope to engage the audience in a conversation about the nuance of implementing these particular techniques in the variety of courses that are typical in computing programs.

The field of computer science has a problem of representation–many groups are not represented in our classroom at levels approaching their composition in society. Unfortunately, the representation issue is a larger societal issue and begins well before students enter our institutions. Though we acknowledge that building inclusive and learner-centered classroom environments cannot increase representation by itself, it can have an impact on retention and inclusion for members of marginalized communities.

Often reduced to a result of personal choice, women have faced historical exclusion, a lack of representation, and implicit bias in computer science/technology fields. The visibility and legitimacy of women in computer science is hindered by the shifts in perception, lack of acknowledgement in education, and media portrayals. Particularly, giving legitimacy to myths of biological determinism results in harmful narratives in the aptitude of women and their belonging. This poster presents preliminary findings in determining the efficacy of an approach being deployed in some ethics courses in one computer science department. While the department has initiatives in place to foster an inclusive and equitable learning environment, this research may identify areas where the curriculum or discussions fall short in reshaping the narrative surrounding the representation of women in computer science.

The field of computer science has a problem of representation-many groups are not represented in our classroom at levels approaching their composition in society. Unfortunately, the representation issue is a larger societal issue and begins well before students enter our institutions. Though we acknowledge that building inclusive and equitable classroom environments cannot increase representation by itself, it can have an impact on retention and inclusion for members of marginalized communities.

The session will present the final report of an NSF sponsored workshop tasked with creating a vision for computing education for the next 15 years. Within several broad themes identified, the panelists were asked to probe further into where the field should be heading and what difficult questions we need to tackle. We will use the final report as a basis for discussion with the attendees and ask them the same hard questions we asked ourselves when writing this report to see how these questions shape their views of where computing education needs to be in the next 15 years.

This innovative practice paper describes our experiences with alternative grading practices in introductory computing courses and two large public universities in the United States. Computing classrooms often use traditional grading practices involving allocating points to assignments, deducting points for mistakes and tardiness, and combining assignment scores using a weighted average to determine grades. Recent research suggests that these practices may diminish achievement, discourage students, and suppress effort to such an extent that they are considered by some as detrimental. We approach our work as an exploratory case study, without predefined research questions or hypotheses. Our experiences began with the adoption of specifications grading. We outline the grading scheme applied to traditional programming assignments and exams/quizzes, and discuss the initial integration of these schemes with conventional auto-grading tools. We delve into student perceptions of alternative grading, their utilization of flexible deadlines, and resubmission opportunities. We conclude with a discussion of two challenges encountered during our exploration: student acceptance of a novel grading form, and the adaptation of tools designed for traditional grading to support alternative grading mechanisms. Our early exploration aims to inspire further research on the use of alternative grading in computing. It is clear from our observations that simply implementing the practices does not ensure the equitable and inclusive outcomes that can be achieved with these practices. If students are not prepared to use these practices, they find them difficult to understand and can feel that they are not being treated fairly. Additionally, we wish to foster a community of practice to assist faculty members exploring these changes, with the goal of creating more equitable and inclusive classrooms.

This innovative practice paper explores two differing approaches to the use of artificial intelligence tools in computer science classrooms. Artificial Intelligence (AI), while not a new technology, has seen a rise in popularity over the past two years, with companies such as OpenAI, Google, and Microsoft making readily available generative AI tools for anyone to use. This surge in AI popularity has led to a rise in its use in educational settings, in some cases allowed and in others discouraged or disallowed. A debate has risen among academics, particularly in higher education, about what AI's place in education is, with some educators actively encouraging its use while others view the use of AI as a form of academic dishonesty. Given AI is only advancing and is becoming more prevalent, it will only continue to become a more dominant force throughout education and the world at large. This paper presents two educators' distinct viewpoints and experiences on how AI should be handled in computer science courses (absolutely forbidden vs. decriminalized). The goal of this paper is to present different perspectives as well as concrete experiences we have had with AI in our own classrooms to encourage others to consider their own positions on its use and its implications for their own learning environments. While the debate on the place of AI in education is a long way from being settled, educators need to think about making choices, clearly articulating policies, and evaluating the positives and negatives of positions about AI in their classrooms.

WIP Research Paper: Identifying the Threshold Concepts within a discipline illuminates the key concepts or components within the curriculum. Once students have overcome the barrier of learning these concepts, they often will be better able to identify as members of that community, and understanding a threshold concept opens the door to learning additional concepts. Within computer science, there has been much debate over what concepts could potentially be threshold concepts. Meyer and Land originally defined threshold concepts as resulting in an individual being placed into a state of uncertainty or liminality, and successfully traversing this liminal state results in a transformation of the individual with potential feelings of accomplishment. While there has been some work attempting to identify threshold concepts within the first year or beginning stages of programming, little work has considered the intermediate years (years 2 and 3) of university study and what potential threshold concepts exist during this time period. Our goal with this work is to help address this gap that exists by answering the following research question: What do intermediate students identify as being troublesome and/or ‘uncomfortable to learn’ within their computer science coursework?” A first cohort of participants were interviewed in late 2022 and coding began in the first quarter of 2023. The coding of these interviews proved challenging. The students who were interviewed often did not give enough information about a concept for the coders to identify whether the concepts had the key characteristics of threshold concepts. It was considered and accepted that the original interview protocol was not supporting the participants well in eliciting the types of information needed to identify a concept as threshold. The interview protocol was redesigned, and new interviews commenced. The work presented here is a continued discussion of the initial findings and the subsequent change in interview protocol, with the primary improvement being an inclusion of concept mapping. Concept maps, or a graphical representation of the interrelationship of topics and ideas, coupled with an intentional simplification of associated terminology, are expected to reduce cognitive load as participants reflect on their learning experiences. Interviews with the revised protocol including concept maps have been more engaging and productive in identifying potential threshold concepts within the intermediate computer science curricula.

It is often the case that computer science classrooms use traditional grading practices where points are allocated to assignments, mistakes result in point deductions, and assignment scores are combined using some form of weighted averaging to determine grades. Unfortunately, traditional grading practices have been shown to reduce achievement, discourage students, and suppress effort to such an extent that some common elements of traditional grading practices have been termed toxic. Using grades to reward or punish student behavior does not encourage learning and instead increases anxiety and stress. These toxic elements are present throughout computing education and computer science classrooms in the form of late penalties, lack of credit for code that doesn't compile or pass certain unit tests, among others. These types of metrics, that evaluate behavior are often influenced by implicit bias, factors outside of the classrooms (e.g., part-time employment), and family life situations (e.g., students who are caregivers). Often, students in these situations are disproportionately from low-socioeconomic backgrounds and predominantly students of color. Through this paper, we will present a case for adoption of equitable grading practices and a call for additional support in classroom and teaching technologies as well as support from administrations both at the department and university level. By adopting a community of practice approach, we argue that we can support new faculty making these changes, which would be more equitable and inclusive. Further, these practices have been shown to better support student learning and can help increase student learning gains and retention.

Recently, a workshop was convened with the goal of creating a vision for computing education for the next 15 years. The first part of the workshop involved 43 members of the computing education community meeting virtually to enumerate themes within the discipline. Those themes included: diversity, equity, inclusion, ethics, broadening participation, teaching, learning, K-12, research to practice, computing's connection to other fields, and computing education research disciplinary issues. Within each of those areas, interesting and provocative questions emerged that led the organizers to dig deeper into some hard questions for the field and probe further into where the field should be heading and what difficult questions we need to tackle. Thus, a final report that probes the issues of curricular bloat, teaching and pedagogy, infusing humanities in computing, improving pathways into the discipline, and integrating social justice into the discipline has been created. This talk will introduce the main themes of the report and point the audience towards where it is available. This material is based upon work supported by the National Science Foundation under Grant No. 2039833 and 2039848. The report and other resources can be found at https://cerfutureworkshop.org.

The field of computer science has a problem of representation-many groups are not represented in our classroom at levels approaching their composition in society. Unfortunately, the representation issue is a larger societal issue and begins well before students enter our institutions. Though we acknowledge that building inclusive and equitable classroom environments cannot increase representation by itself, it can have an impact on retention and inclusion for members of marginalized communities.

Since the 1960s, the ACM has provided routinely updated guidelines for what concepts constitute a computer science curriculum, with the latest version currently in development in 2023. These guidelines have traditionally provided a model curriculum from which universities can choose to adopt or modify for their own purposes. What is unclear, however, is to what degree schools follow the curriculum recommendations that the ACM provides. While most faculty and students likely have knowledge of their own institution's curriculum, as well as what courses are offered at a small selection of other schools, the goal of the work presented in this poster is to distill a cohesive view of what computer science curriculums in their second and third years look like across a broad range of universities across a range of institutions. Our goal with this work was to answer the following question: What do computer science course requirements look like at a wide range of different institutions? We believe the work will help those who are trying to develop curriculum changes within their own institutions and aims to provide a more cohesive view of what trends and patterns exist in course offerings and degree requirements for computer science in the second and third years across a wide range of universities.

Problem. Understanding computer science (CS) teacher CS knowledge primarily relies on self-reported data from participants to understand the learning impact on teachers and improve teacher growth. There is currently a lack of quality instruments to determine where CS teachers need to improve their knowledge.

WIP Research Paper: The ability to identify threshold concepts within a discipline recognizes key components within a curriculum which, when learned, enables an individual to demonstrate as a member of that community. Within computer science, potential threshold concepts have sparked debates among researchers. According to the original definition by Meyer and Land, threshold concepts result in an individual being placed into a state of uncertainty or liminality and successful traversal of this liminal state results in a potentially irreversible transformation. When seeking to identify threshold concept, researchers often search for what concepts students feel are ‘troublesome’ or ‘difficult to learn’ since this state is often difficult to describe or understand when currently inside it or just past it. While threshold concepts have been identified for the beginning stages of programming, there is little to no work on the intermediate years of university computing education (years 2 & 3) and what potential threshold concepts exist during that time for computer science students. Our goal with this research is to help address this gap by answering the following research question: What concepts do intermediate students identify as being troublesome and/or ‘uncomfortable to learn’ within their computer science coursework?

Problem. This full research paper describes the results of a literature review of data collected about K-12 computer science education research and initiatives. Over the last ten years, K-12 computer science (CS) education research has evolved to meet the needs and progress of the computer science education community. As a field, however, we have no empirical evidence of what this evolution is and how it has managed to produce the empirical evidence needed to support the long-term goals of computing education research as computing education grows into every primary and secondary classroom. Research Question. Our question for this study was How has K-12 CS education research evolved over the last decade, including when examining the research based on standards and inclusion of student participants? Methodology. Using a publicly available database of curated articles documenting K-12 computing education research efforts, queries were run to extract pre-specified subsets of data. Descriptive statistics were calculated to identify trends to support answers to the research question. Findings. We consider how this data reflects on changes in the last decade in light of the CS for All initiatives, including the fact that less than 5% of studies include students with disabilities and less than 20% of studies report participants' socioeconomic status. With the growth of computing in schools in the last decade, it is important to consider previous computing experience of students when analyzing results, but our data shows that only 40% of studies do. Implications. The good news is that this increase means that more topics, contexts, and student subgroups can be explored each year, and new insights on best practices are likely to arise from these findings. The bad news is that the attention to the needs of historically marginalized groups is not increasing at the same pace. While our data shows modest to gradual increases in the percent of research articles that report on factors such as the student race and instructor gender, these factors are still rarely reported, which may make it more difficult to ensure that CS education research involves a wide diversity of participants.

In our previous research we found that teaching novice programmers introductory programming in Java using subgoal labels led to deeper knowledge [2] and increased persistence for students potentially at risk of dropping out or failing their first undergraduate course in CS [3]. Subgoals are an instructional tool that is designed to bridge the gap between novices and experts, i.e., students and instructors. Experts often have difficulty explaining concepts at a level that novices understand because they have automatized much low-level knowledge. The task analysis used to identify subgoals makes this knowledge explicit. Subgoals are often expressed to students through subgoal-labeled worked examples that explicitly state the conceptual knowledge expressed through examples. This instructional design of examples allows students to see past superficial details of the example to the structural problem-solving procedure being exemplified [3].

What additional supports do instructors in Computer Science programs need to effectively teach neurodiverse students? Many neurodiverse students (e.g., with ADHD, autism) are enrolled in CS programs, but few graduate. These students often struggle to communicate their needs to instructors, who themselves are often untrained to support neurodiverse students' needs. This can result in instructors unintentionally using practices that are detrimental to student success, lowering students' sense of belonging in CS. Likewise, this can make teaching frustrating for instructors who are unprepared for students' needs. This BoF will provide a platform for instructors and neurodiverse graduates from CS programs to discuss major topics relating to instructor and student needs, such as: experiences working with neurodiverse students, successful strategies, and instructors' professional development needs. Instructors attending will learn more about other instructors' experiences and teaching practices, as well as an NSF-funded participatory design study on training instructors to improve neurodiverse students' retention in undergraduate CS. Participants will be invited to stay up-to-date on the project's progress after the conference, with free access to tools and curriculum as they are developed. Notes and recordings from this discussion will be included as data in our study to support future instructors.

Subgoal labeling is an instructional design framework for breaking down problems into pieces that are small enough for novices to grasp, and often difficult for instructors (i.e., experts) to articulate. Subgoal labels have been shown to improve student performance during problem solving in many disciplines, including computing. Improved student performance occurs because subgoal labels improve student transfer and retention of knowledge. With support from NSF (DUE-1712025, 1712231, 1927906, 2110156, 2111578), subgoal labels have been identified and integrated into a CS1 course (variables, expressions, conditionals, loops, arrays, classes) and an e-book has been created on the Runestone platform to enable students to complete practice problems using the subgoals. This workshop will introduce participants to the materials and demonstrate how the subgoal labels and worked examples are integrated throughout the course. Materials include nearly 50 worked examples and 300 practice problems that increase in complexity and difficulty within each topic. The materials are designed to be integrated into CS1 courses as homework or classroom examples and activities. Assessment of topics using subgoal labels will also be discussed. Participants will leave with access to the e-book containing worked examples and practice problems for common topics in an imperative Java-based CS1 and will also engage in an activity where they create an example for their own course using subgoal labels.

The session will present an overview of findings of a recently funded NSF workshop that set out to examine the pressing issues for computing education research for the next 15 years. Based on dialogs for scholars working in this area, the workshop participants developed a series of themes and topics that they felt should become the focus of computing education research efforts for the next 15 years. Main themes that emerged and will be discussed in this session include: diversity, equity, inclusion, ethics, broadening participation, teaching, learning, K-12, research to practice, computing's connection to other fields, and computing education research disciplinary issues. The session will focus on interesting research questions from each of these areas that are ripe to be explored as well as enablers and blockers to the progress of this work.

In our CS1 course, we found that students were failing at a rate similar to that reported in literature. In an effort to improve this DFW rate, we made a few changes to the course. While the changes led to some improvement, the results were not as significant as we expected. We found that in our CS1 course, first-semester students were passing at a significantly higher rate than non-first-semester students. Upon looking at other STEM and non-STEM courses at the university, we noted that the institution's first term calculus and introductory English courses had similar patterns, however the introductory sociology course did not. Further data analyses challenged several other of our beliefs about who was failing the course. Our findings leave us with several questions. Is this pattern specific to our university? Is this a particular course problem? From the perspective of the discipline, is there something in this data that points to a set of problems for computing education researchers to consider? This poster serves as both a presentation of our initial experiential results and an attempt to start a conversation amongst community members around whom actually is struggling within our early CS courses.

This lightning talk describes our current efforts to isolate and identify potential threshold concepts and threshold capabilities which are experienced by intermediate computer science undergraduate students. We are utilizing the theoretical framework as described by Meyer, Land, Bowden, and Baillie. Two of the key identifiable characteristics surrounding a threshold concept are (1) the transformation one undergoes as a result of incorporating the concept into their knowledge and (2) the state of uncertainty (liminality) experienced as the learner develops this understanding. As a result of this transformation and the resultant traversal of a liminal state, elements of an identity are often developed. For the computer science undergraduate, we hypothesize that advanced computational thinking skills are developed along with these transformations leading to the development of a computer scientist identity. Through identification of the threshold concepts/capabilities experienced by intermediate students the potential corollary concepts behind computational thinking may be uncovered as well. The overall project intends to integrate elements of threshold concepts, liminality, and transformative learning to produce a framework through which computational thinking can be probed and students' progression can be better assessed and supported. These efforts are only beginning - curious faculty and students are invited to attend our presentation and consider potential collaborative participation in our endeavors.

Each year, we graduate a fair number of students with college degrees in computing-related disciplines. Each year, we lose a fair number of students out of the discipline. We spend a good deal of time focusing on what is causing students to leave or fail. We do not spend a lot of time focusing on the learning journey and supporting students throughout their transformative experiences. Threshold concepts are those concepts or learning experiences which define the overall disciplinary learning journey, enabling the learner to see from a new perspective, participate in discourse previously unavailable to them, and engage with the world from a transformed frame of understanding. They are often those concepts difficult for students to understand within a discipline such as "personhood' in Philosophy, 'gravity' in Physics, or 'limit' in Mathematics. As a result of this transformation and the resultant traversal of a liminal state, elements of an identity developed. In this poster, we present our current work identifying potential threshold concepts experienced by intermediate computer science students and consider the impact of these concepts on their development of an identity within computer science.

Changing the language used to teach the AP Computer Science A course is an expensive, time-consuming, and ultimately controversial endeavor. However, it is worth raising the question from time to time to determine if such a change may now be appropriate. This panel offers a set of speakers specifically chosen to provide a diverse set of viewpoints and experiences (high school teacher, curriculum developer, exam reader, exam development committee member, university credit-granting administrator) on the question of whether the language in the AP CS A exam should be switched to Python in the near future.

One of the most ubiquitous techniques for evaluating research, particularly in educational settings, has been Likert-style questionnaires. Having a participant rank their level of engagement, agreement, or interest on a scale can provide powerful insight into an individual's perspective and attitude. However, as computing education matures as a discipline it becomes important for us to examine our practices and ensure that we are employing the techniques of evaluation properly. Likert-style questionnaires can be prone to unintentional biases and noise which, from the perspective of the researcher, may affect the study in unknown, unexpected, and potentially undesirable ways. In this research we seek to aid the computing education researcher not only avoid biases and noise but also improve the reproducibility of their work. First, we establish best practices for these instruments by synthesizing recommendations from the original creator, Rensis Likert, the Center for Disease Control (CDC), the Association of American Medical Colleges (AAMC), and the American Association for Public Opinion Research (AAPOR). We then considered additional sources of unintentional biasing and noise resulting from the measurement scales/response options, specifically possible biasing resulting from how response options are presented to the study participant. With these recommendations, we then examined 121 evaluation instruments used in computer science education research and curated in the csedresearch.org online database to see how often researchers unintentionally fall victim to the pitfalls of ambiguity, awkward phrasing, use of conjunctions, leading or biased statements, and double negatives. We found that the occurrence of at least one of these problematic statements to be in 82.6% of all instruments. We also examine demographic information for the intended study participants, number of response options, and how those options are presented. Overall, while we see many instrument authors falling victim to a few of these common pitfalls, we believe that increased awareness of potentially problematic statements/measurement scales and their impact on research bias and reproducibility will help insulate computing education researchers from avoidable complications and strengthen the discipline throughout.

This WIP Research paper is a follow-up to a study conducted in 2013 by McGill, Decker, and Settle that investigated the effects of pre-college computing experiences on students' decisions to study computer science at university. Their results indicated that the exposure to pre-college computing activities impacted students in different ways, particularly when looking at perceived impact by men and women participants [1], [2]. After six years and a myriad of changes to the K-12 computing landscape, it was time to see if impacts to current undergraduates were different than previously observed. This current study is a pilot qualitative study in which we interviewed seven undergraduate students about their experiences with computing prior to college. We were particularly interested in finding out about the nature of their computing experiences, whether they enjoyed them, and what they would change to make such experiences better for future participants. We used grounded theory and thematic coding to encode the interview transcripts, enabling us to look for common themes among the interview subjects. We are looking for elements in the interviews that could point to key differences in the pre-college computing landscape that have impacted student experiences that are different from those previously observed and thus impacting their experiences in university with computing. The goal of this preliminary study was to get a sense of what aspects of students' exposure to pre-college computing experiences have changed since 2013 and what changes should be made when creating a follow-up to this initial study.

This full research paper considers the resources needed to meet the research and evaluation needs of the many efforts to incorporate computing education throughout primary and secondary schools. In order to support these efforts, we developed csedresearch.org, a site designed to serve as a resource center for primary and secondary computing education research. We first considered criteria and recommendations for resource centers previously established by others. We provide a general description of the purpose of the csedresearch.org as well as a description of how we used quantitative and qualitative methods over numerous phases of development (pre-concept/research, concept, alpha, and beta, launch) to ensure that it meets the needs of potential users. We discuss how the current product compares against the general criteria for resource centers, its original intentions and expectations, and how it has fared through the phases of development one normally expects from a digital resource center. As the resource center evolves, we continue to seek feedback and resources to further meet the needs of the community. We also discuss the data that is being collected and could be collected to further benefit the community to define not only what educational practices work best overall, but what works best for particular demographic groups, including underrepresented groups in computing. And finally, we are faced with the challenge for maintaining the information so that remains robust and current.

The sheer number of tools, languages, and environments (TLEs) used in computing education has proliferated in the last few years as more tools are developed to meet new demands of the growing amount of K-12 computing education that has been undertaken. However, there is little formalized language at either the K-12 or post-secondary level that provides for a way to classify these TLEs for discussing research and for classifying in databases.

With the advent of teaching primary and secondary computing education, tools, languages, and environments (TLEs) are important pedagogical support systems for students and teachers. While there are a number of resources available for teaching K-12 students and teachers, there is little synthesis of the data with respect to usage and adoption rates for various TLEs. Using data extracted from 510 articles related to K-12 education, we conducted an analysis using descriptive statistics to determine what TLEs in K-12 are most frequently studied by researchers. We found 193 TLEs being used in research studies and experience reports, then differentiate between these two types of data and between students and teacher professional development. This preliminary research provides a first descriptive analysis of TLEs being used in K-12 space and simultaneously sets the stage for creating a classification system for TLEs based on the literature, including how they are used and what topics (in or outside of computing education) they are used to teach.

The quality of reporting of experimental results in computing education literature has been previously shown to be less than rigorous. In this study, we first examined research standards set forth by four organizations: American Psychology Association (APA), American Educational Research Association (AERA), What Works Clearinghouse (WWC), and the CONsolidated Standards of Reporting Trials (CONSORT). We selected the most important data standards based on their prominence across all four and the most typical study designs in computing education research. We then examined 76 articles designated as quantitative research studies (K-12) published in ten venues (2012-2018) to determine whether the reporting in these articles met these five standards. Findings indicate that only 48% of these articles report effect size and even fewer (11%) report confidence intervals and levels. We found that reported data did not meet the standard that data should be "reported in a way that the reader could construct effect-size estimates and confidence intervals beyond those supplied in the paper". Additionally, authors used existing instruments less than a quarter of the time (24%) and used instruments with evidence of reliability and validity less than half of the time (39%). We conclude with recommendations for those in the K-12 computing education research community to consider when reporting statistical data in future work so that we can increase the level of rigorous reporting in this growing field.

With the advent of computing education in primary and secondary schools, more research will be needed to determine the best way for students to learn how to think computationally and to design and develop software. As a field, computer science education researchers have been predominantly individuals who are steeped in content knowledge, but often have not had formal training in well-established educational research practices. Likewise, many also focus their research on undergraduate students in their own area of teaching. As the need for research into K-12 computing education grows, so does the opportunity for conducting quantitative studies in the field. In this special session, the presenters will present a broad overview of how to conduct and evaluate quantitative computing education research. The presenters will spend approximately 15 minutes on each of the 4 key areas: designing a study, implementing a study and collecting data, analyzing and interpreting results, and reporting on the study. The remainder of the time will be as a question and answer period with the audience. Since there is considerable overlap between computing education research at the primary/secondary and post-secondary levels, those new to research at the post-secondary level may also benefit from learning about form design, implementation, and reporting of research studies. Key takeaways for this session will include a basic understanding of K-12 computing education research from start to finish as well as a list of well-vetted resources to enable new researchers to explore and learn more.

The availability of computer science education in primary and secondary schools in the United States has grown rapidly over the last decade. Computing education research in K-12 has been increasing as well. In this study, we conducted a longitudinal analysis of seven years of data (2012-2018), curated from over 500 articles across 10 publication venues to identify trends in K-12 computing education research such as geographic location and curriculum and concepts taught. The data shows a decrease in the number of studies covering K-12 students receiving computing education even while there is an increase in the number of states adopting and implementing standards. The number of different concepts being researched is increasing, potentially reflecting the growth in what is being taught in the classroom. Demographic data is underreported (e.g., socio-economic status (SES) and disabilities of participants) which could directly limit generalizability of the studies to different learners as well as the ability to replicate and compare studies. We conclude with recommendations for how to better position this work for others trying to use the results to guide their efforts in creating standards or adopting techniques into their classrooms.

Engineering instructors often rely on lectures as their primary mode of instruction even in project courses. In the lecture mode of instruction student engagement with the course material is often low or non-existent until the date of an assessment activity (assignment or exam) is near. In passive learning environments students often do not get many opportunities to develop their soft skills. Many engineering educators regard experiential learning as a better way to train the next generation of software engineers to do project work. This paper describes the authors' experiences introducing active learning opportunities in a junior level software engineering course and a senior level game design course. The materials created for these courses were developed using a variation of the ADDIE (analyze, design, development, implementation, evaluation) process model. The team created a mix of case-study review, role play, trigger videos, and hands-on exercises involving work with software engineering artifacts or tools to facilitate coverage of the topics. The investigators collected observational data on student engagement while they were involved in various class activities and found that they were least engaged when listening to even short lectures. Student feedback and the authors' own lessons learned are being used to plan the next iterations of these courses.

This work extends previous research on subgoal labeled instructions by examining their effect across a semester-long, Java-based CS1 course. Across four quizzes, students were asked to explain in plain English the process that they would use to solve a programming problem. In this mixed methods study, we used the SOLO taxonomy to categorize student responses about problem-solving processes and compare students who learned with subgoal labels to those who did not. The use of the SOLO taxonomy classification allows us to look deeper than the mere correctness of answers to focus on the quality of the answers produced in terms of completeness of relevant concepts and explanation of relationships among concepts. Students who learned with subgoals produced higher-rated answers in terms of complexity and quality on three of four quizzes. Also, they were three times more likely to discuss issues of data type on a question about assignments and expressions than students who did not learn with subgoal labeling. This suggests that the use of subgoal labeling enabled students to gain a deeper and more complex understanding of the material presented in the course.

Subgoal learning has improved student problem-solving performance in programming, but it has been tested for only one-to-two hours of instruction at a time. Our work pioneers implementing subgoal learning throughout an entire introductory programming course. In this paper we discuss the protocol that we used to identify subgoals for core programming procedures, present the subgoal labels created for the course, and outline the subgoal-labeled instructional materials that were designed for a Java-based course. To examine the effect of subgoal labeled materials on student performance in the course, we compared quiz and exam grades between students who learned using subgoal labels and those who learned using conventional materials. Initial results indicate that learning with subgoals improves performance on early applications of concepts. Moreover, variance in performance was lower and persistence in the course was higher for students who learned with subgoals compared to those who learned with conventional materials, suggesting that learning with subgoal labels may uniquely benefit students who would normally receive low grades or dropout of the course.

In this Tips, Techniques, and Courseware talk, we will provide an overview of csedresearch.org, a site designed to provide resources for primary and secondary computer science (CS) education researchers. This resource center has three primary functions: 1) to provide a quick and easy way to search and access manually-curated data from over 500 articles from ten targeted venues (currently 2012-2018), 2) to provide a centralized location for evaluation instruments, many of which are focused on computing, and 3) to provide resources for computer science education research (checklists for studies, how to design a study, how to write a research question, etc.). Users can search for articles by filtering at a granular level on various curriculum and activity components, including student demographics, instructor demographics, and program components. Users can also search for evaluation instruments on criteria such as type, year published, constructs measured in the instrument, and whether the instrument has been validated or tested for reliability. We will provide a brief summary of the rationale for the site, the current state of the site, and how educators and researchers are using the site to date. We will also discuss future plans for the site and enable the audience to provide feedback that may help us shape the site to better meet the needs of the community.

As computing education research continues to grow and mature as a field, it becomes more important to focus on the quality and rigor of our research studies. One important aspect of any research study is its formal evaluation. Using standardized and validated instruments relevant to computer science education to perform evaluations can increase the quality of the study and the value of its results. However, researchers often create their own instruments rather than using existing ones, perhaps due to their lack of knowledge of the value of using an existing instrument or due to the challenge of finding such instruments. Through a review of relevant computing education literature, this paper presents a listing of 47 evaluation instruments specifically designed for measuring programs or constructs related to computing that can influence student achievement and learning. Analysis of purpose, target audience, reliability, and validity of the instruments is also presented. The paper ends with a call for the community to begin to make more regular use of validated instruments in their studies when possible and to develop and validate additional instruments in areas where few exist.

A growing body of evidence indicates that there is a deep effect of noncognitive factors on academic achievement and learning. In this study, we analyzed a set of 31 evaluation instruments designed to measure noncognitive constructs (e.g., self-efficacy, confidence, motivation) within computing education. Using the Lee and Shute framework, we assigned each of the 115 unique constructs found in the instruments into one of the four components (Student Engagement, Learning Strategies, School Climate, Social-familial Influences) and their subcomponents to determine which constructs are most frequently measured. We found that the majority of constructs were designed to measure Student Engagement (Affect and Cognition) and School Climate (Teacher Variables). Constructs measuring Learning Strategies and Social-Familial Influences (e.g., homework strategies, peer influences) occur the least. This study may enable further discussions of what noncognitive factors are/are not currently being measured within the computing education community.

Subgoal labeling is an instructional design framework for breaking down problems into pieces that are small enough for novices to grasp, and often difficult for instructors (i.e., experts) to articulate. Subgoal labels have been shown to improve student performance during problem solving in disciplines both in and out of computing. Improved student performance occurs because subgoal labels improve student transfer and retention of knowledge. With support from NSF (DUE-1712025, #1712231), subgoal labels have been identified and integrated into a CS1 course (variables, expressions, conditionals, loops, arrays, classes). This workshop will introduce participants to the materials and demonstrate how the subgoal labels and worked examples are integrated throughout the course. Materials include over 100 worked examples and practice problem pairs that increase in complexity and difficulty within each topic. The materials are designed to be integrated into CS1 courses as homework or classroom examples and activities. Assessment of topics using subgoal labels will also be discussed. Participants will also engage in an activity where they create an example for their own course using subgoal labels.

This Research Full Paper will discuss the demonstration of critical thinking of undergraduate students through their required co-op experiences. Critical thinking is an important 21st century skill that is expressed through measurable outcomes such as the ability to successfully use information, design, analyze, and problem solve. At Rochester Institute of Technology, a primarily technically focused university with strong programs in engineering and computing sciences, there is a dedicated effort to enhance the critical thinking ability of the students on campus. This paper will discuss our initiative for the integration of critical thinking into the student experience. In addition, the majority of our students complete one or more paid cooperative education experiences to satisfy their degree requirements. Upon completion of the experience, the employers are asked to rate the student performance on several metrics, one of which is demonstration of critical thinking. By analyzing the data from these surveys, we have found that employers consistently rate our students as proficient and capable. Our analysis shows that there is a slight difference in critical thinking ratings based on field of study, but not based on term in which it occurred, or number of experiences students engaged in.

This Research Full Paper describes the early research conducted to design a searchable repository of peer-reviewed research related to pre-college computing activities. This repository is part of a larger project to enable the computer science education community to gather and analyze data related to the effectiveness of these activities. To ensure that the repository met the needs of the community, we convened a virtual focus group of experienced and expert researchers and educators to discuss the repository's value, what it should contain, and how it should be presented. This paper presents 1) an analysis of these discussions, which shows that participants were equally interested in the repository's content and quality, 2) an initial list of variables that can affect the outcomes of these activities, and 3) an initial set of questions researchers should ask when authoring computer science education research involving precollege computing activities. We also consider these results in light of the larger challenge raised by others of how to improve quality research in computing education.

With the record growth in computing classes, it has been challenging for faculty to keep up with the logistics of the classroom, particularly in assessment. Automatic grading software and unit tests have been employed successfully in many settings to help ensure proper scoring of programming assignments. However, assignments that are not necessarily complete programs (in-class short quizzes, homework problems or worksheets, exams) present different challenges for scoring. In this session, we propose some tips and techniques for scoring such assessment items quickly and consistently, with the added benefit that grading criteria are clearer to the students.

Over the last few years, a number of repositories of information relevant to the computing education community have come online, each with different content and purpose. In this special session, we present an overview of these repositories and the content that each provides. Demonstrations of the functionality of the repositories will be shown and attendees are encouraged to come with their questions and suggestions for improvement if they are currently users of the repositories.

With the growth of computing education research in the last decade, we have found a call for a strengthening of empiricism within the computing education research community. Computer science education researchers are being asked to focus not only the innovation that the research creates or the question it answers, but also on validating the claims we made about the work. In this session, we will explore the relationship between evaluation and computing education research and why it is so vital to the success of the many computing education initiatives underway. It will also help computing faculty engaged in computer science education research understand why it is essential to integrate evaluation and validation from the very first conceptual stages of their intervention programs.

This paper provides a detailed examination of pre-college computing activities as reported in three Association of Computing Machinery (ACM) venues (2012-2016). Ninety-two articles describing informal learning activities were reviewed for 24 program elements (i.e., activity components, and student/instructor demographics). These 24 program elements were defined and shaped by a virtual focus group study and the articles themselves. Results indicate that the majority of authors adequately report age/grade levels of participants, number of participants, the type of activity, when the activity was offered, the tools/languages used in the activity, and whether the activity was required or elective. However, there is a deficiency in reporting many other important and foundational program elements, including contact hours of activity participants, clear learning objectives, the prior experience of participants (students and instructors), and many more. In conjunction with previous work, this paper provides recommendations to reduce these deficiencies. The Recommendations for Reporting Pre-College Computing Activities (Version 1.0) are presented to help researchers improve the quality of papers, set a standard of necessary data needed to replicate studies, and provide a basis for comparing activities and activity outcomes across multiple studies and experiences.

Subgoal labels are function-based instructional explanations that describe the problem-solving steps to the learner, highlighting the solution process. There is strong evidence that the use of subgoal labels within worked examples improves student learning in other STEM fields. Initial research shows that using subgoal labels within computer science improves student learning, but this has only been tested using a single programming concept (indefinite loops) with text-based programming languages. The proposers are currently expanding subgoal labels to the main programming concepts taught in an introductory programming course using an imperative programming language. In this BOF we seek to uncover tacit knowledge that programming instructors have in order to develop instructional materials that bridge the gap between students, who are CS novices, and instructors, who are CS experts, to improve learning for students who are under-prepared for or struggle in CS1. We will be seeking feedback on the selection of programming topics to be covered, the defined subgoals for those topics and the worked examples created for instructional purposes.

Soft skills are increasingly important to the engineering profession and course modifications are often needed to ensure students have opportunities to practice them prior to graduation. This suggests that engineering programs need to go beyond simply offering industry-based capstone courses and internships. Role-play has a long history as a tool for learning. It can be used to simulate real world practices in environments where consequences can be mitigated safely. This paper discusses the use of team role-play activities to simulate the experience of working in a professional, game development studio as a means of enhancing an advanced undergraduate game design course. In conjunction with the role-play, a gamification framework was used within the course to allow students to customize their course participation. Gamification was used to reward students for compliance with software process steps and for taking the initiative to improve their “soft skills”. In this project allowing students to negotiate the nature of their activities and rewards helped them develop those skills. The student feedback obtained and the authors' own lessons learned are being used to plan the next iteration of this course.

Europe, Australia, and North America face two similar challenges with respect to computing--there have not been enough students matriculating to keep up with demand and the lack of diversity remains an issue. To address these challenges, private and public resources have been allocated to educate primary and secondary students in computing. Previous meta-studies on these often segregated outreach efforts within the United States indicate that research on their impact remains spotty, short-term, and inconsistent, leaving a gap in the ability to evaluate their long-term impact. Building upon previous research focused in the U.S., this study examines similarities and differences in the results of 17 formal, peer-reviewed computing education research journals and conferences across Australia, Europe, and North America during 2009-2015. Results indicate that a lacuna exists in international research, paralleling results of the more narrowly focused study of U.S. venues. The paper further defines the major issues researchers face in conducting assessment studies for outreach and recommendations for addressing this gap.

There is a critical need to increase the number of skilled technology workers in part due to the supply of workers in these fields not yet keeping up with demand. In an effort to increase the interest of students in studying within these fields, many commercial, governmental, and non-for-profit educational groups have sponsored numerous activities aimed to expose students to computing prior to college. With the advent of CS for All, there is further need to research these programs and evaluate their effectiveness. This session will gather researchers and practitioners interested in researching the impact that pre-college computing activities have on their participants. Participants will learn about a new initiative underway to make the job of collecting and tracking data easier. To be useful for the broader community, this NSF-funded IUSE initiative requires input from the community it wishes to serve. Participants will have the opportunity to provide valuable input on what their needs are for tracking such activities and what tools may be helpful for them, including methods for data collection to produce meaningful data that can be compared to data provided by other researchers and practitioners. The focus of this discussion will be the long-term impact of these activities; however, we will discuss data collection techniques researchers have used in the past to gauge short-term impact as well.

This workshop provides participants with hands-on approaches to teaching common algorithms in an AP Computer Science A context, but common in CS1 in general. Teachers will explore strategies for teaching students: how to introduce commonly used algorithms to students; how to have students problem solve using hands-on techniques; how to determine which algorithm to use provided a program specification; write and modify the algorithm; and interpret the result of an algorithm. Together, participants will be looking at the commonly taught and assessed algorithms in first semester computing courses, especially those found on the AP Computer Science A assessment. A set of example free response questions from the AP Computer Science A assessment will be examined to identify how these algorithms are used and modified in solutions.

Developing large-scale software applications in teams has long been a popular feature of capstone courses in computing degree programs, particularly in the fields of computer science and software engineering. Instructors utilize real world problems as well as external clients to motivate the students and to provide authentic experiences. Within the field of game design and development, students have strong desires to make and ship a game or series of games during their collegiate experience. To address this need, we created a production studio course offering that addressed the desires of students to create a game as a culminating experience while focusing upon the production process and challenges that are particular to the game industry. In this paper we describe the process undertaken by students, faculty and staff to ship a game. We discuss the successes and failures of the team and in achieving their goals. We show the similarities and differences between the process of developing and shipping a game and other types of large-scale software design and development projects undertaken in similar courses.

Within computer science education, we have spent considerable effort on the introduction to the discipline (particularly to programming) and the teaching of novice programmers. However, we do not often think about the teaching and learning for the intermediate students. Having data about student's perceptions coming into a second year data structures course, it became of interest to systematically analyze the data to see what if any interesting patterns or results we could see in this information. Using the data from a first day of class general survey, we were able to gather information about student's perceived level of difficulty with some of the major topics that they have studied up until this point in the curriculum. The first half of the survey consisted of ten Likert-type items representing ten different topics in the previous courses. We are able to present quantitative analysis of the students perceived level of difficulty with these topics. The second half of the survey asked several open-ended questions about their learning. In this paper, we also provide analysis of the first of these questions which asked them to identify a topic that they studied previously that they still do not understand well. While not yet giving us the ability to generalize, this data shows some interesting information about students' perceived difficulty at a point in the curriculum that is not often studied, the intermediate years, and the results make an interesting case for further exploration of this problem with a larger group of students at more and diverse institutions.

Role play has a long history as a tool for learning. It is often used to simulate practices in the world in an environment where consequences can be mitigated. Despite its effective use by the armed forces, emergency responders, negotiators, and political campaign managers, role play is a tool seldom used in STEM classrooms. In this paper, we will describe the incorporation of certain aspects of role play in a game development processes course. This course, similar to courses in software engineering, focuses students on the entire product life cycle and engages them to complete a project from start to finish during the term. We used role play techniques to engage the students more fully in the processes of development by creating a scenario in which the students have suddenly become the owners of an independent game studio. During the semester, we engaged them as the owners on issues of business and legal concerns along with the typical production and process concerns of making their product. We had them face issues of missed deadlines and products that were not ready for production. We have informal evaluation in terms of feedback from end of the semester debriefs and classroom observations. We are using that for further refinements in the next course cycle.

Within computer science education, we have spent considerable effort on the introduction to the discipline (particularly to programming) and the teaching of novice programmers. However, we do not often think about the teaching and learning for the intermediate students. Having data about student's perceptions coming into a second year data structures course, it became of interest to systematically analyze the data to see what if any interesting patterns or results we could see in this information. In this paper, we present the results of the analysis of the open-ended questions at the end of the survey. The students were asked to identify a concept that they found particularly difficult and to tell us why. They were also asked to describe how they achieved success in learning a concept that they initially found difficult. We developed a system of thematic codes using a grounded theory approach on a sampling of the data, which we then applied to the entire data set. We uncovered patterns and trends in the student responses. During our analysis, we have uncovered that students are expressing blame for the lack of learning on themselves and instructors, pointing to perceived failures in the classroom environment and course structure, and are showing evidence of the understanding of the learning process. We conclude with some overall recommendations for how this findings could be operationalized to better understand the process for this intermediate learner.

A recent study about the effectiveness of subgoal labeling in an introductory computer science programming course both supported previous research and produced some puzzling results. In this study, we replicate the experiment with a different student population to determine if the results are repeatable. We also gave the experimental task to students in a follow-on course to explore if they had indeed mastered the programming concept. We found that the previous puzzling results were repeated. In addition, for the novice programmers, we found a statistically significant difference in performance based on whether the student had previous programming courses in high school. However, this performance difference disappears in a follow-on course after all students have taken an introductory computer science programming course. The results of this study have implications for how quickly students are evaluated for mastery of knowledge and how we group students in introductory programming courses.

Educators responsible for coordinating a staff to grade a large volume of exams or projects have learned, sometimes the hard way, the importance of writing good rubrics: they keep the graders on track, grading equitably, consistently, and quickly. But these benefits of a well-written rubric extend also to those of us without a grading staff, even to single classrooms of twenty or so students. Good rubrics will change your life!

In the past six years, dozens of conference papers and journal articles have been presented in Association of Computing Machinery (ACM) and Institute of Electrical and Electronics Engineers (IEEE) educational forums concerning computing outreach activities. Nearly half of these (47.5%) appeared in SIGCSE venues. In this study, we used the free-form question "What type of data has been collected in formal, peer-reviewed research that has been conducted on computing outreach activities in recent years"? as a basis for a systematic literature review in these venues from 2009-2015. During the analysis of the articles, it was discovered that a majority of efforts focused on middle school and high school students, a majority of the reported events took place in the United States, and almost half had a goal of increasing gender diversity in computing. This paper summarizes the information about the studies, including their data collection techniques and the data that was collected. We also present a list of recommended practices for data collection, methodologies, and reporting for educational researchers engaged in these activities in an effort to provide comparative data and allow us as a community to more scientifically understand the impact that these activities are having on the participants.

A lack of diversity in the computing field has existed for several decades, and although female participation in computing remains low, outreach programs attempting to address the situation are now quite numerous. To begin to understand whether or not these past activities have had long-term impact, we conducted a systematic literature review. Upon discovering that longitudinal studies were lacking, we investigated whether undergraduate students believed that their participation in computing activities prior to college contributed to their decision to major in a computing field. From the 770 participants in the study, we discovered that approximately 20% of males and 24% of females who were required to participate in computing activities chose a computing or related major, but that males perceived that the activity had a greater affect on their decision (20%) than females (6.9%). Females who participated in an outreach activity were more likely to major in computing. Compared with females who chose to major in computing, females who did not were less likely to indicate that the majority of students participating in activities were boys and that they were a welcome part of the groups. Results also showed that female participants who do not ultimately major in computing have a much stronger negative perception of the outreach activities than male participants who also chose a non-computing major. Although many computing outreach activities are designed to diversify computing, it may be the case that, overall, boys receive these activities more favorably than girls, although requiring participation yields approximately the same net positive impact.

Students with a technology focus often express and demonstrate that they find it difficult to communicate their ideas and designs. Students in the Rochester Institute of Technology's School of Interactive Games and Media are further challenged in that in order to be successful in their pursuit of a career in game design and development, they need to effectively convey their game ideas and design specifications while expressing the passion for the ideas that will convince others to climb on board and work on their projects. In this paper, we discuss the way we help our students develop these skills within a course structure. Through several course offerings, the faculty and students anecdotally noted that the students communication skills improved and their comfort in communication improved as well. In order to more accurately determine if this observed improvement was measurable, a survey of comfort with communication skills was created. The paper will present the results of an exploratory study using the instrument, which involved administering the survey to the students in the course as well as students in another course without a focus in development of these skills. The results from both sets of students were analyzed to determine if there was an increase in comfort with communication skills and to begin a process of validating this new instrument.

Hackathons bring developers, artists and designers together around a shared challenge: ideate, plan and create an application in a highly constrained time frame. A way to socialize, solve problems, and strengthen soft and hard skills, hackathons have grown tremendously in popularity in the last half decade. Despite this growth, it has been noted that females do not participate in hackathons with the same frequency as males. Some theorize that the hackathon culture is intimidating, does not appeal to women, or that it acts to amplify pre-existing cultural biases in computing. In this paper we introduce an alternative format for hackathons to address these issues. Think Global Hack Local (TGHL) is a non-competitive, community-based hackathon that connects non-profit organizations with student developers. Students donate a weekend to solve problems that these organizations otherwise lack the resources to solve. To date, there have been two TGHL hackathons, and we have observed many interesting divergences within the culture of TGHL in comparison to other hackathons. Response has been positive, and nearly all of them indicate that they would do it again. By adopting some of these ideas, we believe that hackathons can become an environment that is more inclusive and fun for all.

Over the last decade, there has been a concerted effort to bring more diverse voices to the technology field, with much of this being done through outreach activities to girls and boys. Unfortunately, data demonstrating the long-term impact of outreach activities remains rare. To contribute to knowledge on the longitudinal effect of outreach programs, we used a quantitative methodology that followed a descriptive design approach to explore the impact of participation in outreach activities on the choice of undergraduate major. Of those surveyed, 45.3% of the 770 respondents recalled participating in these activities. The results indicate that these activities had a more positive impact on Asians and more negative impact on Hispanics. Blacks/African Americans were more likely to voluntarily participate in outreach activities than Hispanics, and whites were more likely to feel that they were a welcome part of the group than non-whites. The results also may indicate that when outreach programs are available in earlier grades, they are not reaching non-white participants to the same extent as white participants.

Over the last several years, faculty at the Rochester Institute of Technology's School of Interactive Games and Media have been working with students to refine their ideas regarding game design, game production, and the overall ideation process. Most of our students are technology focused, and have to learn the process of communicating their ideas in a practical manner. They must learn the balance between being passionate about their ideas and forming a cogent argument as to why their idea matters. In short, they must learn the difference between having "the great idea" and being able to ideate over hundreds, if not thousands of ideas. Through the use of the game concept pitch as a tool for communicating their ideas, we are challenging students to express themselves in a particular way. Over the course of the semester, we explore game concept pitch processes, critique, iteration, and refinement of the student's ideas as well as presentation and communication skills. We also examine how the process of "first pitch" is applicable to wider areas of computing, engineering, and science education along with career and field skills.

In this paper we discuss the construction of Martha Madison: Marvelous Machines, a serious video game designed to help students learn about the use of simple machines in the everyday world. Game participants are introduced to real-world, meaningful tasks that must be solved through the placement and manipulation of simple machines. The game provides in-game scaffolding and is played collaboratively by students co-located in the same space, playing on the same computer. The Martha Madison game provides learners specific tasks and goals to complete and an open world in which to complete them. The design of Martha Madison is intended to create a dialogue between players in the service of solving authentic problems presented in the game world and providing opportunities for students to change overall perceptions regarding the social and collaborative aspects of STEM work. The authors discuss the reasons, approaches, and challenges of the design of Martha Madison.

There is little argument that creating experiences for students that will increase student success and retention is a goal that is shared by anyone teaching courses. In computing courses, we have seen a lot of focused attention on the first year introductory courses as it has been shown that it is failure in these courses that drives students away from the discipline. In this paper we describe success we have had with first year students through the creation of achievement system for the undergraduate experience. Through the Just Press Play project and its associated achievements, we observed a significant shift in student behavior in relation to our first year students which lead to a greater level of student success in their first course.

Giving students an appreciation of the benefits of using design patterns and an ability to use them effectively in developing code presents several interesting pedagogical challenges. This paper discusses pedagogical lessons learned at the "Killer Examples" for Design Patterns and Objects First series of workshops held at the Object Oriented Programming, Systems, Languages and Applications (OOPSLA) conference over the past four years. It also showcases three "killer examples" which can be used to support the teaching of design patterns.

Over the last several years, there have been reports of many institutions using undergraduate students as teaching assistants (UTAs) in the classroom for CS1 as well as other courses in the curriculum. The literature has shown successes over a wide range of class sizes and UTA responsibilities. At University at Buffalo, we have been using undergraduates as teaching assistants in our CS1 course since Spring 2002, and have been impressed with the results. Throughout the deployment of the UTA program, the instructors of CS1 have observed that when UTAs are utilized in the classroom, both the students and the UTAs themselves benefit from their interactions. The UTAs have also become actively involved in providing feedback about the course design and have been suggesting improvements to assignments and in-class examples. They have also been involved in the process to hire new UTAs to replace those that are graduating. We have observed that such interactions have improved the UTAs sense of investment and ownership in the CS1 course.

We have recently launched a reorganized discrete structures course being taught in the computer science department for computer science majors. The main idea of the reorganization is to make the course relevant and productive for the computer science students. During the Spring of 2003, students who were taking both discrete structures and CS2 commented that taking both courses at the same time was helpful. An empirical investigation was conducted to see if there was a benefit to students taking both courses concurrently. The results show that there was a benefit to students taking both courses. The students who were taking both courses were performing better in discrete structures than those students who were not.