The call for alternative grading practices has been made both inside and outside the computing education community. Various practices exist to provide assessment and feedback to students that do not rely strictly on points out of one hundred percent, weighted averages, high stakes assignments, and grading for behaviors instead of learning. However, modern classrooms, especially computer science classrooms, rely on a myriad of digital tools to organize and maintain the course structure. Tools like learning management systems, automatic grading systems, submission systems, and practice systems all exist for computing students and faculty to use to help support the learning of programming concepts. By and large, these systems all rely on an underlying mechanism of points and aggregating points for scoring. In the face of such technological choices, adopting alternative grading practices can prove challenging for instructors and confusing for students. In this position paper, we advocate addressing key research problems to make these systems easier to use with alternative grading practices. These include comprehensive support for categorical grading, comprehensive support for rework and resubmission, and improved protocols for communication of scores and feedback. We propose an extension to LTI to support the needs of alternative grading practices, and we provide an initial design for this LTI extension. We discuss current problems and potential solutions and challenge the community to work on these problems and consider the design of future systems to embrace grading approaches that go beyond just points-based scoring.

This innovative practice paper describes our experiences with alternative grading practices in introductory computing courses and two large public universities in the United States. Computing classrooms often use traditional grading practices involving allocating points to assignments, deducting points for mistakes and tardiness, and combining assignment scores using a weighted average to determine grades. Recent research suggests that these practices may diminish achievement, discourage students, and suppress effort to such an extent that they are considered by some as detrimental. We approach our work as an exploratory case study, without predefined research questions or hypotheses. Our experiences began with the adoption of specifications grading. We outline the grading scheme applied to traditional programming assignments and exams/quizzes, and discuss the initial integration of these schemes with conventional auto-grading tools. We delve into student perceptions of alternative grading, their utilization of flexible deadlines, and resubmission opportunities. We conclude with a discussion of two challenges encountered during our exploration: student acceptance of a novel grading form, and the adaptation of tools designed for traditional grading to support alternative grading mechanisms. Our early exploration aims to inspire further research on the use of alternative grading in computing. It is clear from our observations that simply implementing the practices does not ensure the equitable and inclusive outcomes that can be achieved with these practices. If students are not prepared to use these practices, they find them difficult to understand and can feel that they are not being treated fairly. Additionally, we wish to foster a community of practice to assist faculty members exploring these changes, with the goal of creating more equitable and inclusive classrooms.

It is often the case that computer science classrooms use traditional grading practices where points are allocated to assignments, mistakes result in point deductions, and assignment scores are combined using some form of weighted averaging to determine grades. Unfortunately, traditional grading practices have been shown to reduce achievement, discourage students, and suppress effort to such an extent that some common elements of traditional grading practices have been termed toxic. Using grades to reward or punish student behavior does not encourage learning and instead increases anxiety and stress. These toxic elements are present throughout computing education and computer science classrooms in the form of late penalties, lack of credit for code that doesn't compile or pass certain unit tests, among others. These types of metrics, that evaluate behavior are often influenced by implicit bias, factors outside of the classrooms (e.g., part-time employment), and family life situations (e.g., students who are caregivers). Often, students in these situations are disproportionately from low-socioeconomic backgrounds and predominantly students of color. Through this paper, we will present a case for adoption of equitable grading practices and a call for additional support in classroom and teaching technologies as well as support from administrations both at the department and university level. By adopting a community of practice approach, we argue that we can support new faculty making these changes, which would be more equitable and inclusive. Further, these practices have been shown to better support student learning and can help increase student learning gains and retention.

While there were previous efforts to increase the number of women and students of color graduating with a computer science (CS) degree, the establishment of the National Science Foundation's Broadening Participation in Computing (BPC) program in 2005 created an upsurge in this area of research. This program encouraged the creation and growth of many alliances that are still well-established today. While the CS field continues to grow drastically, with an increase of 300% in students earning a bachelor's in computer science since 2006, there is still much work to be done toward BPC efforts. When looking at intersectional groups such as Hispanic women and Black women, there is no significant change and a consistent decline in earned bachelor's computing degrees, respectively, within the last 20 years. Much of BPC research has become identity-focused, however, there is a dearth of research on intersectional identity groups. This paper presents a historic overview of BPC initiatives, research, and an overview of the seminal work being conducted about intersectional identity in computing. Also discussed are challenges specific to BPC research, opportunities to advance computing education research, and future avenues for advocacy of equitable computing education and BPC work.

Multi-pronged programs that involve students in a combination of proven interventions (i.e., tutoring other students, building community, developing skills, etc.) constitute one pedagogical approach to increasing the number and diversity of computing professionals. In this manuscript, we evaluate the efficacy of one such multi-pronged program, the STARS Computing Corps, a Broadening Participation in Computing Alliance program funded by the National Science Foundation. These analyses improve upon previous efforts to assess the efficacy of STARS by examining dosage effects of the program, adding controls for students' initial intentions to pursue computing, and conducting these analyses at various points in a student's participation in STARS. We also conduct analyses to determine the efficacy of various STARS activities. Controlling for students' initial intentions to persist in computing, we find robust evidence that spending more time each week on STARS' activities positively predicts students' intentions to persist in a computing career, and that STARS has a heightened positive impact on Black and Hispanic students. We do not find evidence that the number of semesters a student spends in STARS is predictive of computing persistence, nor do we find differences in the efficacy of various STARS activities. In sum, these results suggest that STARS has a positive impact on students' intentions to persist in computing and that multi-pronged programs like STARS should focus on the intensity of participation (as opposed to the length of participation or a particular activity) to increase students' desire to persist in computing careers.

This full paper presents the Collaborative Active Learning and Inclusiveness (CALI) inventory, and an analytical model using the CALI inventory, demographic data, mindset surveys, and knowledge mastery assessment, to explore relationships between classroom climate and student experiences. The CALI inventory enables the investigation of the impact of the student experience in an active learning classroom by distinguishing the factors that characterize the structure, social learning, and inclusive practices. The Structure Index includes components related to course setup, organization, assessment, grading, and communications. The Sociality Index includes components related to opportunities for students to interact with each other. The Inclusiveness Index includes components related to how the instructor communicates a sense of belonging to the students through a growth mindset and inclusive policies and practices. A CS Mindset Instrument was developed based on research that measured students' self-efficacy by evaluating the extent of variation in their self-perceived ability to accomplish a task, sense of belonging in computing, and professional identity development. Demographic data is collected that allows for an analysis using an intersectional lens to acknowledge the complexity of social and cultural contexts. The knowledge and mastery assessments capture changes in competency through pre-post mastery quizzes. The combination of CALI with other instruments, including those that characterize student mindset, identity, and levels of mastery, enables investigation of how various practices of inclusive and collaborative active learning have differential effects on students with different identities in computer science.

Google's CS Research Mentorship Program (CSRMP) cultivates pursuit and persistence in the computing research trajectory for students from historically marginalized groups through virtual career mentorship from industry professionals, a peer community, and just-in-time resources. Since 2018, 287 Google mentors have engaged 1,018 students from 247 institutions in the U.S. and Canada. The program employs socioemotional support and advocacy to navigate systemic barriers by validating students' intersectional identities in order to improve outcomes in core constructs for students: self-efficacy, sense of belonging, research skills, motivation to pursue graduate school and research careers, and intersectional capital. Evaluation outcomes from 400 matched respondents (68% response rate) indicate that CSRMP affects positive, statistically significant change in those constructs that largely persists across demographic subgroups. 80% aim to pursue computing research careers, and significantly fewer students are undecided about their future career. We were also able to identify disaggregated learnings: Black, Indigenous, and Latinx students are significantly less likely to submit to a research conference, and students from Historically Marginalized Groups (defined within) are significantly less likely to apply to a CS graduate program. We discuss key design elements of the program, how the findings are informing future iterations, and the potential for the model to scale.

This full paper presents CALI 1.0: an inventory to capture the various components of collaborative active learning and inclusive practices in introductory CS courses, as a basis for studying their impact on a student’s ability to engage with course material and be successful in the course. Collaborative active learning is defined as a pedagogy where learners work with one another on activities that involve actively engaging with the course material through discussions, problem-solving, case studies, role plays, and other methods. CALI consists of three separate indices: Structure, Sociality, and Inclusiveness. The Structure Index includes components related to course setup, organization, assessment, grading, and communications. The Sociality Index includes components related to opportunities for students to interact with each other. The Inclusiveness Index includes components related to how the instructor communicates a sense of belonging to the students through a growth mindset and inclusive policies and practices. In this paper, we present the inventory and results from a focus group of faculty to gauge responses to using this inventory to capture the teaching components of their courses. We present CALI as a tool that can be used to study how teaching practices, e.g. inclusive and collaborative active learning pedagogies, impact course experiences for students in different demographic groups as well as a reflective tool for use by faculty while designing their courses.

We have established a research-practice partnership (RPP) to build a computer science (CS) and computational thinking (CT)-focused STEM ecosystem at two middle schools. Creating such an ecosystem to broaden student participation in computing through an RPP approach involves all stakeholders in the research process. Borrowing upon visual participatory research methods, we developed a graphic research instrument to engage teachers in the research process and elicit their perspectives on strategies for building the ecosystem. This experience report describes our research methodology across two distinct cases to demonstrate the utility of this drawing activity as an investigative and partnership development tool. The contribution is in offering a flexible approach to other university-based RPP teams that enables a synergistic partnership development tool and data collection instrument that can be tailored to a variety of RPP contexts, facilitating more productive and equitable ways of engaging stakeholders in the research process. We describe our project contexts and share results from the pilot study with practitioner-members of our RPP teams. We discuss two cases to highlight the contribution this approach made to the development of our partnerships.

In this experience report, we describe the rationale and need for immersive research experiences (IREs) in computer science (CS) that are designed to foster an inclusive community that encourages pursuit of graduate education for undergraduate women. Google's exploreCSR supports institutions across the US to execute IREs in computing throughout the academic year. We describe the program design and framework, the evaluation model, and present outcomes from two years of implementation across 29 institutions, with 1,983 (92% female) student participants collectively. The unique features of the program are that it aligns goals, measurements, and best practices across a national network of hands-on, localized IREs, resulting in peer communities and a sizable sample of undergraduates who identify as women and/or African-American/Black, American Indian/Alaska Native/Native American, Hispanic/Latinx, and/or Native Hawaiian/Pacific Islander (AAHN). We discuss recommendations for effective IRE programming based on our evaluation and the features found to be particularly salient for AAHN women. The contribution of this work is in describing how a national initiative for IREs builds community and creates conditions known to support persistence of women in computer science.

This Research to Practice work in progress paper presents a model to bridge the gap in community college engineering students successfully transferring to 4-year institutions. In 2015, the state of Tennessee (TN) launched TN Promise, a scholarship and mentoring program that enables tuition-free attendance at two-year community colleges for eligible Tennessee high school graduates. With over 18,000 students already enrolled in TN Promise, the number of students who may choose to transfer to four-year institutions is expected to grow exponentially in the coming years. To prepare for the expected influx of transfer students, and to address known barriers transfer students face, we have designed the Academic Intervention, Social Supports and Scholarships for Engineering Transfer Students program (ASSETS). While transfer students are ready for college, they often face institutional barriers to success that need addressing, particularly in Engineering.The ASSETS program is a comprehensive support ecosystem designed to improve retention and reduce time to graduation for engineering transfer students. In the second year of operation, with 23 enrolled ASSETS scholars, the program has implemented and begun studying the effectiveness of evidence-based strategies to reduce these institutional barriers to success, improve retention rates, and reduce time to graduation among engineering transfer students.This paper presents the initial findings and outcomes of the ASSETS program to enhance opportunities and success of transfer students.

We investigated learning outcomes within an undergraduate C Programming course that is taught in multiple modalities: in-person, online and blended. Because this course has been taught by the same instructor, using the same scaffolding activities, materials and approaches, we were uniquely positioned to conduct a quasi-experimental study of learning outcomes between courses and within students. The overarching goal was to glean knowledge and implications about assessment practices for undergraduate courses that are taught in multiple modalities. The objectives of our research are primarily to discern what differential impacts, if any, are found between the in-person and the online course delivery. We aimed to discover learning outcome patterns among the students who participate in these modalities. Findings from this study provide valuable information for undergraduate Computer Science programs by identifying any differential learning outcomes that students experience between in-person and online course instruction. The research questions addressed by the study were as follows: 1) What impact does modality have on student learning outcomes? 2) What patterns are discernable across student groups? 3) What relationship is there between final course grades and assignment module learning outcomes? In earlier work, we were surprised that no significant differences were obtained between course modality. While this was an encouraging finding, we believed that further data collection and analysis were needed, before making general conclusions about the two modalities. This paper reports on our efforts to collect additional data, while considering additional variables, such as instructor, multiple modalities, and online course design approaches.

This project is a Research to Practice Partnership (RPP) between two middle schools and two universities. It focuses on investigating problems and on identifying solutions around increasing participation and interest in computer science (CS). We aim to do this by identifying, experimenting with, and fine-tuning methods to help students develop computational thinking (CT) skills. The research employs a STEM ecosystem model, which facilitates a support structure that aims to mitigate barriers and impact students as they progress in STEM areas. While this RPP is still a work in progress, we present data from the first year of our collaboration with one of the middle schools. While the research questions guiding this RPP are intended to be iterative and revised annually, year one data provides perspectives on (1) barriers to developing a STEM ecosystem that supports CS/CT for every student through integration into science, math, and language arts courses, (2) the factors or interventions needed for the development of a CS/CT focused ecosystem that supports everyone in the school, (3) the indicators of success for a CS/CT focused STEM ecosystem in a school, and (4) how the ecosystem prepares and engages all students for CS/CT work in high school. Year one data is discussed in terms of the STEM ecosystem framework and in how it will guide the next steps in this partnership. This project contributes to the understanding of how to prepare future generations for participation in a workforce where knowledge of the foundations of CS/CT is integral to success.

This innovative practice work-in-progress paper presents measurement and analysis of a comprehensive model of pedagogical change in the computing college of a large, urban university in the United States. Our approach to change has two major thrusts: (1) a model for systemic change in the culture and climate of computing education, and (2) pedagogical patterns that transform students who enter undergraduate computing programs from people with an interest in computing to people who identify as computing professionals. We collected a total of 940 student survey responses across four semesters. Survey items gauged students’ connection to their peers and their profession and were used to create a composite measure of peer learning and professional identity. Exploratory Factor Analysis indicates an internally consistent factor structure composed of 18 items with a Cronbach’s alpha of 0.91. In this paper, we present the methodology for measurement and analysis, describe our rationale in an engaged computing education context, and discuss how studying the constructs of peer learning and professional identity contributes to the knowledge base of the computing education community.

We describe a study of learning outcomes in an undergraduate C Programming course taught in two modalities: in-person and online. Because this course has been taught by the same instructor, using the same scaffolding activities, materials and approaches, we were uniquely positioned to conduct a quasi-experimental study of learning outcomes between courses and within students. Our goal was to glean knowledge and implications about assessment practices for undergraduate courses that are taught in two modalities. The objectives of our research are to discern what differential impacts are found between in-person and online delivery modalities. We aimed to discover learning outcome patterns among the students who participate in these modalities. Findings from this study provide valuable information for undergraduate Computer Science programs by identifying any differential learning outcomes that students experience between in-person and online course instruction. The research questions addressed were: 1) What impact does modality have on student learning outcomes? 2) What patterns are discernable across student groups 3) What relationship is there between final course grades and assignment module learning outcomes? We were surprised that no significant differences were obtained between course modality. While this is an encouraging finding, we believe that further data collection and analysis are needed, before making general conclusions. We are planning future studies to account for additional variables, such as instructor, multiple modalities, and online course design approaches.

This Research-to-Practice Work-in-Progress Paper presents a midway report on a change initiative underway in the College of Computing and Informatics (CCI) at UNC Charlotte. Comprised of approximately 100 faculty members and nearing 2,000 undergraduate majors, CCI was awarded a 5-year, $2 million grant from the National Science Foundation in 2015 to revolutionize computer science education at the collegiate level. This initiative, which seeks to simultaneously achieve both pedagogical and organizational change, is built upon a foundation of educational innovation through engaged teaching practices. To sustain educational innovation in CCI beyond the funded scope of the project, pedagogical change is thus strategically embedded in CCI's organizational structure through a 3-stage model of faculty adoption, redesigned student course evaluations, and realigned values in the reappointment, promotion, and tenure process. By recognizing that sustained organizational change takes root when there is concurrent buy-in from organizational members—both bottom-up and top-down— this reform initiative seeks to embed two-pronged change not only via educational innovation (pedagogical change), but also in the day-to-day practices, policies, and physical environment of the College itself (organizational change).

It has been long established that discrete structures is an important and foundational component of the computer science curriculum. However, the topics covered in this course tend to be more abstract than those covered in most other introductory computer science courses. This leads to additional challenges for instructors and students. To deal with these challenges, we introduce a new pedagogy for teaching this course. Our approach is based on a variant of the flipped classroom paradigm and is comprised of four main components: before class preparatory work, in-class mini lecture, in-class team based problem solving activities, and weekly individual assignments. In this paper, we discuss these components in detail. Our approach is informed by several cutting-edge teaching methodologies including active learning, light weight teams, and gamification. We conclude the paper by discussing the results of a survey taken by the students and a summary of the grades attained in the class. These show that our approach was well received by the students and has led to good learning outcomes.

We present results of a college wide undergraduate retention study tracking student retention in computing, comparing students who experience a flipped, active learning version of CS1 against those who experience a traditional lecture and lab version of CS1. We examine demographic subgroups to understand retention differences between sexes and racial/ethnic groups. Specifically, we examine which students exit computing majors in the semester immediately after taking CS1, and those who leave following one academic year. This allows us to focus on how the immediate experience of the CS1 teaching approach impacts desire to continue in computer science during the critical first year in CS. Our dataset includes 698 CS majors who took CS1 in either the flipped or traditional style, between Fall 2013 and Fall 2016, at a large, comprehensive, urban research university in the US. Our results show that women were less likely to switch majors after taking the flipped version than after taking the traditional version. Conversely, male students were more likely to be retained following the traditional course, and less likely to be retained following the flipped course. Performance of CS majors in CS1, as measured by DFW rates, is statistically higher in the flipped classes than performance in the traditional classes. One-year retention in the major for under-represented groups (women and racial minorities) was higher in the flipped classes for new freshmen taking CS1, but not for transfer students.

In this paper, we describe a large-scale, long-term study of a National Science Foundation undergraduate research program via the Computer and Information Science and Engineering (CISE) Research Experiences for Undergraduates (REU) Evaluation Toolkit project. The overarching research questions are: what attitudinal shifts occur for undergraduate research experience student participants, and do experience perceptions differ based upon student demographics? Results across two cohort years showed statistically significant gains for several key program level outcomes. However, intent to go to graduate school decreased in both cohorts. Gender and ethnicity differences were observed. Women and underrepresented minority group students had significantly higher scores for several factors. These findings are discussed in context of culturally relevant pedagogy for undergraduate research programs. This study is unique in its scale and scope, providing a depiction of student outcomes across multiple cohorts of a national sample including over 58 computing research sites and more than 900 undergraduate students.

The Connected Learner is a re-orientation of undergraduate computing education that focuses on connecting students to peers, the profession, and purpose. The College of Computing and Informatics at UNC Charlotte—comprised of three departments with an undergraduate enrollment of approximately 1,500 students—is a large research institution in an urban setting with a diverse student population. Within this unique context, the project aim is to build a sustainable practice of educational innovation across the undergraduate computing curriculum by increasing faculty awareness of teaching innovations, resources for pedagogical change, and support for teaching practices that engage students. The vision is to create an active learning environment that transforms the student entering the computing undergraduate program from a person with an interest in computing to a person with an affinity identity [1] as a computing professional through these ongoing connections. We employ a systems theory of change for the Connected Learner based on two foundational concepts: flipped classrooms and engagement theory. To achieve related goals of improving student retention and time to graduation, Connected Learner teaching strategies are being integrated across the undergraduate curriculum with an initial focus on introductory gateway courses. Change is occurring via infrastructure supports to sustain learning practices across our college through faculty development initiatives (hiring, training, mentoring, and incentives) that are designed to inform faculty about engagement pedagogies, motivate faculty to adopt these practices, and shift pedagogical attitudes. During the first year of the five-year project, quantitative and qualitative data from students and faculty was collected and analyzed. This data provides a baseline of student attitudes and performance as well as a baseline of faculty attitudes and teaching practices. In this paper, we present the background, context, organizational structure, and research questions for the project. Findings from Year 1 are discussed, including academic outcomes, project milestones, and student attitudes towards new teaching practices. Long-term project goals and expectations are presented.

The basic principles of a flipped classroom teaching method are to deliver content outside of the class and to move active learning into the classroom. There are many strategies for delivering the content online, such as having instructors prepare online lectures, wrapping the course around a MOOC, and curating online videos from various sources. There are also many strategies for including active learning in the classroom that go beyond providing programming labs, and can include various forms of peer instruction. In this paper we describe our experiences flipping four different computer science classes across multiple semesters over two years. This breadth of experience with classroom flipping has enabled us to compare strategies and approaches and develop an understanding of which approaches appear to work under which circumstances. We discuss how we structured out-of-class preparatory work, how we created or sourced online videos, how we used active learning activities in-class to scaffold skills development and identify students' misconceptions, and how we structured teams for in class activities. This paper contributes a set of flexible strategies to consider for provision of curricular content out-of-class, structuring students' preparatory work, applying active learning of skills and concepts, and leveraging social interaction and peer instruction for CS education. We present the impact of our approaches based upon leading indicators of course evaluations and student surveys. We discuss lessons learned and students' responses to our strategies.

The Students and Technology in Academia, Research, and Service (STARS) Alliance is a nationally-connected system of regional partnerships among higher education, K-12 schools, industry and the community with a mission to broaden the participation of women, under-represented minorities and persons with disabilities in computing (BPC). Each regional partnership is led by a STARS member college or university with partners such as local chapters of the Girl Scouts, the Black Data Processors Association, public libraries, Citizen Schools, and companies that employ computing graduates. STARS goals include retaining and graduating undergraduates and recruiting and bridging undergraduates into graduate programs. The alliance works toward these goals through activities that advance the central values of Technical Excellence, Leadership, Community, and Service and Civic Engagement. In particular, all STARS college and university members implement the STARS Leadership Corps (SLC), an innovative model for enveloping a diverse set of BPC practices within a common framework for implementation within multiple organizations, common assessment, and sustainability through curricula integration. Herein, we describe the SLC model and its implementation in the STARS schools, including details of an SLC service-learning course that has been adopted by eight STARS schools. We report the results of our three-year study of the SLC in the 20 STARS schools. Our study found a positive effect of participation in the SLC on important student success variables, including self-efficacy, perceived social relevance of computing, grade point average, and commitment to remain in computing. Results indicate that the SLC model is effective for students under-represented in computing, as well as for those not from under-represented groups.

Recent years have seen a growing awareness in the computing education community that initiatives outside the classroom are vital for retaining students and preparing them for a collaborative and dynamic professional environment. Particularly important are programs that develop rich technical skills while increasing students' interest in computing disciplines. We present Computing Identity Mentoring, an intervention designed to increase commitment to computing while enhancing students' technical and leadership skills. This program was implemented at seven universities during 2008-2009. Preliminary results suggest that Computing Identity Mentoring contributes to students' self-efficacy regarding computing and leadership, and solidifies students' commitment to a career in computing. This paper presents early findings on the effectiveness of the approach and illustrates Computing Identity Mentoring in the context of three of the seven institutions where it has been implemented.

Research experiences for undergraduates are considered an effective means for increasing student retention and encouraging undergraduate students to continue on to graduate school. However, managing a cohort of undergraduate researchers, with varying skill levels, can be daunting for faculty advisors. We have developed a program to engage students in research and outreach in visualization, virtual reality, networked robotics, and interactive games. Our program immerses students into the life of a lab, employing a situated learning approach that includes tiered mentoring and collaboration to enable students at all levels to contribute to research. Students work in research teams comprised of other undergraduates, graduate students and faculty, and participate in professional development and social gatherings within the larger cohort. Results from our first two years indicate this approach is manageable and effective for increasing students' ability and desire to conduct research.