Metacognition is widely acknowledged as a key soft skill in collaborative software development. The ability to plan, monitor, and reflect on cognitive and team processes is crucial to the efficient and effective functioning of a software team. To explore students' use of reflection--one aspect of metacognition--in undergraduate team software projects, we analyzed the online chat channels of teams participating in agile software development projects in two undergraduate courses that took place exclusively online (n = 23 teams, 117 students, and 4,915 chat messages). Teams' online chats were dominated by discussions of work completed and to be done; just two percent of all chat messages showed evidence of reflection. A follow-up analysis of chat vignettes centered around reflection messages (n = 63) indicates that three-fourths of the those messages were prompted by a course requirement; just 14% arose organically within the context of teams' ongoing project work. Based on our findings, we identify opportunities for computing educators to increase, through pedagogical and technological interventions, teams' use of reflection in team software projects.

Assessing team software development projects is notoriously difficult and typically based on subjective metrics. To help make assessments more rigorous, we conducted an empirical study to explore relationships between subjective metrics based on peer and instructor assessments, and objective metrics based on GitHub and chat data. We studied 23 undergraduate software teams ( n = 117 students) from two undergraduate computing courses at two North American research universities. We collected data on teams’ (a) commits and issues from their GitHub code repositories, (b) chat messages from their Slack and Microsoft Teams channels, (c) peer evaluation ratings from the CATME peer evaluation system, and (d) individual assignment grades from the courses. We derived metrics from (a) and (b) to measure both individual team members’ contributions to the team, and the equality of team members’ contributions. We then performed Pearson analyses to identify correlations among the metrics, peer evaluation ratings, and individual grades. We found significant positive correlations between team members’ GitHub contributions, chat contributions, and peer evaluation ratings. In addition, the equality of teams’ GitHub contributions was positively correlated with teams’ average peer evaluation ratings and negatively correlated with the variance in those ratings. However, no such positive correlations were detected between the equality of teams’ chat contributions and their peer evaluation ratings. Our study extends previous research results by providing evidence that (a) team members’ chat contributions, like their GitHub contributions, are positively correlated with their peer evaluation ratings; (b) team members’ chat contributions are positively correlated with their GitHub contributions; and (c) the equality of team’ GitHub contributions is positively correlated with their peer evaluation ratings. These results lend further support to the idea that combining objective and subjective metrics can make the assessment of team software projects more comprehensive and rigorous.

Students' experience with software testing in undergraduate computing courses is often relatively shallow, as compared to the importance of the topic. This experience report describes introducing industrial-strength testing into CMPSC 156, an upper division course in software engineering at UC Santa Barbara. We describe our efforts to modify our software engineering course to introduce rigorous test-coverage requirements into full-stack web development projects, requirements similar to those the authors had experienced in a professional software development setting. We present student feedback on the course and coverage metrics for the projects. We reflect on what about these changes worked (or didn't), and provide suggestions for other instructors that would like to give their students a deeper experience with software testing in their software engineering courses.

Providing students with authentic software development experiences is essential to preparing them for careers in industry. To that end, many undergraduate courses include a team-based software development experience in which each team works on a different software project. This raises significant challenges for assessing student work and measuring the impact of pedagogical interventions: What do we measure and how, when each team is working on a different project? To address this question, we present a collection of metrics developed using the Goal-Question-Metric framework from the empirical software engineering literature, and an empirical study in which we applied those metrics to assess 23 team software projects involving 94 students at three institutions. Study results suggest that these metrics, which gauge commit, issue, and overall product quality, are sensitive to differences in the quality of teams' processes and products. This work contributes a new metric-based approach to evaluating key aspects of software development processes and products in a wide variety of computing courses.

In this interactive special session, participants will learn to facilitate a workshop for training teaching assistants (TAs) based on a card game. The game invites players to respond to challenging scenarios that TAs may face in the course of their duties. Each scenario appears on a card, and players discuss how they would respond. Session participants will receive a deck of 52 of these cards created with funding from a SIGCSE Special Projects Grant. This session uses the format of two previous well attended SIGCSE special sessions that focused on bias and had a target audience of faculty. While the intended audience is those interested in TA training, everyone interested in inclusive teaching practices is welcome to join in the rich small and large group discussions that we expect to take place!

Undergraduates have been an important part of the teaching staff at many universities for decades. Recent work such as the Peer Teaching Summit at SIGCSE 2019 [1] and a systematic literature review [2] have focused more attention on issues related to the use of undergraduates in teaching assistant roles. This BOF provides a forum to discuss open research questions about undergrad TA/mentor programs at various stages in their evolution. Attendees will have an opportunity to discuss research questions, research methods, and explore possible collaborations. Discussion Leader(s): Diba Mirza will open the session by summarizing the key findings of literature review on UTAs from ICER'19 [1]-in particular, the fact that while there is widespread consensus that using UTAs is a good idea, the evidence backing up this consensus is mostly anecdotal. This creates many opportunities to establish the effectiveness of current practices, and the claimed benefits of the use of UTAs through more rigorous research and to discuss innovative ways to incorporate UTAs in teaching outside of what has been reported in the literature. This will be followed by a themed discussion to brainstorm about research within each of these areas. Based on the large turnout at the BOF on Undergraduate Teaching Assistants in SIGCSE'19, we plan to organize the discussion in smaller groups. Diba Mirza, Phillip Conrad, and Cynthia Lee will lead the discussion within each group. The leads will also document the discussions and share it with the participants.

We present a systematic literature review of the prior work on Undergraduate Teaching Assistants (UTAs) in Computer Science with two goals: (1) to create a taxonomy of practices that relate to the design and implementation of UTA programs, (2) to identify the benefits of using UTAs as claimed by the literature and characterize the level of evidence for those claims. We analyze 336 excerpts from 40 papers related to these goals. We use content analysis on excerpts describing practices to extract high-level themes that include recruiting, UTA and program coordinator duties, training, evaluation and organization of UTA programs. We perform a more fine-grained analysis within each theme to identify specific questions about UTA programs and the answers provided by the literature. Using a similar technique, we report on the claimed benefits of UTA programs to students, UTAs, instructors and institutions. Our analysis follows well-defined protocols involving multiple reviewers and we report on the inter-rater reliability. The results provided in this paper lay the groundwork for developing evidence-based best practices in UTA programs and inform practice and policy related to the use of UTAs at tertiary institutions. As such, it is relevant to educators establishing a new UTA program, expanding an existing program, or continuously improving an established program, as well as those designing research studies of such programs.

Packaging materials is a generalized term to capture a broad array of tasks (creating, revising, sharing, finding, crediting, etc.) for materials such as assignments, teacher notes, and evaluation data. Substantial effort has gone into creating materials over the years, but the community still struggles to find ways to effectively manage these. This BoF provides an opportunity to identify needs, concerns, prior efforts, and future plans. A primary goal is the formation of a Working Group tasked to develop a standard for curricular material creation and sharing, joining with broader efforts of standardization (e.g., CSSPLICE) and existing initiatives for creating repositories, tools, and materials.

The goal of the session is to help attendees who are committed to diversity and inclusion learn to talk about different dimensions of identity (e.g., race, class, gender, sex, sexuality, etc.). The landscape of terms is always changing and we want SIGCSE attendees to feel more comfortable using current language to talk about issues related to diversity and inclusion. This special session will include six lightning talks, individual reflection, small-group discussion, and a Q&A with presenters. This is a reprise of a special session held at the NCWIT Summit in 2018; it was well received and we hope to offer it to the larger SIGCSE audience.

We believe that more Computing faculty should spend some time in industry in order to experience for themselves the types of jobs their students are likely to take at graduation and throughout their careers. Each panelist has spent at least one sabbatical in industry and found the experience to be far more beneficial than anticipated. We all found that our sabbatical experiences helped us to (1) develop our skills, (2) discover and practice the skills that are currently being used in industry, (3) gain credibility with our students, (4) improve our understanding of the computing profession beyond software development, and (5) build relationships with local businesses. Many of us returned to the classroom thinking "I can't believe they let me teach college without any industry experience".

Undergraduates have been an important part of the teaching staff at many universities for decades, but this is not a universal practice. This BOF provides a forum to discuss undergraduate TA/mentor programs at various stages in their evolution. Attendees will have an opportunity to discuss the benefits, best practices, and research questions related to the use of undergraduates as teaching assistants and/or mentors. The audience is expected to consist of faculty that have already implemented such programs, and those who might be considering doing so, and want to share information about successes and challenges. Discussion Leader(s): Phill Conrad will facilitate the discussion and invite Colleen Lewis, Cynthia Lee and Diba Mirza to briefly speak about the Undergraduate TA/Mentor programs at their respective institutions. During the balance of the time participants will be invited to contribute ideas or ask questions related to the use of undergraduate TAs and mentors. Expertise of Discussion Leader(s): Diba Mirza and Phill Conrad are faculty in the Computer Science department at UC Santa Barbara, where they have created an undergraduate mentor program in the past two years. Colleen Lewis is an Associate Professor of CS at Harvey Mudd College, and is the lead PI for the csteachingtips.org project, a project to collect and document CS pedagogical content knowledge. Cynthia Lee is a Lecturer in CS at Stanford University, and works closely with the longstanding undergraduate TA program in the department.

Replication research is rare in CS education. For this reason, it is often unclear to what extent our findings generalize beyond the context of their generation. The present paper is a replication and extension of Achievement Goal Theory research on CS1 students. Achievement goals are cognitive representations of desired competence (e.g., topic mastery, outperforming peers) in achievement settings, and can predict outcomes such as grades and interest. We study achievement goals and their effects on CS1 students at six institutions in four countries. Broad patterns are maintained --- mastery goals are beneficial while appearance goals are not --- but our data additionally admits fine-grained analyses that nuance these findings. In particular, students' motivations for goal pursuit can clarify relationships between performance goals and outcomes.

A good suite of test inputs is an indispensable tool both for manual and automated assessment of student submissions to programming assignments. Yet, without a way to evaluate our test suites, it is difficult to know how well we are doing, much less improve our practice. We present a technique for evaluating a hand-generated test suite by comparing its ability to find defects against that of a test suite generated automatically using Constraint Logic Programming (CLP). We describe our technique and present a case study using student submissions for an assignment from a second-year programming course. Our results show that a CLP-generated test suite was able to identify significant defects that the instructor-generated suite missed, despite having similar code coverage.

This session will demonstrate best practices for introducing peer instruction and pedagogical code reviews into the computer science classroom. Presenters play the role of instructors on the first day of class. Audience members play the role of students as they collaborate on a variety of sample activities. By providing models of active learning in computing contexts, we seek to motivate instructors to adopt these pedagogies in their classrooms. Sufficient time will be reserved for questions and discussion.

Scratch programming has risen in prominence, not only as a potential language for K-12 computer science, but also in introductory college courses. Unfortunately, grading Scratch programs is time-consuming, requiring manual execution of each program. Automation of this process is greatly complicated by the very reason Scratch is an attractive introductory language--the projects are multimedia in nature, requiring eyes and ears to fully appreciate.

Many institutions have created and deployed outreach programs for middle school students with the goal of increasing the number and diversity of students who later pursue careers in computer science. While these programs have been shown to increase interest in computer science, there has been less work on showing whether participants learn computer science content.

Studio-based learning (SBL), the centerpiece of architecture and fine arts education for over a century, has become increasingly attractive to computing educators. SBL can be conceptualized as an iterative process of solution refinement that relies heavily on shared physical space and design critiques. This session will explore ways in which CS educators can transition aspects of traditional or face-to-face SBL to an online learning environment. We will discuss the potential for online studios, how they might look and function, and how their effectiveness in promoting student learning might be gauged. As more and more CS courses assume an online presence, we seek to understand both the opportunities and the limitations associated with conducting critical studio work in a digital medium.

A popular approach to introducing students to computer science is to involve middle-school students in engaging programming activities. One challenge in such a program is attracting students who are not already positively predisposed to computing.

Reliable server pooling (RSP) allows a pool of redundant information sources to be viewed as a single transport endpoint and, therefore, it is able to provide persistent connections and balanced traffic for different applications. The Internet Engineering Task Force RSerPool Working Group has proposed an architecture to implement the RSP; it defines an overlay network providing an upper layer protocol or an application with a range of reliability services, from simple server selection to a fully automatic session-failover capability. The simulation experiments conducted in both wired and wireless environments show that the current version of the RSerPool works well in fixed and relatively reliable environments, but its performance worsens rapidly as the networks become more unreliable or mobile. The issues we have identified in wireless mobile ad hoc networks include network partition, high signaling overhead, difficulty in synchronization among name servers, and excessive aggressiveness in handling failures. Alternative design options for the RSP in wireless and mobile environments are introduced and evaluated.

This paper presents results from performance experiments that demonstrate and quantify performance improvements when a PO/R transpor5t service is used instead of an ordered/reliable service (O/R e.g., TCP) or an unordered/unreliable service (e.g. UDP). We first describe the Remote Multimedia Document Retrieval system (ReMDoR), an experimental application developed by the authors to evaluate the performance of remote document retrieval over a variety of transport protocols. We then provide a detailed analysis of experiments comparing O/R service to PO/R service for retrieval of a multimedia document. Our results show that between 5% and 10% loss, user-perceivable improvements in progressive display are obtained when PO/R service is used. These results suggest that when packet losses occur in an underlying packet-switched network, transport services providing reliable delivery over independent streams (such the emerging Internet protocol SCTP) are beneficial for retrieval of streaming multimedia.

An analytic model is presented for a partially reliable transport protocol based on retransmissions. The model illustrates tradeoffs between two QoS parameters (delay and throughput), and various levels of reliability. The model predicts that the use of a reliable transport service when an application only needs a partially reliable one causes considerable throughput decreases and delay increases in lossy networks. On the other hand, over lossy networks, an unreliable transport service is unable to respect an application's loss tolerance. In lossy environments, partially reliable transport service avoids the extra cost of a reliable transport service, and, simultaneously, guarantees the minimal reliability that an application requires. A retransmission-based partially reliable transport service can be provided through either sender-based or receiver-based loss detection and recovery. Results show that both techniques provide almost identical reliability and delay. However, a sender-based approach provides better throughput than a receiver-based approach at high acknowledge loss rates.

Many applications such as video and audio can tolerate loss. When the network layer provides a best-effort service such as on the Internet, the loss rate of the underlying network service may be higher than an application's tolerance for loss. This paper analytically studies retransmission-based partially reliable transport (layer protocol) service. Results show that a partially reliable transport service provides increasingly higher throughput and lower delay than a reliable transport service as an application's loss tolerance increases and as the underlying network service gets more lossy. Also, to some degree, a partially reliable transport service eases the negative effects of ack losses on throughput. Three cost functions associated with the reliability level that a system can support are introduced. These cost functions help demonstrate the penalty when a transport service does not support the ideal reliability level for an application. Results show that the use of a reliable transport service when an application only needs a partially reliable transport service can cause considerable throughput drops and delay increases in lossy networks. On the other hand, at high loss rates, an unreliable transport service is unable to respect an application's loss tolerance. Thus, in lossy environments, a partially reliable transport service is necessary to avoid the extra cost of a reliable transport service, and, at the same time, to guarantee the minimal reliability that an application requires.

Investigates a partial-order connection (POC) service/protocol. Unlike classic transport services that deliver objects either in the exact order transmitted or according to no particular order, POC provides a partial-order service, i.e. a service that requires some, but not all objects to be received in the order transmitted. Two versions of POC are proposed: reliable, which requires that all transmitted objects are eventually delivered, and unreliable, which permits the service to lose a subset of the objects. In the unreliable version, objects are more finely categorized into one of three reliability classes depending on their temporal value. Two metrics based on e/sub i/(P), the number of linear extensions of partial-order P in the presence of i lost objects, are proposed as complexity measures of different combinations of partial order and reliability. Formulae for calculating e/sub i/(P) are derived when P is series-parallel. A formal specification of a POC protocol, written in Estelle, is presented and discussed. This specification was designed and validated using formal description tools and provides a basis for future implementations.>