Developers frequently move into new teams or environments across software companies. Their onboarding experience is correlated with productivity, job satisfaction, and other short-term and long-term outcomes. The majority of the onboarding process comprises engineering tasks such as fixing bugs or implementing small features. Nevertheless, we do not have a systematic view of how tasks influence onboarding. In this paper, we present a case study of Microsoft, where we interviewed 32 developers moving into a new team and 15 engineering managers onboarding a new developer into their team - to understand and characterize developers' onboarding experience and expectations in relation to the tasks performed by them while onboarding. We present how tasks interact with new developers through three representative themes: learning, confidence building, and socialization. We also discuss three onboarding strategies as inferred from the interviews that managers commonly use unknowingly, and discuss their pros and cons and offer situational recommendations. Furthermore, we triangulate our interview findings with a developer survey (N = 189) and a manager survey (N = 37) and find that survey results suggest that our findings are representative and our recommendations are actionable. Practitioners could use our findings to improve their onboarding processes, while researchers could find new research directions from this study to advance the understanding of developer onboarding. Our research instruments and anonymous data are available at https://zenodo.org/record/4455937#.YCOQCs 0lFd.

While project-based software engineering courses aim to provide learning opportunities grounded in professional processes, it is not always possible to replicate every process in classrooms due to course constraints. Previous studies observed how students react to various processes and gave retroactive recommendations. In this study, we instead combine a field study on professional Agile (eXtreme Programming, XP) teams and an established team process taxonomy to proactively select team processes to incorporate in a project-based software engineering course. With collected knowledge from the field study, we choose three XP processes to augment the design of a mature software engineering project course. We choose processes that are 1) considered important by professionals, and 2) complete with respect to coverage of the taxonomy's main categories. We then compare the augmented course design with the original design in a case study. Our results suggest that 1) even without extra resources, adding these new processes does not interfere with learning opportunities for XP processes previously existing in the course design; 2) student teams experience similar benefits from these new processes as professional teams do, and students appreciate the usefulness and value of the processes. In other words, our approach allows instructors to make conscious choices of XP processes that improve student learning outcomes while exposing students to a more complete set of processes and thus preparing them better for professional careers. Course designers with limited resources are encouraged to use our methodology to evaluate and improve the designs of their own project-based courses.

Agile/XP (Extreme Programming) software teams are expected to follow a number of specific practices in each iteration, such as estimating the effort (”points”) required to complete user stories, properly using branches and pull requests to coordinate merging multiple contributors’ code, having frequent ”standups” to keep all team members in sync, and conducting retrospectives to identify areas of improvement for future iterations. We combine two observations in developing a methodology and tools to help teams monitor their performance on these practices. On the one hand, many Agile practices are increasingly supported by web-based tools whose ”data exhaust” can provide insight into how closely the teams are following the practices. On the other hand, some of the practices can be expressed in terms similar to those developed for expressing service level objectives (SLO) in software as a service; as an example, a typical SLO for an interactive Web site might be ”over any 5-minute window, 99% of requests to the main page must be delivered within 200ms” and, analogously, a potential Team Practice (TP) for an Agile/XP team might be ”over any 2-week iteration, 75% of stories should be ’1-point’ stories”. Following this similarity, we adapt a system originally developed for monitoring and visualizing service level agreement (SLA) compliance to monitor selected TPs for Agile/XP software teams. Specifically, the system consumes and analyzes the data exhaust from widely-used tools such as GitHub and Pivotal Tracker and provides team(s) and coach(es) a ”dashboard” summarizing the teams’ adherence to various practices. As a qualitative initial investigation of its usefulness, we deployed it to twenty student teams in a four-sprint software engineering project course. We find an improvement of the adherence to team practice and a positive students’ self-evaluations of their team practices when using the tool, compared to previous experiences using an Agile/XP methodology. The demo video is located at https://youtu.be/A4xwJMEQh9c and a landing page with a live demo at https://isa-group.github.io/2019-05-eagle-demo/.

Project-based learning is an important teaching method in software engineering education. However, it is unclear how student projects can be evaluated objectively and systematically in classrooms. Measurements used in industry, such as quality of the codebase, are not the only expected outcomes in classrooms; informative assessments in project-based learning require more details about how students behave as individuals and as a team. In this paper, we establish the importance of measuring processes in project-based software engineering courses and present metrics mined from software development tools for monitoring and observing processes to facilitate teaching. A case study at a US university confirms that 1) teams with better conformance to software development processes achieve better outcomes, and 2) our approach can be used to design metrics that serve as early detectors of violations to software development processes. Our results suggest that instructors for software engineering courses can use our approach to design process metrics for systematic, targeted, and automatic evaluation of team projects. Furthermore, metrics designed using our approach can be used as building blocks for automated systems, and thus increase the scalability of project-based software engineering courses.

Pencil-and-paper coding questions on computer science exams are unrealistic: real developers work at a keyboard with extensive resources at hand, rather than on paper with few or no notes. We address the challenge of administering a proctored exam in which students must write code that passes instructor-provided test cases as well as writing test cases of their own. The exam environment allows students broad access to Internet resources they would use for take-home programming assignments, while blocking their ability to use that facility for direct communication with colluders. Our system supports cumulative questions (in which later parts depend on correctly answering earlier parts) by allowing the test-taker to reveal one or more hints by sacrificing partial credit for the question. Autograders built into the exam environment provide immediate feedback to the student on their exam grade. In case of grade disputes, a virtual machine image reflecting all of the student's work is preserved for later inspection. While elements of our scheme have appeared in the literature (autograding, semi-locked-down computer environments for exam-taking, hint "purchasing"), we believe we are the first to combine them into a system that enables realistic in-class coding-based exams with broad Internet access. We report on lessons and experience creating and administering such an exam, including autograding-related pitfalls for high-stakes exams, and invite others to use and improve on our tools and methods.

Constructed-response, code-tracing questions ("What would Python print?") are good formative assessments. Unlike selected-response questions simply marked correct or incorrect, a constructed wrong answer can provide information on a student's particular difficulty. However, constructed-response questions are resource-intensive to grade manually, and machine grading yields only correct/incorrect information. We analyzed incorrect constructed responses from code-tracing questions in an introductory computer science course to investigate whether a small subsample of such responses could provide enough information to make inspecting the subsample worth the effort, and if so, how best to choose this subsample. In addition, we sought to understand what insights into student difficulties could be gained from such an analysis.

Teaching software development teams can be difficult to scale. Based on various cloud-based software development tools, Teamscope provides automated or semi-automated metrics to improve the scalability of a course with team projects. Metrics developed in Teamscope provide a synthesized view of a student team. Our preliminary results have shown the validity of these metrics. We also present a case study of applying metrics to teaching software development course in this paper.

In contrast to multiple-choice or selected response questions, constructed response questions can result in a wide variety of incorrect responses. However, constructed responses are richer in information. We propose a technique for using each student's constructed responses in order to identify a subset of their stable conceptual misunderstandings. Our approach is designed for courses with so many students that it is infeasible to interpret every distinct wrong answer manually. Instead, we label only the most frequent wrong answers with the misunderstandings that they indicate, then predict the misunderstandings associated with other wrong answers using statistical co-occurrence patterns. This tiered approach leverages a small amount of human labeling effort to seed an automated procedure that identifies misunderstandings in students. Our approach involves much less effort than inspecting all answers, substantially outperforms a baseline that does not take advantage of co-occurrence statistics, proves robust to different course sizes, and generalizes effectively across student cohorts.