Problem reports at NASA are similar to bug reports: they capture defects found during test, post-launch operational anomalies, and document the investigation and corrective action of the issue. These artifacts are a rich source of lessons learned for NASA, but are expensive to analyze since problem reports are comprised primarily of natural language text. We apply topic modeling to a corpus of NASA problem reports to extract trends in testing and operational failures. We collected 16,669 problem reports from six NASA space flight missions and applied Latent Dirichlet Allocation topic modeling to the document corpus. We analyze the most popular topics within and across missions, and how popular topics changed over the lifetime of a mission. We find that hardware material and flight software issues are common during the integration and testing phase, while ground station software and equipment issues are more common during the operations phase. We identify a number of challenges in topic modeling for trend analysis: 1) that the process of selecting the topic modeling parameters lacks definitive guidance, 2) defining semantically-meaningful topic labels requires non-trivial effort and domain expertise, 3) topic models derived from the combined corpus of the six missions were biased toward the larger missions, and 4) topics must be semantically distinct as well as cohesive to be useful. Nonetheless, topic modeling can identify problem themes within missions and across mission lifetimes, providing useful feedback to engineers and project managers.

Before a community can learn general principles, it must share individual experiences. Data sharing is the fundamental step of cross project defect prediction, i.e. the process of using data from one project to predict for defects in another. Prior work on secure data sharing allowed data owners to share their data on a single-party basis for defect prediction via data minimization and obfuscation. However the studied method did not consider that bigger data required the data owner to share more of their data. In this paper, we extend previous work with LACE2 which reduces the amount of data shared by using multi-party data sharing. Here data owners incrementally add data to a cache passed among them and contribute "interesting" data that are not similar to the current content of the cache. Also, before data owner i passes the cache to data owner j, privacy is preserved by applying obfuscation algorithms to hide project details. The experiments of this paper show that (a) LACE2 is comparatively less expensive than the single-party approach and (b) the multi-party approach of LACE2 yields higher privacy than the prior approach without damaging predictive efficacy (indeed, in some cases, LACE2 leads to better defect predictors).

Determining whether systems achieve desired emergent properties, such as safety or reliability, requires an analysis of the system as a whole, often in later development stages when changes are difficult and costly to implement. In this article we propose the Process Risk Indicator (PRI) methodology for analyzing and evaluating emergent properties early in the development cycle. A fundamental assumption of system engineering is that risk mitigation processes reduce system risks, yet these processes may also be a source of risk: (1) processes may not be appropriate for achieving the desired emergent property; or (2) processes may not be followed appropriately. PRI analyzes development process artifacts (e.g., designs pertaining to reliability or safety analysis reports) to quantify process risks that may lead to higher system risk. We applied PRI to the hazard analysis processes of a network-centric, Department of Defense system-of-systems and two NASA spaceflight projects to assess the risk of not achieving one such emergent property, software safety, during the early stages of the development lifecycle. The PRI methodology was used to create measurement baselines for process indicators of software safety risk, to identify risks in the hazard analysis process, and to provide feedback to projects for reducing these risks.

We know surprisingly little about how professional developers define debugging and the challenges they face in industrial environments. To begin exploring professional debugging challenges and needs, we conducted and analyzed interviews with 15 professional software engineers at Microsoft. The goals of this study are: 1) to understand how professional developers currently use information and tools to debug, 2) to identify new challenges in debugging in contemporary software development domains (web services, multithreaded/multicore programming), and 3) to identify the improvements in debugging support desired by these professionals that are needed from research. The interviews were coded to identify the most common information resources, techniques, challenges, and needs for debugging as articulated by the developers. The study reveals several debugging challenges faced by professionals, including: 1) the interaction of hypothesis instrumentation and software environment as a source of debugging difficulty, 2) the impact of log file information on accurate debugging of web services, and 3) the mismatch between the sequential human thought process and the non-sequential execution of multithreaded environments as source of difficulty. The interviewees also describe desired improvements to tools to support debugging, many of which have been discussed in research but not transitioned to practice.

Amazon's Mechanical Turk is a crowd sourcing technology that enables requesters to create tasks to be completed by human agents in exchange for compensation. Researchers in computer science have successfully used this service to quickly reach large numbers of subjects for a relatively low cost. However, the Mechanical Turk's model and policies introduce several experimental limitations and threats that must be controlled. In this short paper, we describe limitations imposed using Amazon's Mechanical Turk during an experiment on cyber-attack investigation techniques. While the experiment was successful, we were forced to change our experimental design and had to recover from some costly mistakes. The goal of this short paper is to identify these limitations and pitfalls and provide eight considerations for experimental design so that other researchers can maximize the benefits of using the Mechanical Turk as a research platform.

Existing research is unclear on how to generate lessons learned for defect prediction and effort estimation. Should we seek lessons that are global to multiple projects or just local to particular projects? This paper aims to comparatively evaluate local versus global lessons learned for effort estimation and defect prediction. We applied automated clustering tools to effort and defect datasets from the PROMISE repository. Rule learners generated lessons learned from all the data, from local projects, or just from each cluster. The results indicate that the lessons learned after combining small parts of different data sources (i.e., the clusters) were superior to either generalizations formed over all the data or local lessons formed from particular projects. We conclude that when researchers attempt to draw lessons from some historical data source, they should 1) ignore any existing local divisions into multiple sources, 2) cluster across all available data, then 3) restrict the learning of lessons to the clusters from other sources that are nearest to the test data.

In this case study, we examine software safety risk in three flight hardware systems in NASA's Constellation spaceflight program. We applied our Technical and Process Risk Measurement (TPRM) methodology to the Constellation hazard analysis process to quantify the technical and process risks involving software safety in the early design phase of these projects. We analyzed 154 hazard reports and collected metrics to measure the prevalence of software in hazards and the specificity of descriptions of software causes of hazardous conditions. We found that 49-70% of 154 hazardous conditions could be caused by software or software was involved in the prevention of the hazardous condition. We also found that 12-17% of the 2013 hazard causes involved software, and that 23-29% of all causes had a software control. The application of the TRPM methodology identified process risks in the application of the hazard analysis process itself that may lead to software safety risk.

Art. 46

Code churn, the amount of code change taking place within a software unit over time, has been correlated with fault-proneness in software systems. We investigate the use of code churn and static metrics collected at regular time intervals during the development cycle to predict faults in an iterative, in-process manner. We collected 159 churn and structure metrics from six, four-month snapshots of a 1 million LOC Microsoft product. The number of software faults fixed during each period is recorded per binary module. Using stepwise logistic regression, we create a prediction model to identify fault-prone binaries using three parameters: code churn (the number of new and changed blocks); class Fan In and class Fan Out (normalized by lines of code). The iteratively-built model is 80.0% accurate at predicting fault-prone and non-fault-prone binaries. These fault-prediction models have the advantage of allowing the engineers to observe how their fault-prediction profile evolves over time.

In the software development process, scheduling and predictability are important components to delivering a product on time and within budget. Effort estimation artifacts offer a rich data set for improving scheduling accuracy and for understanding the development process. Effort estimation data for 55 features in the latest release of Visual Studio Team System (VSTS) were collected and analyzed for trends, patterns, and differences. Statistical analysis shows that actual estimation error was positively correlated with feature size, and that in-process metrics of estimation error were also correlated with the final estimation error. These findings suggest that smaller features can be estimated more accurately, and that in-process estimation error metrics can be provide a quantitative supplement to developer intuition regarding high-risk features during the development process.

Despite many professed benefits of collaboration, some computer science educators feel students need to master work individually, particularly in the courses early in the curriculum that feed into software engineering courses. In the natural sciences, however, students almost always work with one or more partners in the laboratory. What can computer science educators learn about collaborative lab settings from our natural science counterparts? We conducted a survey of science and computer science educators to compare views and use of collaboration in their classes. The positive and negative aspects of collaboration, as reported by the natural science educators, are strikingly similar to those of computer science educators. These results suggest that computer science educators should be more open to the use of collaborative labs, as is done in the natural sciences, for the overall benefit to students.

The longer a fault remains in the code from the time it was injected, the more time it will take to fix the fault. Increasingly, automated fault detection (AFD) tools are providing developers with prompt feedback on recently-introduced faults to reduce fault fix time. If however, the frequency and content of this feedback does not match the developer's goals and/or workflow, the developer may ignore the information. We conducted a controlled study with 18 developers to explore what factors are used by developers to decide whether or not to address a fault when notified of the error. The findings of our study lead to several conjectures about the design of AFD tools to effectively notify developers of faults in the coding phase. The AFD tools should present fault information that is relevant to the primary programming task with accurate and precise descriptions. The fault severity and the specific timing of fault notification should be customizable. Finally, the AFD tool must be accurate and reliable to build trust with the developer.

Millennial students (those born after 1982), particularly African Americans and women, have demonstrated a propensity toward collaborative activities. We conducted a collective case study at North Carolina State University and North Carolina A&T to ascertain the role of collaboration and social interaction in attracting and retaining students in information technology. Responses from semi-structured interviews with 11 representative African American students in these classes were coded and analyzed. The responses from these minority students were used to evolve a social interaction model. The conjectures generated from the model suggest that pair programming and agile software methodologies effectively create a collaborative environment that is desirable to Millennial students, male and female, and, with the new evidence, minority and majority. Additionally, the African American Millennial students enjoy learning from their peers and believe that a collaborative environment better prepares them for the "real world".

In addition to "learning by doing," programming assignments and projects are also the mechanism by which students learn about the utility of computer science -- or not. Recent research indicates that the current generation of students is in search of a career with meaning, and women and minorities have long been known to desire careers that help society. In this paper, we provide student testimonials on the importance and benefits of practical and socially-relevant assignments. We then examined approximately 200 first year (CS1) and software engineering assignments at top computer science institutions. Only 34% of the CS1 projects had a practical or socially-relevant context, 41% had no context at all, and 15% were games. For software engineering projects, 62% were practical or socially-relevant, but still 16% had no practical context. We recommend that educators, through their assignments, place increased emphasis on demonstrating that computer science can be used to aid society and/or produce products of practical value to society.

Collaborative work has been in use as an instructional tool to increase student understanding through collaborative learning and to improve student performance in computer science courses. However, little work has been done to understand how the act of collaboration, through pair programming or group work, impacts a student's knowledge of the benefits and difficulties of collaborative work experience in collaborative work is essential preparation for professional software development. A study was conducted at North Carolina State University to assess changes in advanced undergraduate students' perceptions of pair programming and collaboration. Student personality types, learning styles, and other characteristics were gathered during two semesters of an undergraduate software engineering course. The study found that, after experiencing pair programming, most students indicated a stronger preference to work with another student, believed that pairing made them more organized, and believed that pairing saved time on homework assignments. Students who disliked their collaborative experiences were predominantly reflective learners, introverts, and strong coders. Those students also cited that a non-participatory partner and difficulties scheduling meeting times outside of the classroom were primary reasons for disliking pair programming. Personality type and learning style had little effect on the changes in perceptions of collaboration

This paper describes an initiative at North Carolina State University in which the undergraduate software engineering class was restructured in layout and in presentation. The change was made from a lecture-based coursed that followed the waterfall method to a lab-oriented course emphasizing practical tools and agile processes. We examine the new course layout from the perspective of Myers-Briggs personality types and Felder-Silverman learning styles to discuss how the new software engineering class format appeals to a wide variety of students. The new course format resulted in some of the highest student evaluations in recent course history. It is now the standard for the undergraduate software engineering course at the university and has since been used in other North Carolina institutions.