Summary Concurrent programs are difficult to debug because concurrency faults usually occur under specific inputs and thread interleavings. Fault localization techniques for sequential programs are often ineffective because the root causes of concurrency faults involve memory accesses across multiple threads rather than single statements. Previous research has proposed techniques to analyse passing and failing executions obtained from running a set of test cases for identifying faulty memory access patterns. However, stand‐alone access patterns do not provide enough contextual information, such as the path leading to the failure, for developers to understand the bug. We present an approach, Coadec, to automatically generate interthread control flow paths that can link memory access patterns that occurred most frequently in the failing executions to better diagnose concurrency bugs. Coadec consists of two phases. In the first phase, we use feature selection techniques from machine learning to localize suspicious memory access patterns based on failing and passing executions. The patterns with maximum feature diversity information can point to the most suspicious pattern. We then apply a data mining technique and identify the memory access patterns that occurred most frequently in the failing executions. Finally, Coadec identifies faulty program paths by connecting both the frequent patterns and the suspicious pattern. We also evaluate the effectiveness of fault localization using test suites generated from different test adequacy criteria. We introduce and have evaluated Coadec on 10 real‐world multithreaded Java applications. Results indicate that Coadec outperforms state‐of‐the‐art approaches for localizing concurrency faults and that Coadec's context debugging can help developers understand concurrency fault by inspecting a small percentage of code.

Concurrent programs are difficult to test due to their inherent non-determinism. To address this problem, testing often requires the exploration of thread schedules of a program; this can be time-consuming when applied to real-world programs. Software defect prediction has been used to help developers find faults and prioritize their testing efforts. Prior studies have used machine learning to build such predicting models based on designed features that encode the characteristics of programs. However, research has focused on sequential programs; to date, no work has considered defect prediction for concurrent programs, with program characteristics distinguished from sequential programs. In this paper, we present ConPredictor, an approach to predict defects specific to concurrent programs by combining both static and dynamic program metrics. Specifically, we propose a set of novel static code metrics based on the unique properties of concurrent programs. We also leverage additional guidance from dynamic metrics constructed based on mutation analysis. Our evaluation on four large open source projects shows that ConPredictor improved both within-project defect prediction and cross-project defect prediction compared to traditional features.

This paper presents the REquirements TRacing On target (RETRO).NET dataset. The dataset includes the requirement specification, the source code files (C# and Visual Basic), the gold standard/answer set for tracing the artifacts to each other, as well as the script used to parse the requirements from the specification (to put in RETRO.NET format). The dataset can be used to support tracing and other tasks.

The goal of RE Cares is to apply our requirements engineering and design and prototyping skills to a problem of societal importance to stakeholders residing in the RE conference locale.

Empirical validation is an important component of sound requirements engineering research. Many researchers develop a gold standard or answer set against which to compare techniques that they also developed in order to calculate common measures such as recall and precision. This poses threats to validity as the researchers developed the gold standard and the technique to be measured against it. To help address this and to help reduce bias, we introduce a prototype of Multi-user Input in Determining Answer Sets (MIDAS), a web-based tool to permit communities of researchers to jointly determine the gold standard for a given research data set. To date, the tool permits community members to add items to the answer set, vote on items in the answer set, comment on items, and view the latest status of community opinion on answer set items. It currently supports traceability data sets and classification data sets.

Automated traceability has been investigated for over a decade with promising results. However, a human analyst is needed to vet the generated trace links to ensure their quality. The process of vetting trace links is not trivial and while previous studies have analyzed the performance of the human analyst, they have not focused on the analyst's information needs. The aim of this study is to investigate what context information the human analyst needs. We used design science research, in which we conducted interviews with ten practitioners in the traceability area to understand the information needed by human analysts. We then compared the information collected from the interviews with existing literature. We created a prototype tool that presents this information to the human analyst. To further understand the role of context information, we conducted a controlled experiment with 33 participants. Our interviews reveal that human analysts need information from three different sources: 1) from the artifacts connected by the link, 2) from the traceability information model, and 3) from the tracing algorithm. The experiment results show that the content of the connected artifacts is more useful to the analyst than the contextual information of the artifacts.

Concurrent programs are difficult to test due to their inherent non-determinism. To address the nondeterminism problem, testing often requires the exploration of thread schedules of a program, this can be time-consuming for testing real-world programs. We believe that testing resources can be distributed more effectively if testability of concurrent programs can be estimated, so that developers can focus on exploring the low testable code. Voas introduces a notion of testability as the probability that a test case will fail if the program has a fault, in which testability can be measured based on fault-based testing and mutation analysis. Much research has been proposed to analyze testability and predict defects for sequential programs, but to date, no work has considered testability prediction for concurrent programs, with program characteristics distinguished from sequential programs. In this paper, we present an approach to predict testability of concurrent programs at the function level. We propose a set of novel static code metrics based on the unique properties of concurrent programs. To evaluate the performance of our approach, we build a family of testability prediction models combining both static metrics and a test suite metric and apply it to real projects. Our empirical study reveals that our approach is more accurate than existing sequential program metrics.

Configuration bugs are among the dominant causes of software failures. Software organizations often use bug tracking systems to manage bug reports collected from developers and users. In order for software developers to understand and reproduce configuration bugs, it is vital for them to know whether a bug in the bug report is related to configuration issues, this is not often easily discerned due to a lack of easy to spot terminology in the bug reports. In addition, to locate and fix a configuration bug, a developer needs to know which configuration options are associated with the bug. To address these two problems, we introduce CoLUA, a two-step automated approach that combines natural language processing, information retrieval, and machine learning. In the first step, CoLUA selects features from the textual information in the bug reports, and uses various machine learning techniques to build classification models, developers can use these models to label a bug report as either a configuration bug report or a non-configuration bug report. In the second step, CoLUA identifies which configuration options are involved in the labeled configuration bug reports. We evaluate CoLUA on 900 bug reports from three large open source software systems. The results show that CoLUA predicts configuration bug reports with high accuracy and that it effectively identifies the root causes of configuration options.

Software project artifacts such as source code, requirements, and change logs represent a gold-mine of actionable information. As a result, software analytic solutions have been developed to mine repositories and answer questions such as "who is the expert?," "which classes are fault prone?," or even "who are the domain experts for these fault-prone classes?" Analytics often require training and configuring in order to maximize performance within the context of each project. A cold-start problem exists when a function is applied within a project context without first configuring the analytic functions on project-specific data. This scenario exists because of the non-trivial effort necessary to instrument a project environment with candidate tools and algorithms and to empirically evaluate alternate configurations. We address the cold-start problem by comparatively evaluating 'best-of-breed' and 'profile-driven' solutions, both of which reuse known configurations in new project contexts. We describe and evaluate our approach against 20 project datasets for the three analytic areas of artifact connectivity, fault-prediction, and finding the expert, and show that the best-of-breed approach outperformed the profile-driven approach in all three areas; however, while it delivered acceptable results for artifact connectivity and find the expert, both techniques underperformed for cold-start fault prediction.

Abstract Several techniques are used by requirements engineering practitioners to address difficult problems such as specifying precise requirements while using inherently ambiguous natural language text and ensuring the consistency of requirements. Often, these problems are addressed by building processes/tools that combine multiple techniques where the output from 1 technique becomes the input to the next. While powerful, these techniques are not without problems. Inherent errors in each technique may leak into the subsequent step of the process. We model and study 1 such process, for checking the consistency of temporal requirements, and assess error leakage and wasted time. We perform an analysis of the input factors of our model to determine the effect that sources of uncertainty may have on the final accuracy of the consistency checking process. Convinced that error leakage exists and negatively impacts the results of the overall consistency checking process, we perform a second simulation to assess its impact on the analysts' efforts to check requirements consistency. We show that analyst's effort varies depending on the precision and recall of the subprocesses and that the number and capability of analysts affect their effort. We share insights gained and discuss applicability to other processes built of piped techniques.

This paper describes ongoing work to characterize the inherent ease or “traceability” with which a textual artifact can be traced using an automated technique. Software traceability approaches use varied measures to build models that automatically recover links between pairs of natural language documents. Thus far, most of the approaches use a single-step model, such as logistic regression, to identify new trace links. However, such approaches require a large enough training set of both true and false trace links. Yet, the former are by far in the minority, which reduces the performance of such models. Therefore, this paper formulates the problem of identifying trace links as the problem of finding, for a given logistic regression model, the subsets of links in the training set giving the best accuracy (in terms of G-metric) on a test set. Using hill climbing with random restart for subset selection, we found that, for the ChangeStyle dataset, we can classify links with a precision of up to 40% and a recall of up to 66% using a training set as small as one true candidate link (out of 33) and 41 false links. To get better performance and learn the best possible logistic regression classifier, we must “discard” links in the trace dataset that increase noise to avoid learning with links that are not representative. This preliminary work is promising because it shows that few correct examples may perform better than several poor ones. It also shows which inherent characteristics of the artifacts make them good candidates to learn efficient traceability models automatically, i.e., it reveals their traceability.

Researchers in software testing are often faced with the following problem of empirical validation: does a new testing technique actually help analysts find more faults than some baseline method? Researchers evaluate their contribution using statistics to refute the null hypothesis that their technique is no better at finding faults than the state of the art. The decision as to which statistical methods are appropriate is best left to an expert statistician, but the reality is that software testing researchers often don't have this luxury. We developed an algorithm, Means Test, to help automate some aspects of statistical analysis. We implemented Means Test in the statistical software environment R, encouraging reuse and decreasing the need to write and test statistical analysis code. Our experiment showed that Means Test has significantly higher F-measures than several other common hypothesis tests. We applied Means Test to systematically validate the work presented at the 2013 IEEE Sixth International Conference on Software Testing, Verification, and Validation (ICST'13). We found six papers that potentially misstated the significance of their results. Means Test provides a free and easy-to-use possibility for researchers to check whether their chosen statistical methods and the results obtained are plausible. It is available for download at coest.org.

Though the primary goal of requirements engineering research is to propose, develop, and validate effective solutions for important practical problems, practice has shown that successful projects take from 20–25 years to reach full industry adoption, while many projects fade and never advance beyond the initial research phase. In this interactive panel, teams of researchers, representing different requirements engineering research areas, bring ideas for technology transfer to a panel of industrial and government practitioners. The teams make interactive presentations and receive feedback from panelists. Beneath the game-show genre of the panel is the serious goal to foster conversation between practitioners and researchers to improve the effectiveness of technology transfer in the requirements engineering community.

As Requirements Engineering research continues to grow into a mature and rigorous discipline, an increasing focus is placed on the need for sound evaluation techniques that compare the benefits of a new solution against existing ones. In this tool demonstration we introduce TraceLab, an instrumented environment for modeling, executing, and comparatively evaluating experimental results. While initially developed for the Software Traceability domain, TraceLab provides a framework which can be populated with experiments, datasets, and reusable components for almost any empirical software engineering domain. In this demo we present examples from the Requirements Engineering domain.

We posit that machine learning can be applied to effectively address requirements engineering problems. Specifically, we present a requirements traceability method based on the machine learning technique Reinforcement Learning (RL). The RL method demonstrates a rather targeted generation of candidate links between textual requirements artifacts (high level requirements traced to low level requirements, for example). The technique has been validated using two real-world datasets from two problem domains. Our technique demonstrated statistically significant better results than the Information Retrieval technique.

Software Engineering challenges and contests are becoming increasingly popular for focusing researchers' efforts on particular problems. Such contests tend to follow either an exploratory model, in which the contest holders provide data and ask the contestants to discover “interesting things” they can do with it, or task-oriented contests in which contestants must perform a specific task on a provided dataset. Only occasionally do contests provide more rigorous evaluation mechanisms that precisely specify the task to be performed and the metrics that will be used to evaluate the results. In this paper, we propose actionable and crowd-sourced contests: actionable because the contest describes a precise task, datasets, and evaluation metrics, and also provides a downloadable operating environment for the contest; and crowd-sourced because providing these features creates accessibility to Information Technology hobbyists and students who are attracted by the challenge. Our proposed approach is illustrated using research challenges from the software traceability area as well as an experimental workbench named TraceLab.

TraceLab is designed to empower future traceability research, through facilitating innovation and creativity, increasing collaboration between researchers, decreasing the startup costs and effort of new traceability research projects, and fostering technology transfer. To this end, it provides an experimental environment in which researchers can design and execute experiments in TraceLab's visual modeling environment using a library of reusable and user-defined components. TraceLab fosters research competitions by allowing researchers or industrial sponsors to launch research contests intended to focus attention on compelling traceability challenges. Contests are centered around specific traceability tasks, performed on publicly available datasets, and are evaluated using standard metrics incorporated into reusable TraceLab components. TraceLab has been released in beta-test mode to researchers at seven universities, and will be publicly released via CoEST.org in the summer of 2012. Furthermore, by late 2012 TraceLab's source code will be released as open source software, licensed under GPL. TraceLab currently runs on Windows but is designed with cross platforming issues in mind to allow easy ports to Unix and Mac environments.

Traceability underlies many important software and systems engineering activities, such as change impact analysis and regression testing. Despite important research advances, as in the automated creation and maintenance of trace links, traceability implementation and use is still not pervasive in industry. A community of traceability researchers and practitioners has been collaborating to understand the hurdles to making traceability ubiquitous. Over a series of years, workshops have been held to elicit and enhance research challenges and related tasks to address these shortcomings. A continuing discussion of the community has resulted in the research roadmap of this paper. We present a brief view of the state of the art in traceability, the grand challenge for traceability and future directions for the field.

Human analysts working with results from automated traceability tools often make incorrect decisions that lead to lower quality final trace matrices. As the human must vet the results of trace tools for mission- and safety-critical systems, the hopes of developing expedient and accurate tracing procedures lies in understanding how analysts work with trace matrices. This paper describes a study to understand when and why humans make correct and incorrect decisions during tracing tasks through logs of analyst actions. In addition to the traditional measures of recall and precision to describe the accuracy of the results, we introduce and study new measures that focus on analyst work quality: potential recall, sensitivity, and effort distribution. We use these measures to visualize analyst progress towards the final trace matrix, identifying factors that may influence their performance and determining how actual tracing strategies, derived from analyst logs, affect results.

Our research group recently discovered that human analysts, when asked to validate candidate traceability matrices, produce predictably imperfect results, in some cases less accurate than the starting candidate matrices. This discovery radically changes our understanding of how to design a fast, accurate and certifiable tracing process that can be implemented as part of software assurance activities. We present our vision for the new approach to achieving this goal. Further, we posit that human fallibility may impact other software engineering activities involving decision support tools.

Smart phones, gaming consoles, and wireless routers are ubiquitous; the increasing diffusion of such devices with limited resources, together with society's unsatiated appetite for new applications, pushes companies to miniaturize their programs. Miniaturizing a program for a hand-held device is a time-consuming task often requiring complex decisions. Companies must accommodate conflicting constraints: customers' satisfaction with features may be in conflict with a device's limited storage, memory, or battery life. This paper proposes a process, MoMS, for the multi-objective miniaturization of software to help developers miniaturize programs while satisfying multiple conflicting constraints. It can be used to support the reverse engineering, next release problem, and porting of both software and product lines. The process directs the elicitation of customer pre-requirements, their mapping to program features, and the selection of the features to port. We present two case studies based on Pooka, an email client, and SIP Communicator, an instant messenger, to demonstrate that MoMS supports optimized miniaturization and helps reduce effort by 77%, on average, over a manual approach.

Assisted requirements tracing is a process in which a human analyst validates candidate traces produced by an automated requirements tracing method or tool. The assisted requirements tracing process splits the difference between the commonly applied time-consuming, tedious, and error-prone manual tracing and the automated requirements tracing procedures that are a focal point of academic studies. In fact, in software assurance scenarios, assisted requirements tracing is the only way in which tracing can be at least partially automated. In this paper, we present the results of an extensive 12 month study of assisted tracing, conducted using three different tracing processes at two different sites. We describe the information collected about each study participant and their work on the tracing task, and apply statistical analysis to study which factors have the largest effect on the quality of the final trace.

Abstract Fault links represent relationships between the types of code faults, or defects, and the types of components in which faults are detected. For example, our prior work validated that a fault link exists between Controller components and Control/Logic faults (such as unreachable code). Fault link information can guide code reviews, walkthroughs, testing, maintenance, and can advise fault seeding. In this paper, we use fault links to augment code reviews. Two experiments were undertaken to evaluate the usefulness of fault links, one with 26 Computer Science students and another with 24 software engineering professionals. The first experiment showed that fault link information assisted in finding more total defects and more ‘hard to detect’ defects, in the same amount of time, in a Java component of an online course management application. The experiment was repeated with professionals, adding a second Java component from the same application. For the second experiment, more total defects were found by the participants using fault link information for one of the two components and more hard to detect defects were found, in the same amount of time, in both Java components. The group using fault link information for code walkthroughs found, on average, 1.7–2 times more faults and 2–3 times more hard faults than the control group. Copyright © 2010 John Wiley & Sons, Ltd.

The requirements traceability matrix (RTM) supports many software engineering and software verification and validation (V&V) activities such as change impact analysis, reverse engineering, reuse, and regression testing. The generation of RTMs is tedious and error-prone, though, thus RTMs are often not generated or maintained. Automated techniques have been developed to generate candidate RTMs with some success. When using RTMs to support the V&V of mission-or safety-critical systems, however, a human analyst must vet the candidate RTMs. The focus thus becomes the quality of the final RTM. This paper investigate show human analysts perform when vetting candidate RTMs. Specifically, a study was undertaken at two universities and had 26 participants analyze RTMs of varying accuracy for a Java code formatter program. The study found that humans tend to move their candidate RTM toward the line that represents recall = precision. Participants who examined RTMs with low recall and low precision drastically improved both.

We posit that swarm intelligence can be applied to effectively address requirements engineering problems. Specifically, this paper demonstrates the applicability of swarm intelligence to the requirements tracing problem using a simple ant colony algorithm. The technique has been validated using two real-world datasets from two problem domains. The technique can generate requirements traceability matrices (RTMs) between textual requirements artifacts (high level requirements traced to low level requirements, for example) with equivalent or better accuracy than traditional information retrieval techniques.

Abstract Software developers have individual styles of programming. This paper empirically examines the validity of theconsistent programmer hypothesis: that a facet or set of facets exist that can be used to recognize the author of a given program based on programming style. The paper further postulates that the programming style means that different test strategies work better for some programmers (or programming styles) than for others. For example, all‐edges adequate tests may detect faults for programs written by Programmer A better than for those written by Programmer B. This has several useful applications: to help detect plagiarism/copyright violation of source code, to help improve the practical application of software testing, and to help pursue specific rogue programmers of malicious code and source code viruses. This paper investigates this concept by experimentally examining whether particular facets of the program can be used to identify programmers and whether testing strategies can be reasonably associated with specific programmers. Copyright © 2009 John Wiley & Sons, Ltd.

This paper introduces the automation of satisfaction assessment: the process of determining the satisfaction mapping of natural language textual requirements to natural language design elements. Satisfaction assessment is useful because it assists in discovering unsatisfied requirements early in the lifecycle when such issues can be corrected with lower cost and impact than later. We define the basic terms and concepts for this process and explore the feasibility of developing baseline methods for its automation. This paper describes the satisfaction assessment approach algorithmically and then evaluates the effectiveness of two proposed information retrieval (IR) methods in two industrial studies - one based on a large dataset including a complete requirements specification and design specification for a NASA science instrument, and one based on a smaller dataset for an open source project management dataset. We found that both approaches have merit, and that the more sophisticated approach outperformed the simpler approach in terms of overall accuracy of the results.

Some researchers have had the good fortune to collaborate with practitioners to see their work applied. Some practitioners have had the opportunity to perform research. Some educators have the opportunity to perform research and apply ideas in the classroom. Few have been able to perform research, work with practitioners, be an educator, and be a practitioner. Nancy Mead has done all these, and software engineering education has been one of the fortunate benefactors. This paper presents an examination of Dr. Meadpsilas contributions to software engineering education, in terms of research and role modeling, by using her security quality requirements engineering (SQUARE) process as an organizing theme.

Undocumented software systems are a common challenge for developers performing maintenance and/or reuse. The challenge is two-fold: (1) when no comments or documentation exist, it is difficult for developers to understand how a system works; (2) when no requirements exist, it is difficult to know what the system actually does. We present a method, named ReORe (Reuse or Rewrite) that assists developers in recovering requirements for a competitor system and in deciding if they should reuse parts of their existing system or rewrite it from scratch. Our method requires source code and executable for the system and assumes that requirements are preliminarily recovered. We apply ReORe to Lynx, a Web browser written in C. We provide evidence of ReORe accuracy: 56% for validation based on textual and static analysis and 94% for the final validation using dynamic analysis.

Software artefact traceability is widely recognised as an important factor for the effective development and maintenance of a software system. Unfortunately, the lack of automatic or semi-automatic supports makes the task of maintaining links among software artefacts a tedious and time consuming one. For this reason, often traceability information becomes out of date or it is completely absent during software development. In this working session, we discuss problems and challenges related to various aspects of trace-ability in software systems.

In determining whether to permit a safety-critical software system to be certified and in performing independent verification and validation (IV&V) of safety- or mission-critical systems, the requirements traceability matrix (RTM) delivered by the developer must be assessed for accuracy. The current state of the practice is to perform this work manually, or with the help of general-purpose tools such as word processors and spreadsheets; Such work is error-prone and person-power intensive. In this paper, we extend our prior work in application of Information Retrieval (IR) methods for candidate link generation to the problem of RTM accuracy assessment. We build voting committees from five IR methods, and use a variety of voting schemes to accept or reject links from given candidate RTMs. We report on the results of two experiments. In the first experiment, we used 25 candidate RTMs built by human analysts for a small tracing task involving a portion of a NASA scientific instrument specification. In the second experiment, we randomly seeded faults in the RTM for the entire specification. Results of the experiments are presented.

Predicting future success of students as software engineers is an open research area. We posit that current grading means do not capture all the information that may predict whether students will become good software engineers. We use one such piece of information, traceability of project artifacts, to illustrate our argument. Traceability has been shown to be an indicator of software project quality in industry. We present the results of a case study of a University of Waterloo graduate-level software engineering course where traceability was examined as well as course grades (such as mid-term, project grade, etc.). We found no correlation between the presence of good traceability and any of the course grades, lending support to our argument

During software evolution a collection of related artifacts with different representations are created. Some of these are composed of structured data (e.g., analysis data), some contain semi-structured information (e.g., source code), and many include unstructured information (e.g., text). Research efforts exist that are trying to extract, represent, and analyze the unstructured information in software. Information retrieval (IR) techniques are used quite successfully in the past years to represent and extract textual information from software artifacts, with application to many maintenance tasks. This working session will focus on the state on the art in the application of IR-based techniques to support software maintenance activities. The session aims to identify the main research and practical issues in the field, to determine future work directions, and to foster collaborations among the participants

This paper addresses the issues related to improving the overall quality of the dynamic candidate link generation for the requirements tracing process for verification and validation and independent verification and validation analysts. The contribution of the paper is four-fold: we define goals for a tracing tool based on analyst responsibilities in the tracing process, we introduce several new measures for validating that the goals have been satisfied, we implement analyst feedback in the tracing process, and we present a prototype tool that we built, RETRO (REquirements TRacing On-target), to address these goals. We also present the results of a study used to assess RETRO's support of goals and goal elements that can be measured objectively.

In order to build predictors of the maintainability of evolving software, we first need a means for measuring maintainability as well as a training set of software modules for which the actual maintainability is known. This paper describes our success at building such a predictor. Numerous candidate measures for maintainability were examined, including a new compound measure. Two datasets were evaluated and used to build a maintainability predictor. The resulting model, Maintainability Prediction Model (MainPredMo), was validated against three held-out datasets. We found that the model possesses predictive accuracy of 83% (accurately predicts the maintainability of 83% of the modules). A variant of MainPredMo, also with accuracy of 83%, is offered for interested researchers.

Establishing and maintaining traceability links and consistency between software artifacts produced or modified in the software life-cycle are costly and tedious activities that are crucial but frequently neglected in practice. Traceability between the free text documentation associated with the development and maintenance cycle of a software system and its source code are crucial in a number of tasks such as program comprehension, software maintenance, and software verification & validation. Finally, maintaining traceability links between subsequent releases of a software system is important for evaluating relative source code deltas, highlighting effort/code variation inconsistencies, and assessing the change history. The main theme of the workshop is focused on understanding and defining the foundations for consistency and change management of software systems within the scope of artifact-to-artifact (model-to-model) traceability.The workshop will address the following issues:A formal definition of model to model traceabilityTraceability between artifacts and processesThe semantics of traceability linksRecovery of traceability linksVisualization of traceability linksInteroperable approaches to support traceabilityTraceability in emerging forms of software engineering including production lines, frameworks, components, etc..The goals of the workshop are to:Broaden awareness within the software engineering community of the potential for the application of traceabilityFacilitate the exchange of ideas and interaction between international researchersDefine open research problems faced in realizing usable approaches for traceabilityConstruct a foundation of materials for future research on traceability .For more information please visit the workshop web site is: http://re.cs.depaul.edu/tefse05/. The workshop proceedings are available through the ACM digital library.

The building of traceability matrices by those other than the original developers is an arduous, error prone, prolonged, and labor intensive task. Thus, after-the-fact requirements tracing is a process where the right kind of automation can definitely assist an analyst. Recently, a number of researchers have studied the application of various methods, often based on information retrieval after-the-fact tracing. The studies are diverse enough to warrant a means for comparing them easily as well as for determining areas that require further investigation. To that end, we present here an experimental framework for evaluating requirements tracing and traceability studies. Common methods, metrics and measures are described. Recent experimental requirements tracing journal and conference papers are catalogued using the framework. We compare these studies and identify areas for future research. Finally, we provide suggestions on how the field of tracing and traceability research may move to a more mature level.

We address the assessment and improvement of the software maintenance function by proposing a maturity model for daily software maintenance activities: the Software Maintenance Maturity Model (SMmm). The software maintenance function suffers from a scarcity of management models to facilitate its evaluation, management, and continuous improvement. The SMmm addresses the unique activities of software maintenance while preserving a structure similar to that of the Capability Maturity Model integration (CMMi). It is designed to be used as a complement to that model. The SMmm is based on practitioners' experience, international standards, and the seminal literature on software maintenance. We present the model's purpose, scope, foundation, and architecture, followed by its initial validation. Copyright © 2005 John Wiley & Sons, Ltd.

It is widely acknowledged that group or team projects are a staple of undergraduate and graduate software engineering courses. Such projects provide students with experiences that better prepare them for their careers, so teamwork is often required or strongly encouraged by accreditation agencies. While there are a multitude of educational benefits of group projects, they also pose considerable challenge in fairly and accurately discerning individual contribution for evaluation purposes. Issues, approaches, and best practices for evaluating individual contribution are presented from the perspectives of the University of Kentucky, University of Ottawa, University of Southern California, and others. The techniques utilized within a particular course generally are a mix of (1) the group mark is everybody's mark, (2) everybody reports what they personally did, (3) other group members report the relative contributions of other group members, (4) pop quizzes on project details, and (5) cross-validating with the results of individual work.

Fault-based analysis is an early lifecycle approach to improving software quality by preventing and/or detecting pre-specified classes of faults prior to implementation. It assists in the selection of verification and validation techniques that can be applied in order to reduce risk. This paper presents our methodology for requirements-based fault analysis and its application to National Aeronautics and Space Administration (NASA) projects. The ideas presented are general enough to be applied immediately to the development of any software system. We built a NASA-specific requirement fault taxonomy and processes for tailoring the taxonomy to a class of software projects or to a specific project. We examined requirement faults for six systems, including the International Space Station (ISS), and enhanced the taxonomy and processes. The developed processes, preliminary tailored taxonomies for critical/catastrophic high-risk (CCHR) systems, preliminary fault occurrence data for the ISS project, and lessons learned are presented and discussed.

We present an approach for improving requirements tracing based on framing it as an information retrieval (IR) problem. Specifically, we focus on improving recall and precision in order to reduce the number of missed traceability links as well as to reduce the number of irrelevant potential links that an analyst has to examine when performing requirements tracing. Several IR algorithms were adapted and implemented to address this problem. We evaluated our algorithms by comparing their results and performance to those of a senior analyst who traced manually as well as with an existing requirements tracing tool. Initial results suggest that we can retrieve a significantly higher percentage of the links than analysts, even when using existing tools, and do so in much less time while achieving comparable signal-to-noise levels.

Abstract We introduce the observe‐mine‐adopt (OMA) paradigm that assists organizations in making improvements to their software development processes without committing to and undertaking large‐scale sweeping organizational process improvement. Specifically, the approach has been applied to improve software practices focused on maintainability. This novel approach is based on the theory that software teams naturally make observations about things that do or do not work well. Teams then mine their artifacts and their recollections of events to find the software products, processes, metrics, etc. that led to the observation. In the case of software maintainability, it is then necessary to perform some measurement to ensure that the methods result in improved maintainability. We introduce two maintainability measures, maintainability product and perceived maintainability, to address this need. Other maintainability measures that may be used in the mine step are also examined. Finally, if the mining activities lead to validated discoveries of processes, techniques or practices that improve the software product, they are formalized and adopted by the team. OMA has been studied experimentally using two project studies and a Web‐based health care system which is maintained by a large industrial software organization. Copyright © 2003 John Wiley & Sons, Ltd.

Our experience shows that a typical industrial project can enhance software engineering research and bring theories to life. The University of Kentucky (UK) is in the initial phase of developing a software engineering curriculum. The first course, a graduate-level survey of software engineering, strongly emphasized quality engineering. assisted by the UK clinic, the students undertook a project to develop a phenylalanine milligram tracker. It helps phenylketonuria (PKU) sufferers to monitor their diet as well as assists PKU researchers to collect data. The project was also used as an informal experimental study. The applied project approach to teaching software engineering appears to be successful thus far. The approach taught many important software and quality engineering principles to inexperienced graduate students in an accurately simulated industrial development environment. It resulted in the development of a framework for describing and evaluating such a real-world project, including evaluation of the notion of a user advocate. It also resulted in interesting experimental trends, though based on a very small sample. Specifically, estimation skills seem to improve over time and function point estimation may be more accurate than LOC estimation.

This paper presents an approach for generating test data for unit-level, and possibly integration-level, testing based on sampling over intervals of the input probability distribution, i.e., one that has been divided or layered according to criteria. Our approach is termed "spathic" as it selects random values felt to be most likely or least likely to occur from a segmented input probability distribution. Also, it allows the layers to be further segmented if additional test data is required later in the test cycle. The spathic approach finds a middle ground between the more difficult to achieve adequacy criteria and random test data generation, and requires less effort on the part of the tester. It can be viewed as guided random testing, with the tester specifying some information about expected input. The spathic test data generation approach can be used to augment "intelligent" manual unit-level testing. An initial case study suggests that spathic test sets defect more faults than random test data sets, and achieve higher levels of statement and branch coverage.

The input validation testing (IVT) technique has been developed to address the problem of statically analyzing input command syntax as defined in an English textual interface and requirements specifications and then generating test cases for input validation testing. The technique does not require design or code, so it can be applied early in the life cycle. A proof-of-concept tool has been implemented and validation has been performed. Empirical validation on industrial software shows that the IVT method found more requirements specification defects than senior testers, generated test cases with higher syntactic coverage than senior testers, and found defects that were not found by the test cases of senior testers. Additionally, the tool performed at a much-reduced cost.

Program faults are artifacts that are widely studied, but there are many aspects of faults that we still do not understand. In addition to the simple fact that one important goal during testing is to cause failures and thereby detect faults, a full understanding of the characteristics of faults is crucial to several research areas in testing. These include fault-based testing, testability, mutation testing, and the comparative evaluation of testing strategies. In this workshop paper, we explore the fundamental nature of faults by looking at the differences between a syntactic and semantic characterization of faults. We offer definitions of these characteristics and explore the differentiation. Specifically, we discuss the concept of "size" of program faults --- the measurement of size provides interesting and useful distinctions between the syntactic and semantic characterization of faults. We use the fault size observations to make several predictions about testing and present preliminary data that supports this model. We also use the model to offer explanations about several questions that have intrigued testing researchers.