Spectrum-Based Fault Localization (SBFL) is a popular technique to assist developers in pinpointing faulty elements within their code based on test outcomes and code coverage. In this paper, we examine the impact of context switching, i.e., when developers must frequently shift their attention between different code parts (such as methods and classes) while going down the SBFL ranked list to find the faulty statement. The basis of our study is the observation that it requires less effort to investigate statements that are next to each other rather than those in different methods and classes. In particular, we analyse the number of visited methods and classes, as well as the frequency of switches between them during the fault localization process. We found that, in programs from the Defects4J benchmark, developers need to explore 40 methods and 12 classes on average, before finding the faulty statement, leading to 53 method- and 40 class switches, respectively.

Numerous Deep Learning (DL)-based fault localization (FL) methods are developed with the aim of leveraging the code coverage matrix and failure vector to identify the connection between program elements and defects. The imbalanced data on which these approaches train their models poses a substantial challenge to the effectiveness of fault localization techniques. This study explores the stability of fault localization models in deep learning, specifically, their performance when trained repeatedly using the same input but varying random initializations. Using the Defect4J benchmark, we trained deep learning models (MLP, CNN, and RNN) independently and found that 86 cases resulted in (partly) consistent rankings among all five models and versions, while 621 exhibited varying outcomes, meaning that 90 % of the produced ranks were different in subsequent trainings. The models showed significant variability in ranking results, with maximum ranks sometimes five times that of the minimum. We also adapted the churn metric from DL research to evaluate models, confirming their instability. To improve stability, meta-parameter optimization, model simplification and resampling has been applied. Although some of these techniques proved effective, even with the improvements, the models remained insufficiently stable to produce reliable results.

In this paper, we present an approach for improving Spectrum-Based Fault Localization (SBFL) by integrating static and dynamic information about code elements. This is achieved by giving more importance to code elements that include mathematical operators compared to other types of elements (e.g., declaration, selection, iteration, or function call) and appear in failed tests. The intuition is that these elements are more likely to have bugs than others. The proposed approach is applicable to any SBFL formula without requiring any modifications to their structures because the weighting is done on the ranking list and not on the formulas. The experimental results of a preliminary study show that our approach achieved a much better performance in terms of average ranking compared to the underlying SBFL formulas. It also improved the Top-N categories; it doubled the number of cases in which the faulty method became the top-ranked element, and in all cases the fault became part of Top-5 of the ranking list.

Many tools for enabling developers to locate bugs in their programs have been proposed in the literature. The majority of programs they target are based on C/C++ and Java. In this paper, we offer a tool named "SFLaaS" for locating faults in programs written in Python and is provided as a service rather than as a plugin or a command-line tool to be installed. Thus, our tool can be accessed anytime and from anywhere. The tool employs Spectrum-based fault localization (SBFL) to help Python developers automatically analyze their programs and generate useful data at run-time to be used to produce a ranked list of potentially faulty program elements (i.e., statements). Our tool supports different important features in fault localization such as supporting about 80 SBFL formulas, different tie-breaking methods, showing code elements with different colors, ranging from most suspicious (red) not suspicious (green) based on their suspicious scores, allowing the user to define his/her own formula, etc. Using our tool could help developers to efficiently find the locations of different types of faults in their programs.

Many tools for enabling developers locating faults in their programs have been proposed in the literature. The majority of the programs they target are those created in the C/C++ and Java languages. In this paper, we offer a tool named "SFLaaS" for locating faults in programs written in Python, a popular programming language, and is provided as a service rather than as a plugin or a command-line tool to be installed. Thus, our tool can be accessed anytime and from anywhere. The tool employs Spectrum-based fault localization (SBFL) to help Python developers automatically analyze their programs and generate useful data at run-time to be used to produce a ranked list of potentially faulty program elements (i.e., statements). Our proposed tool supports different important features in fault localization such as supporting about 80 SBFL formulas, different tie-breaking methods, showing code elements with different colors, ranging from most suspicious (red) to not suspicious (green) based on their suspicious scores, allowing the user to define his/her own formula, etc. Using our tool could help developers to efficiently find faults in their programs.

Spectrum-Based Fault Localization (SBFL) is a semi-automated debugging technique that gained popularity in the last decades due to its intuitive approach and relatively simple implementability. Despite this, the performance of practical SBFL techniques in terms of fault localization capability does not reach the threshold that would enable their acceptance by professional programmers. Almost all modern SBFL approaches are based on the code coverage-based spectrum, and on the assumption that a code element covered by failing tests should be treated as suspicious. However, it is easy to see that this is an over-approximation because many code elements may be executed that do not contribute to the test output, hence serving as noise in the process. A possible solution is to use backward dynamic program slices as program spectra computed from the output statement as the criterion, instead of the coverage. There are very few theoretical and practical results about this approach, so in this work we revisit the method and show how much more inferior coverage-based spectra are compared to slice-based spectra, both on theoretical and practical levels. We argue that code coverage-based SBFL is currently in a research pit due to this inherent approximation, and research on slice-based spectra should once more attain a much higher focus.

Spectrum-Based Fault Localization (SBFL) is based on risk formulas to rank program elements, which work generally well in various situations. However, it cannot be ruled out that zero division might happen during score calculation, which has negative consequences, e.g., essential elements will not be in the top part of the rank list. The literature has given several strategies to tackle the problem, although there is little knowledge on which one to use. In our work, we performed mathematical analysis and an empirical study to find out how this phenomenon affects SBFL. Results show that division by zero happens in many cases, and the strategies can mitigate their consequences with varying success. Thus, we propose a combined method to avoid the threat of division by zero and improve the trustworthiness of SBFL. Our proposals should be taken into consideration whenever a formula is being used or a new one is proposed.

Spectrum-Based Fault Localization (SBFL) uses a mathematical formula to determine a suspicion score for each program element (such as a statement, method, or class) based on fundamental statistics (e.g., how many times each element is executed and not executed in passed and failed tests) taken from test coverage and results. Based on the calculated scores, program elements are then ordered from most suspicious to least suspicious. The elements with the highest scores are thought to be the most prone to error. The final ranking list of program elements aids developers in debugging when looking for the source of a fault in the program under test. In this paper, we present a new SBFL ranking formula that enhances a base formula by ranking code elements slightly higher than others that are executed by more failed tests and less passing ones. Its novelty is that it breaks ties between the elements that share the same suspicion score of the base formula. Experiments were conducted on six single-fault programs of the Defects4J dataset to evaluate the effectiveness of the proposed formula. The results show that our new formula when compared to three widely-studied SBFL formulas, achieved a better performance in terms of average ranking. It also achieved positive results in all of the Top-N categories and increased the number of cases where the faulty element became the top-ranked element by 13–23%.

Abstract Tool support for automated fault localization in program debugging is limited because state-of-the-art algorithms often fail to provide efficient help to the user. They usually offer a ranked list of suspicious code elements, but the fault is not guaranteed to be found among the highest ranks. In Spectrum-Based Fault Localization (SBFL) – which uses code coverage information of test cases and their execution outcomes to calculate the ranks –, the developer has to investigate several locations before finding the faulty code element. Yet, all the knowledge she a priori has or acquires during this process is not reused by the SBFL tool. There are existing approaches in which the developer interacts with the SBFL algorithm by giving feedback on the elements of the prioritized list. We propose a new approach called iFL which extends interactive approaches by exploiting contextual knowledge of the user about the next item in the ranked list (e. g., a statement), with which larger code entities (e. g., a whole function) can be repositioned in their suspiciousness. We implemented a closely related algorithm proposed by Gong et al., called Talk. First, we evaluated iFL using simulated users, and compared the results to SBFL and Talk. Next, we introduced two types of imperfections in the simulation: user’s knowledge and confidence levels. On SIR and Defects4J, results showed notable improvements in fault localization efficiency, even with strong user imperfections. We then empirically evaluated the effectiveness of the approach with real users in two sets of experiments: a quantitative evaluation of the successfulness of using iFL, and a qualitative evaluation of practical uses of the approach with experienced developers in think-aloud sessions.

In traditional Spectrum-Based Fault Localization (SBFL), hit-based spectrum is used to estimate a program element’s suspiciousness to contain a fault, i.e., only the binary information is used if the code element was executed by the test case or not. Count-based spectra can potentially improve the localization effectiveness due to the number of executions also being available. In this work, we use function-level granularity and define count-based spectra which use function call frequencies. We investigate the naïve approach, which simply counts the function call instances. We also define a novel method which is based on counting the different function call contexts, i.e., the frequency of the investigated function occurring in unique call stack instances during test execution. The basic intuition is that if a function is called in many different contexts during a failing test case, it will be more probable to be accountable for the fault. We empirically evaluated the fault localization capability of different variations of the approach and compared them to 9 traditional SBFL techniques using the Defects4J benchmark. We show that: (i) naïve counts result in worse rank positions than the hit-based approach, but (ii) unique counts produce better rank positions with some of the algorithm variants.

Fault localization is one of the most time-consuming and error-prone parts of software debugging. There are several tools for helping developers in the fault localization process, however, they mostly target programs written in Java and C/C++ programming languages. While these tools are splendid on their own, we must not look over the fact that Python is a popular programming language, and still there are a lack of easy-to- use and handy fault localization tools for Python developers. In this paper, we present a tool called “CharmFL” for software fault localization as a plug-in for PyCharm IDE. The tool employs Spectrum-based fault localization (SBFL) to help Python developers automatically analyze their programs and generate useful data at run-time to be used, then to produce a ranked list of potentially faulty program elements (i.e., statements, functions, and classes). Thus, our proposed tool supports different code coverage types with the possibility to investigate these types in a hierarchical approach. The applicability of our tool has been presented by using a set of experimental use cases. The results show that our tool could help developers to efficiently find the locations of different types of faults in their programs.

In Spectrum-Based Fault Localization (SBFL), a suspiciousness score is assigned to each code element based on test coverage and test outcomes. The scores are then used to rank the code elements relative to each other in order to aid the programmer during the debugging process when seeking the source of a fault. However, probably none of the known SBFL formulae are guaranteed to produce different scores for all the program elements, hence ties emerge between the code elements. Based on our experiments, ties in SBFL are prevalent: in Defects4J, 54–56% of buggy methods are members of ties, i.e., there is at least one other method with the same score in these cases (but typically much more, on average 6), and this inevitably reduces the effectiveness of any SBFL approach. In this work, we present a technique to break ties in such cases based on the so-called method calls frequencies. This counts the number of different contexts of method calls (both as callees and as callers) in failing test cases. The intuition is that if a method appears in many different calling contexts during a failing test case, it will be more suspicious and get a higher rank position compared to other methods with the same scores. This method can be applied to any underlying SBFL formula, and can favourably break the occurring ranks in the ties in many cases. The experimental results show that our novel tie-breaking strategy achieved a significant reduction in both size and number of critical ties in our benchmark. In 72-73% of the cases, the ties were completely eliminated and the average reduction rate was more than 80%.

Spectrum-Based Fault Localization (SBFL), in its basic form, uses only local information about a program element’s (such as a method’s) coverage to predict its faultiness, and rarely is any additional (contextual) information leveraged about the element itself, nor the test cases. As such an additional context, in the presented approach, we rely on the frequency of the investigated method occurring in call stack instances during the course of executing the failing test cases. The basic intuition is that if a method is called in many different contexts during a failing test case, it will be more probable to be accountable for the fault compared to other methods. We empirically evaluated the fault localization capability of the approach compared to five traditional SBFL techniques using the bug benchmark Defects4J. We found that the new algorithms (i) find the location of bugs at higher rank positions more often, (ii) can achieve 38%–52% rank position improvement compared to the baseline algorithms with statistical significance, and (iii) place more items at the top-10 positions of the suspiciousness ranking.

Summary JavaScript is a popular programming language that is also error‐prone due to its asynchronous, dynamic, and loosely typed nature. In recent years, numerous techniques have been proposed for analyzing and testing JavaScript applications. However, our survey of the literature in this area revealed that the proposed techniques are often evaluated on different datasets of programs and bugs. The lack of a commonly used benchmark limits the ability to perform fair and unbiased comparisons for assessing the efficacy of new techniques. To fill this gap, we propose BugsJS, a benchmark of 453 real, manually validated JavaScript bugs from 10 popular JavaScript server‐side programs, comprising 444k lines of code (LOC) in total. Each bug is accompanied by its bug report, the test cases that expose it, as well as the patch that fixes it. We extended BugsJS with a rich web interface for visualizing and dissecting the bugs' information, as well as a programmable API to access the faulty and fixed versions of the programs and to execute the corresponding test cases, which facilitates conducting highly reproducible empirical studies and comparisons of JavaScript analysis and testing tools. Moreover, following a rigorous procedure, we performed a classification of the bugs according to their nature. Our internal validation shows that our taxonomy is adequate for characterizing the bugs in BugsJS. We discuss several ways in which the resulting taxonomy and the benchmark can help direct researchers interested in automated testing of JavaScript applications. © 2021 The Authors. Software Testing, Verification & Reliability published by John Wiley & Sons, Ltd.

Fault localization is considered a difficult and time consuming activity. However, tool support for automated fault localization is still limited because state-of-the-art algorithms often fail to provide efficient help to the user. They usually offer a ranked list of suspicious code elements, but the fault is not guaranteed to be found among the highest ranks. In Spectrum-Based Fault Localization (SBFL) - which uses code coverage information of test cases and their execution outcomes to calculate the ranks -, the developer has to investigate several locations before finding the faulty code element. Yet, all the knowledge she a priori has or acquires during this process is not reused by the SBFL tool. We propose an approach in which the developer interacts with the SBFL algorithm by giving feedback on the elements of the prioritized list. We exploit contextual knowledge of the user about the next item in the ranked list (e. g., a statement), with which larger code entities (e. g., a whole function) can be repositioned in their suspiciousness. First, we evaluated the approach using simulated users incorporating two types of imperfections, their knowledge and confidence levels. On SIR and Defects4J, results showed notable improvements in fault localization efficiency, even with strong user imperfections. We then empirically evaluated the effectiveness of the approach with real users, which also showed promising results.

In Spectrum Based Fault Localization, program elements such as statements or functions are ranked according to a suspiciousness score which can guide the programmer in finding the fault more efficiently. However, such a ranking does not include any additional information about the suspicious code elements. In this work, we propose to complement function-level spectrum based fault localization with function call chains - i.e., snapshots of the call stack occurring during execution - on which the fault localization is first performed, and then narrowed down to functions. Our experiments using defects from Defects4J show that (i) 69% of the defective functions can be found in call chains with highest scores, (ii) in 4 out of 6 cases the proposed approach can improve Ochiai ranking of 1 to 9 positions on average, with a relative improvement of 19–48%, and (iii) the improvement is substantial (66–98%) when Ochiai produces bad rankings for the faulty functions.

Spectrum-Based Fault Localization (SBFL) is a well-understood statistical approach to software fault localization, and there have been numerous studies performed that tackle its effectiveness. However, mostly Java and C/C++ programs have been addressed to date. We performed an empirical study on SBFL for JavaScript programs using a recent bug benchmark, BugsJS. In particular, we examined (1) how well some of the most popular SBFL algorithms, Tarantula, Ochiai and DStar, can predict the faulty source code elements in these JavaScript programs, (2) whether there is a significant difference between the effectiveness of the different SBFL algorithms, and (3) whether there is any relationship between the bug-fix types and the performance of SBFL methods. For the latter, we performed a manual classification of each benchmark bug according to an existing classification scheme. Results show that the performance of the SBFL algorithms is similar but there are some notable differences among them as well, and that certain bug-fix types can be significantly differentiated from the others (in both positive and negative direction) based on the fault localization effectiveness of the investigated algorithms.

In many different engineering fields, fault localization means narrowing down the cause of a failure to a small number of suspicious components of the system. This activity is an important concern in many areas, and there have been a large number of techniques proposed to aid this activity. Some of the basic ideas used are common to different fields, but generally quite diverse approaches are applied. Our long-term goal with the presented research is to identify potential techniques from non-software domains that have not yet been fully leveraged to software faults, and investigate their applicability and adaptation to our field. We performed an analysis of related literature, not limiting the search to any specific engineering field, with the aim to find solutions in non-software areas that could be most successfully adapted to software fault localization. We found out that few areas have significant literature in the topic that are good candidates for adaptation (computer networks, for instance), and that although some classes of methods are less suitable, there are useful ideas in almost all fields that could potentially be reused. As an example of potential novel techniques for software fault localization, we present three concrete techniques from other fields and how they could potentially be adapted.

Spectrum-Based ones are a popular class of Fault Localization (FL) methods among researchers due to their relative simplicity. However, recent studies highlighted some barriers to the wider adoption of the technique in practical settings. One possibility to increase the practical usefulness of related tools is to involve interactivity between the user and the core FL algorithm. In this setting, the developer interacts with the fault localization algorithm by giving feedback on the elements proposed by the algorithm. This way, the proposed elements can be influenced in the hope to reach the faulty element earlier (we call the proposed approach Interactive Fault Localization, or iFL). With this work, we present our recent achievements in this topic. In particular, we overview the basic approach, our preliminary experimentation with user simulation, and the supporting tool for the actual usage of the method, iFL for Eclipse. Our aim is to provide a basis for the investigation of the feasibility and effectiveness of the technique, before moving on to more comprehensive experiments with actual human subjects. We invite researchers for further discussion on the topic, and for that, the method and tool will be made accessible.

JavaScript is a popular programming language that is also error-prone due to its asynchronous, dynamic, and loosely-typed nature. In recent years, numerous techniques have been proposed for analyzing and testing JavaScript applications. However, our survey of the literature in this area revealed that the proposed techniques are often evaluated on different datasets of programs and bugs. The lack of a commonly used benchmark limits the ability to perform fair and unbiased comparisons for assessing the efficacy of new techniques. To fill this gap, we propose BugsJS, a benchmark of 453 real, manually validated JavaScript bugs from 10 popular JavaScript server-side programs, comprising 444k LOC in total. Each bug is accompanied by its bug report, the test cases that detect it, as well as the patch that fixes it. BugsJS features a rich interface for accessing the faulty and fixed versions of the programs and executing the corresponding test cases, which facilitates conducting highly-reproducible empirical studies and comparisons of JavaScript analysis and testing tools.

In our recent work, we proposed BUGSJS, a benchmark of several hundred bugs from popular JavaScript server-side programs. In this abstract paper, we report the results of our initial evaluation in adopting BUGSJS to support an experiment in fault localization. First, we describe how BUGSJS facilitated accessing the information required to perform the experiment, namely, test case code, their outcomes, their associated code coverage and related bug information. Second, we illustrate how BUGSJS can be improved to further enable easier application to fault localization research, for instance, by filtering out failing test cases that do not directly contribute to a bug. We hope that our preliminary results will foster researchers in using BUGSJS to enable highly-reproducible empirical studies and comparisons of JavaScript analysis and testing tools.

In industrial practice the clone-and-own strategy is often applied when in the pressure of high demand of customized features. The adoption of software product line (SPL) architecture is a large one time investment that affects both technical and organizational issues. The analysis of the feature structure is a crucial point in the SPL adoption process involving domain experts working at a higher level of abstraction and developers working directly on the program code. We propose automatic methods to extract feature-to-program links starting from very high level set of features provided by domain experts. For this purpose we combine call graph information with textual similarity between code and high level features. In addition, in depth understanding of the feature structure is supported by finding communities between programs and relating them to features. As features are originated from domain experts, community analysis reveals discrepancies between expert view and internal code structure. We found that communities correspond well to the high level features, with usually more than half of feature code located in specialized communities. We report experiments at two levels of features and more than 2000 Magic 4GL programs in an industrial SPL adoption project.

Recovering test-to-code traceability links may be required in virtually every phase of development. This task might seem simple for unit tests thanks to two fundamental unit testing guidelines: isolation (unit tests should exercise only a single unit) and separation (they should be placed next to this unit). However, practice shows that recovery may be challenging because the guidelines typically cannot be fully followed. Furthermore, previous works have already demonstrated that fully automatic test-to-code traceability recovery for unit tests is virtually impossible in a general case. In this work, we propose a semi-automatic method for this task, which is based on computing traceability links using static and dynamic approaches, comparing their results and presenting the discrepancies to the user, who will determine the final traceability links based on the differences and contextual information. We define a set of discrepancy patterns, which can help the user in this task. Additional outcomes of analyzing the discrepancies are structural unit testing issues and related refactoring suggestions. For the static test-to-code traceability, we rely on the physical code structure, while for the dynamic, we use code coverage information. In both cases, we compute combined test and code clusters which represent sets of mutually traceable elements. We also present an empirical study of the method involving 8 non-trivial open source Java systems.

New customers often require custom features of a successfully marketed product. As the number of variants grow, new challenges arise in the maintenance and evolution activities. Software product line (SPL) architecture is a timely answer to these challenges. The SPL adoption however is a large one time investment that affects both technical and organizational issues. From the program code point of view, the extractive approach is appropriate when there are already several product variants. Analyzing the feature structure, the differences and commonalities of the variants lead to the new common architecture. In this work in progress paper we report initial experiments of feature extraction from a set of product variants written in the Magic fourth generation language (4GL). Since existing approaches are mostly designed for mainstream languages, we adapted and reused reverse engineering approaches to the 4GL environment. We followed a semi-automatic feature extraction method, where the higher level features are provided by domain experts. These features are then linked to the internal structure of Magic applications using a textual similarity (IR-based) method. We demonstrate the feasibility of 4GL feature extraction method and validate it on two variants of a real life logistical system each consisting of more than 2000 Magic programs.

Newer technologies - programming languages, environments, libraries - change very rapidly. However, various internal and external constraints often prevent projects from quickly adopting to these changes. Customers may require specific platform compatibility from a software vendor, for example. In this work, we deal with such an issue in the context of the C++ programming language. Our industrial partner is required to use SDKs that support only older C++ language editions. They, however, would like to allow their developers to use the newest language constructs in their code. To address this problem, we created a source code transformation framework to automatically backport source code written according to the C++11 standard to its functionally equivalent C++03 variant. With our framework developers are free to exploit the latest language features, while production code is still built by using a restricted set of available language constructs. This paper reports on the technical details of the transformation engine, and our experiences in applying it on two large industrial code bases and four open-source systems. Our solution is freely available and open-source.

The software development industry has adopted written and de facto standards for creating effective and maintainable unit tests. Unfortunately, like any other source code artifact, they are often written without conforming to these guidelines, or they may evolve into such a state. In this work, we address a specific type of issues related to unit tests. We seek to automatically uncover violations of two fundamental rules: 1) unit tests should exercise only the unit they were designed for, and 2) they should follow a clear packaging convention. Our approach is to use code coverage to investigate the dynamic behaviour of the tests with respect to the code elements of the program, and use this information to identify highly correlated groups of tests and code elements (using community detection algorithm). This grouping is then compared to the trivial grouping determined by package structure, and any discrepancies found are treated as "bad smells." We report on our related measurements on a set of large open source systems with notable unit test suites, and provide guidelines through examples for refactoring the problematic tests.

Code coverage measurement is an important element in white-box testing, both in industrial practice and academic research. Other related areas are highly dependent on code coverage as well, including test case generation, test prioritization, fault localization, and others. Inaccuracies of a code coverage tool sometimes do not matter that much but in certain situations they can lead to serious confusion. For Java, the prevalent approach to code coverage measurement is to use bytecode instrumentation due to its various benefits over source code instrumentation. However, if the results are to be mapped back to source code this may lead to inaccuracies due to the differences between the two program representations. In this paper, we systematically investigate the amount of differences in the results of these two Java code coverage approaches, enumerate the possible reasons and discuss the implications on various applications. For this purpose, we relied on two widely used tools to represent the two approaches and a set of benchmark programs from the open source domain.

During software projects, several parts of the source code are usually re-written due to imperfect solutions before the code is released. This wasted effort is of central interest to the project management to assure on-time delivery. Although the amount of thrown-away code can be measured from version control systems, stakeholders are more interested in productivity dynamics that reflect the constant change in a software project. In this paper we present a field study of measuring the productivity of a medium-sized J2EE project. We propose a productivity analysis method where productivity is expressed through dynamic profiles - the so-called Micro-Productivity Profiles (MPPs). They can be used to characterize various constituents of software projects such as components, phases and teams. We collected detailed traces of developers' actions using an Eclipse IDE plug-in for seven months of software development throughout two milestones. We present and evaluate profiles of two important axes of the development process: by milestone and by application layers. MPPs can be an aid to take project control actions and help in planning future projects. Based on the experiments, project stakeholders identified several points to improve the development process. It is also acknowledged, that profiles show additional information compared to a naive diff-based approach.

Dependence clusters are (maximal) collections of mutually dependent source code entities according to some dependence relation. Their presence in software complicates many maintenance activities including testing, refactoring, and feature extraction. Despite several studies finding them common in production code, their formation, identification, and overall structure are not well understood, partly because of challenges in approximating true dependences between program entities. Previous research has considered two approximate dependence relations: a fine-grained statement-level relation using control and data dependences from a program's System Dependence Graph and a coarser relation based on function-level control-flow reachability. In principal, the first is more expensive and more precise than the second. Using a collection of twenty programs, we present an empirical investigation of the clusters identified by these two approaches. In support of the analysis, we consider a hybrid cluster type that works at the coarser function-level but is based on the higher-precision statement-level dependences. The three types of clusters are compared based on their slice sets using two clustering metrics. We also perform extensive analysis of the programs to identify linchpin functions - functions primarily responsible for holding a cluster together. Results include evidence that the less expensive, coarser approaches can often be used as effective proxies for the more expensive, finer-grained approaches. Finally, the linchpin analysis shows that linchpin functions can be effectively and automatically identified.

The graphical representations of software (code visualization in particular) may provide both professional programmers and students learning only the basics with support in program comprehension. Among the numerous proposed approaches, our research applies the city metaphor for the visualisation of such code elements as classes, functions, or attributes by the tool CodeMetropolis. It uses the game engine of Minecraft for the graphics, and is able to visualize various properties of the code based on structural metrics. In this work, we present our approach to integrate our visualization tool into the Eclipse IDE environment. Previously, only standalone usage was possible, but with this new version the users can invoke the visualization directly from the IDE, and all the analysis is performed in the background. The new version of the tool now includes an Eclipse plug-in and a Minecraft modification in addition to the analysis and visualization modules which have also been extended with some new features. Possible use cases and a detailed scenario are presented.

Code coverage is often used in academic and industrial practice of white-box software testing. Various test optimization methods, e.g. Test selection and prioritization, rely on code coverage information, but other related fields benefit from it as well, such as fault localization. These methods require access to the fine details of coverage information and efficient ways of processing this data. The purpose of the (free) SoDA library and toolset is to provide an efficient set of data structures and algorithms which can be used to prepare, store and analyze in various ways data related to code coverage. The focus of SoDA is not on the calculation of coverage data (such as instrumentation and test execution) but on the analysis and manipulation of test suites based on such information. An important design goal of the library was to be usable on industrial-size programs and test suites. Furthermore, there is no limitation on programming language, analysis granularity and coverage criteria. In this paper, we demonstrate the purpose and benefits of the library, the associated toolset, which also includes a graphical user interface, as well as possible usage scenarios. SoDA also includes a repository of prepared programs, which are from small to large sizes and can be used for experimentation and as a benchmark for code coverage related research.

SUMMARY Impact analysis based on code dependence can provide opportunities to identify parts of the software affected by a change. Because changes usually have far reaching effects in programs, effective and efficient impact analysis is vital. Static Execute After (SEA) is a relation on procedures that is efficiently computable and accurate enough to be a candidate for the use in impact analysis in practice. To assess the applicability of SEA in terms of capturing real defects, we present results on integrating it into the build system of WebKit, a large, open source software system, and on related experiments. We show that a large number of real defects can be captured by impact sets computed by SEA, albeit many of them are large. We demonstrate that this is not an issue in applying it to regression test prioritization, but generally it can be an obstacle in the path to efficient use of impact analysis. We believe that the main reason for large impact sets is the formation of dependence clusters in code. As apparently dependence clusters cannot be easily avoided in the majority of cases, we focus on determining the effects these clusters have on impact analysis and regression test prioritization. Copyright © 2013 John Wiley & Sons, Ltd.

Data visualisation with high expressive power plays an important role in code comprehension. Recent visualization tools try to fulfil the expectations of the users and use various analogies. For example, in an architectural metaphor, each class is represented by a building. Buildings are grouped into districts according to the structure of the namespaces. We think that these unique ways of code representation have great potential, but in our opinion they use very simple graphical techniques (shapes, figures, low resolution) to visualize the structure of the source code.On the other hand, computer games use high quality graphic and good expressive power. A good example is Minecraft, a popular role playing game with great extensibility and interactivity from another (third party) software. It supports both high definition, photo-realistic textures and long range 3D scene displaying. Our main contribution is to connect data visualisation with high end-user graphics capabilities. To achieve this, a conversion tool was implemented. It processes the basic source code metrics as input and generates a Minecraft world with buildings, districts, and gardens. The tool is in the prototype state, but it can be used to investigate the possibilities of this kind of data visualisation.

Dependence clusters are (maximal) groups of source code entities that each depend on the other according to some dependence relation. Such clusters are generally seen as detrimental to many software engineering activities, but their formation and overall structure are not well understood yet. In a set of subject programs from moderate to large sizes, we observed frequent occurrence of dependence clusters using Static Execute After (SEA) dependences (SEA is a conservative yet efficiently computable dependence relation on program procedures). We identified potential linchpins inside the clusters; these are procedures that can primarily be made responsible for keeping the cluster together. Furthermore, we found that as the size of the system increases, it is more likely that multiple procedures are jointly responsible as sets of linchpins. We also give a heuristic method based on structural metrics for locating possible linchpins as their exact identification is unfeasible in practice, and presently there are no better ways than the brute-force method. We defined novel metrics and comparison methods to be able to demonstrate clusters of different sizes in programs.

Data visualisation with high expressive power plays an important role in code comprehension. Recent visualisation tools try to fulfill the expectations of the users and use various analogies. For example, in an architectural metaphor, each class is represented by a building. Buildings are grouped into districts according to the structure of the namespaces. We think that these unique ways of code representation have great potential, but in our opinion they use very simple graphical techniques (shapes, figures, low resolution) to visualise the structure of the source code. On the other hand, computer games use high quality graphic and have high expressive power. A good example is Minecraft, a popular role playing game that supports both high definition, photorealistic textures and long range 3D scene displaying. Additionally, it provides great extensibility and interactivity for third party software. In this paper, we introduce our mission to create a virtual world of source code in which developers and other stakeholders could explore and evaluate their project collaboratively in a virtual Minecraft world. Code properties are represented by graphical primitives offered by the game engine, and various interactivity features are planned. Besides challenges of the implementation there are some fundamental research issues considering the selection of a set of visual elements and mapping to source code properties. These elements have to be compatible not only with the visualisation and with the data model but also with the thinking of developers.

Automated regression testing is often crucial in order to maintain the quality of a continuously evolving software system. However, in many cases regression test suites tend to grow too large to be suitable for full re-execution at each change of the software. In this case selective retesting can be applied to reduce the testing cost while maintaining similar defect detection capability. One of the basic test selection methods is the one based on code coverage information, where only those tests are included that cover some parts of the changes. We experimentally applied this method to the open source web browser engine project WebKit to find out the technical difficulties and the expected benefits if this method is to be introduced into the actual build process. Although the principle is simple, we had to solve a number of technical issues, so we report how this method was adapted to be used in the official build environment. Second, we present results about the selection capabilities for a selected set of revisions of WebKit, which are promising. We also applied different test case prioritization strategies to further reduce the number of tests to execute. We explain these strategies and compare their usefulness in terms of defect detection and test suite reduction.

Impact analysis based on code dependence can be an integral part of software quality assurance by providing opportunities to identify those parts of the software system that are affected by a change. Because changes usually have far reaching effects in programs, effective and efficient impact analysis is vital, which has different applications including change propagation and regression testing. Static Execute After (SEA) is a relation on program elements (procedures) that is efficiently computable and accurate enough to be a candidate for use in impact analysis in practice. To assess the applicability of SEA in terms of capturing real defects, we present results on integrating it into the build system of Web Kit, a large, open source software system, and on related experiments. We show that a large number of real defects can be captured by impact sets computed by SEA, albeit many of them are large. We demonstrate that this is not an issue in applying it to regression test prioritization, but generally it can be an obstacle in the path to efficient use of impact analysis. We believe that the main reason for large impact sets is the formation of dependence clusters in code. As apparently dependence clusters cannot be easily avoided in the majority of cases, we focus on determining the effects these clusters have on impact analysis.

Software estimation is used in various contexts including cost, maintainability or defect prediction. To make the estimate, different models are usually applied based on attributes of the development process and the product itself. However, often only one type of attributes is used, like historical process data or product metrics, and rarely their combination is employed. In this report, we present a project in which we started to develop a framework for such complex measurement of software projects, which can be used to build combined models for different estimations related to software maintenance and comprehension. First, we performed an experiment to predict modification complexity (cost of a unity change) based on a combination of process and product metrics. We observed promising results that confirm the hypothesis that a combined model performs significantly better than any of the individual measurements.

In risk-based testing, compromises are often made to release a system in spite of knowing that it has outstanding defects. In an industrial setting, time and cost are often the “exit criteria” and - unfortunately - not the technical aspects like coverage or defect ratio. In such situations, the stakeholders accept that the remaining defects will be found after release, so sufficient resources are allocated to the “stabilization” phases following the release. It is hard for many organizations to see that such an approach is significantly costlier than trying to locate the defects earlier. We performed an empirical investigation of this for one of our industrial partners (a financial company). In this project, significant perfective maintenance was performed on the large information system. Based on changes made to the system, we carried out procedure level code coverage measurements with code level change impact analysis, and a similarity-based comparison of test cases in order to quantitatively check the completeness and redundancy of the tests performed. In addition, we logged and compared the number of defects found during testing and live operation. The data obtained were surprising for both the developers and the customer as well, leading to a major reorganization of their development, testing, and operation processes. After the reorganization, a significant improvement in these indicators for testing efficiency was observed.

The paper explores Static Execute After (SEA) dependencies in the program and their dual Static Execute Before (SEB) dependencies. It empirically compares the SEA/SEB dependencies with the traditional dependencies that are computed by System Dependence Graph (SDG) and program slicers. In our case study we use about 30 subject programs that were previously used by other authors in empirical studies of program analysis. We report two main results. The computation of SEA/SEB is much less expensive and much more scalable than the computation of the SDG. At the same time, the precision declines only very slightly, by some 4% on average. In other words, the precision is comparable to that of the leading traditional algorithms, while intuitively a much larger difference would be expected. The paper then discusses whether based on these results the computation of the SDG should be replaced in some applications by the computation of the SEA/SEB.

Slicing C programs has been one of the most popular ways for the implementation of slicing algorithms; out of the very few practical implementations that exist many deal with this programming language. Yet, preprocessor related issues have been addressed very marginally by these slicers, despite the fact that ignoring (or handling poorly) these constructs may lead to serious inaccuracies in the slicing results and hence in the comprehension process. Recently, an accurate slicing method for preprocessor related constructs has been proposed which - when combined with existing C/C++ language slicers - can provide a more complete comprehension of these languages. In this paper, we overview our approach for this combination and report its benefits in terms of the completeness of the resulting slices.

In this paper, we introduce static execute after (SEA) relationship among program components and present an efficient analysis algorithm. Our case studies show that SEA may approximate static slicing with perfect recall and high precision, while being much less expensive and more usable. When differentiating between explicit and hidden dependencies, our case studies also show that SEA may correlate with direct and indirect class coupling. We speculate that SEA may find applications in computation of hidden dependencies and through it in many maintenance tasks, including change propagation and regression testing.

Design patterns present good solutions to frequently occurring problems in object-oriented software design. Thus their correct application in a system's design may significantly improve its internal quality attributes such as reusability and maintainability. In software maintenance the existence of up-to-date documentation is crucial, so the discovery of as yet unknown design pattern instances can help improve the documentation. Hence a reliable design pattern recognition system is very desirable. However, simpler methods (based on pattern matching) may give imprecise results due to the vague nature of the patterns' structural description. In previous work we presented a pattern matching-based system using the Columbus framework with which we were able to find pattern instances from the source code by considering the patterns' structural descriptions only, and therefore we could not identify false hits and distinguish similar design patterns such as state and strategy. In the present work we use machine learning to enhance pattern mining by filtering out as many false hits as possible. To do so we distinguish true and false pattern instances with the help of a learning database created by manually tagging a large C++ system.

Automatic fact extraction from software systems is the fundamental building block in the process of understanding the relationships among a system's elements. We demonstrate the reverse engineering framework called Columbus which is able to automatically extract facts from C++ source code and how the extracted facts can be used in practice. We also mention a special-purpose tool that was developed on top of the Columbus framework. This tool, called SourceAudit, is a code auditor that is able to investigate source code and check it against rules that describe the preferred properties of the code.

Owing to their relative simplicity and wide range of applications, static slices are specifically proposed for software maintenance and program understanding. Unfortunately, in many cases static slices are overly conservative and therefore too large to supply useful information to the software maintainer. Dynamic slicing methods can produce more precise results, but only for one test case. In this paper we introduce the concept of union slices (the union of dynamic slices for many test cases) and suggest using a combination of static and union slices. This way the size of program parts that need to be investigated can be reduced by concentrating on the most important parts first. We performed a series of experiments with our experimental implementation on three medium size C programs. Our initial results suggest that union slices are in most cases far smaller than static slices, and that the growth rate of union slices (by adding more test cases) significantly declines after several representative executions of the program.

One of the most critical issues in large-scale software development and maintenance is the rapidly growing size and complexity of software systems. As a result of this rapid growth there is a need to better understand the relationships between the different parts of a large software system. In this paper we present a reverse engineering framework called Columbus that is able to analyze large C++ projects, and a schema for C++ that prescribes the form of the extracted data. The flexible architecture of the Columbus system with a powerful C++ analyzer and schema makes it a versatile and readily extendible toolset for reverse engineering. This tool is free for scientific and educational purposes and we fervently hope that it will assist academic persons in any research work related to C++ re- and reverse engineering.