Background . Code coverage and mutation score serve as pivotal test effectiveness metrics used to assess a test suite’s ability to uncover actual defects. However, prior research has produced inconsistent or even conflicting findings regarding their correlation with defect detection capability, particularly concerning the impact of test suite size. Problem. The extent of the potentially confounding effect of test suite size in test effectiveness evaluation context is not clear, nor is the method to remove the potentially confounding effect, or the influence of this removal on the performance of test suite optimization. Objective . Our goal is to deeply understand how test suite size affects the true relationship between test effectiveness metrics and a test suite’s ability to detect actual defects. Method. We first employ statistical methods to examine the extent of the potentially confounding effect of test suite size in the context of test effectiveness evaluation. After that, we propose a linear regression-based method to remove the potentially confounding effect of test suite size. Finally, we empirically explore the impact of this removal method on test suite optimization. Result. Our experimental results, based on the Defects4J defect dataset, uncovers that: (1) the confounding effect of test suite size on the associations between test effectiveness metrics and defect detection capability in general exists; (2) the proposed linear regression-based method can effectively remove the confounding effect; and (3) after removing the confounding effect, mutation score demonstrates superior effectiveness in predicting test suite effectiveness, while statement coverage is the least effective metric. Furthermore, both coverage-based and mutation-based test suite reduction exhibit enhanced cost-effectiveness in defect detection, and there is a marginal improvement in the speed of defect detection for coverage-based test case prioritization. Conclusion . When using test effectiveness metrics to assess test suite effectiveness, it is crucial to eliminate the influence of test suite size.

Machine Learning (ML) software employs statistical algorithms to perform high-stake tasks in our daily lives, whose results are usually discriminatory due to protected features (e.g., gender), i.e., one part (called privileged, e.g., male) may be more likely to obtain beneficial decisions than the other part (called unprivileged, e.g., female). In alleviating the unfairness, developers have obtained widely held beliefs about the tradeoff between performance and fairness for ML software. Surprisingly, recent research on feature engineering suggests that enlarging the feature set is the perfect way to kill two birds with one stone, i.e., achieving both higher performance and fairness. However, the experiments used in the prior study did not remove the effect of protected features, which have been suggested to be excluded in both industrial applications and academic studies. As a result, the study did not fully explore the tradeoff between performance and fairness. In this article, we first conduct an empirical study to replicate this prior study after excluding the protected features and observe that there is still a tradeoff between performance and fairness with enlarging the features, i.e., more features are not perfect, which would lead to higher performance and lower fairness. Due to more features causing more collection and pre-processing budgets, we aim to search for an effective alternative. Inspired by the “less is more” principle, we propose a novel feature ranking method, Hybrid-importance and Early-validation based Feature Ranking (HEFR) , to find an efficient subset to replace the full feature set with comparable performance and fairness. Our method, HEFR, employs hybrid feature importances to combine performance and fairness and conducts early validation to check the effectiveness of hybrid importances. We conduct experiments on seven datasets and three classifiers to evaluate our method with five baselines. The results have shown that (a) HEFR is efficient for ML software feature engineering: applying HEFR to choose about 10% of features would construct ML software with better or comparable performance and fairness, and (b) HEFR is actionable with small dataset sizes: applying HEFR with only 10% data size would still help choose the proper feature subset.

Code-line-Ievel defect prediction (CLDP) is an effective technique to incorporate comprehensive measures for buggy line identification to optimize efforts in Software Quality Assurance activities. Most CLDP methods either consider the textual information of the code or rely merely on file-level label information, which have not fully leveraged the essential information in the CLDP context, with historical code-line-level labels being incredibly overlooked in their application. Due to the vast number of code lines and the sparsity of the tokens they contain, leveraging historical code-line-level label information remains a significant challenge. To address this issue, we propose a novel CLDP method, Spectrum infOrmation and caUsality aNalysis based coDe-line-level defect prediction (SOUND). SOUND incorporates two key ideas: (a) it introduces a spectrum information perspective, utilizing labels from historical defective lines to quantify the contribution of tokens to line-level defects, and (b) it applies causal analysis to obtain a more systematic and comprehensive understanding of the causal relationships between tokens and defects. After conducting a comprehensive study involving 142 releases across 19 software projects, the experimental results demonstrate that our method significantly outperforms existing state-of-the-art (SOTA) CLDP baseline methods in terms of its ability to rank defective lines under three indicators, IFA, Recall@Top20%LOC, and Effort@Top20%Recall. Notably, in terms of IFA, our method achieves a score of 0 in most cases, indicating that the first line in the ranking list generated by our method is actually defective, significantly enhancing its practicality.

During software maintenance and evolution, developers spend more than half of their time on code comprehension activities. In order to understand an unfamiliar code base, they would naturally ask different types of questions related to code snippets and try to find the answers. In this paper, we conduct an initial work to explore the possibility of automatic question generation for program comprehension. We construct a large-scale data set containing pairs of source code and questions that are automatically transformed from inline comments based on dependency analysis and semantic role labeling. We also build a comprehensive taxonomy of question types so as to generate questions concerning different aspects of code snippets, such as purpose, implementation details and so on. Then, we propose a deep learning-based prototype CodeQG to automatically generates multiple types of questions for code snippets. We evaluate CodeQG by using both typical performance metrics and manual evaluation. The results show that (1) we can achieve a value of 42.02 on BLEU4 and 60.81 on ROUGE-L for the generated questions; (2) overall, the questions are very correct in grammatical, semantic and format; (3) the questions are related to the corresponding code snippet and are helpful for developers in source code comprehension activities. Our work gives insights into automatically generating multiple types of questions for code comprehension. We expect this exploration will improve the applicability and generality of machine code comprehension.

ABSTRACT The relationship between test code and production code, that is, test‐to‐code traceability, plays an essential role in the verification, reliability, and certification of software systems. Prior work on test‐to‐code traceability focuses mainly on Java. However, as Python allows more flexible testing styles, it is still unknown whether existing traceability approaches work well on Python projects. In order to address this gap in knowledge, this paper evaluates whether existing traceability approaches can accurately identify test‐to‐code links in Python projects. We collected seven popular Python projects and carried out an exploratory study at both the method and module levels (involving a total of 3198 test cases). On these projects, we evaluated 15 individual traceability techniques along with cross‐level information propagation and four combining resolution strategies. The results reveal that the performance of test‐to‐code traceability approaches on Python has many differences with Java: (1) most of the existing techniques have poor effectiveness for Python; (2) after augmenting with cross‐level information, the recall surprisingly drops; and (3) machine learning based combination approach achieves the best recall but the worst precision. These findings shed light on the best traceability approaches for Python projects, and also provide guidelines for researchers and the Python community.

Deep learning (DL) models have proven to be highly successful and are now essential to our everyday routines. However, DL models, like traditional software, inevitably contain bugs that affect their performance in real-world scenarios. Effective software engineering techniques are necessary to ensure their dependability. In recent years, fault localization methods for DL models have gained significant attention as a valuable tool for improving the reliability of DL models. Owing to the data-driven programming paradigm, traditional fault localization techniques are challenging to apply directly to DL programs. Previous studies have shown that neuron errors within models can lead to abnormal behavior, and they fix the DL model errors from the perspective of neurons. Nonetheless, there remains a significant gap between the DL program statement and model errors. To tackle this problem, this paper proposes a novel fault localization method for DL models, named wei G hted s U sp I ciousness an D balanc E d agg R egation ( \(\mathsf{GUIDER}\) ) that revisits the idea and challenge of spectrum-based fault localization in the context of DL models. For pre-trained DL models, \(\mathsf{GUIDER}\) utilizes neuron coverage information and test case confidence to compute weighted neuron suspiciousness values and employs balanced aggregation methods to elevate these values from the neuron level to the layer level, which establishes a bridge between the DL model and the DL program, facilitating the developers’ debugging process. We evaluate \(\mathsf{GUIDER}\) using 161 real model bugs collected from StackOverflow and five state-of-the-art fault localization methods for DL models as baselines. The results indicate that (a) our method successfully localizes 67% of the model bugs by ranking the buggy layer to the first place (i.e., top- \(1\) ), significantly outperforming all five baselines, and (b) our method maintains an acceptable time overhead compared with all baseline methods.

The co-evolution of production and test code (PT co-evolution) has received increasing attention in recent years. However, we found that existing work did not comprehensively study various PT co-evolution scenarios, such as the qualification and persistence of their effects on software. Inspired by technical debt (TD), we refer to TD generated during the co-evolution between production and test code as PT co-evolution technical debt (PTCoTD). To better understand PT co-evolution, we first conducted an exploratory study on its characteristics on 15 open-source projects, finding unbalanced PT co-evolution prevalent and summarizing five potential PT flaws. Then we proposed an approach to identify and quantify PTCoTDs of these flaw patterns, considering evolutionary and structural relationships. We also built prediction models to describe cost trajectories and rank all PTCoTDs to prioritize expensive ones. The evaluation on the 15 projects shows that our approach can identify PTCoTDs that deserve attention. The identified PTCoTDs account for about half of the project's total maintenance costs, and the cost proportion of the expensive Top-5 is 1.8x more than the file proportion they contain. Almost all covered maintenance costs persist as PTCoTD in the future, with an average increase of 6.8% between the last two releases. Our approach also accurately predicts the costs of PTCoTD with an average prediction deviation of only 8.3%. Our study provides valuable insights into PT co-evolution scenarios and their effects, which can guide practices and inspire future work on software testing and maintenance.

Tests, as an essential artifact, should co-evolve with the production code to ensure that the associated production code satisfies specification. However, developers often postpone or even forget to update tests, making the tests outdated and lag behind the code. To predict which tests need to be updated when production code is changed, it is challenging to identify all related tests and determine their change probabilities due to complex change scenarios. This paper fills the gap and proposes a hybrid approach named COTE to predict code-to-test co-evolution. We first compute the linked test candidates based on different code-to-test dependencies. After that, we identify common co-change patterns by building a method-level dependence graph. For the remaining ambiguous patterns, we leverage a pre-trained language model which captures the semantic features of code and the change reasons contained in commit messages to judge one test’s likelihood of being updated. Experiments on our datasets consisting of 6,314 samples extracted from 5,000 Java projects show that COTE outperforms state-of-the-art approaches, achieving a precision of 89.0% and a recall of 71.6%. This work can help practitioners reduce test maintenance costs and improve software quality.

Test automation intrusive to the devices under test is difficult to apply on closed or uncommon touch screen systems, e.g., a Switch game console or a digital instrument running a self-defined operating system. There is a lack of non-intrusive test automation techniques for situations where intrusive testing is impossible or not easy to apply. This paper presents RoScript, a novel robotic visual GUI testing system for truly non-intrusive test automation of touch screen applications. RoScript expresses GUI actions in visual test scripts and executes them via a physical robot. A key innovation of RoScript is a test engine armed with environment calibration techniques to achieve automated test execution without manually setting any environment parameter or adjusting the robot arms for a new subject under test. Additionally, two complementary computer vision-based methods are also introduced to record test scripts from videos of human actions on a touch screen. The RoScript test automation does not rely on the internal system of a device under test, making it truly non-intrusive and suitable for touch screen applications running on almost any platform. We evaluated RoScript on a diverse range of devices--including three Android/iOS phones, a Windows tablet, a Linux-based Raspberry Pi, a GoPro camera, and a Switch game console--across over 1100 GUI actions in 160 test scenarios. The results demonstrate RoScript’s high accuracy in test execution: 94% for executing test scripts and 97% for replicating GUI actions. Furthermore, RoScript accurately recorded about 85% of human touch screen actions into test code. These results highlight RoScript’s potential as a truly non-intrusive, cross-platform solution for GUI test automation.

In the rapidly advancing field of software development, the demand for practical code translation tools has surged, driven by the need for interoperability across different programming environments. Existing learning-based approaches often need help with low-resource programming languages that lack sufficient parallel code corpora for training. To address these limitations, we propose a novel training framework that begins with monolingual seed corpora, generating parallel datasets via back-translation and incorporating compiler feedback to optimize the translation model.As a case study, we apply our method to train a code translation model for a new-born low-resource programming language, Cangjie. We also construct a parallel test dataset forJava-to-Cangjietranslation and test cases to evaluate the effectiveness of our approach. Experimental results demonstrate that compiler feedback greatly enhances syntactical correctness, semantic accuracy, and test pass rates of the translatedCangjiecode. These findings highlight the potential of our method to support code translation in low-resource settings, expanding the capabilities of learning-based models for programming languages with limited data availability.

In the wake of developments in the field of Natural Language Processing, Question Answering (QA) software has penetrated our daily lives. Due to the data-driven programming paradigm, QA software inevitably contains bugs, i.e., misbehaving in real-world applications. Current testing techniques for testing QA software include two folds, reference-based testing and metamorphic testing.

Search-based unit test generation methods have been considered effective and widely applied, and Large Language Models (LLMs) have also demonstrated their powerful generation ability. Therefore, some scholars have proposed using LLMs to enhance search-based unit test generation methods and have preliminarily confirmed that LLMs can help alleviate the problem of test coverage plateaus. However, it is still unclear when and how LLMs should intervene in the time-consuming test generation process. This paper explores the application of LLMs at various stages of search-based test generation (SBTG) (including the initial stage, the test generation period, and the test coverage plateaus), as well as strategies for controlling the frequency of LLM intervention. A comprehensive empirical study was conducted on 486 Python benchmark modules from 27 projects. The experimental results show that 1) LLM intervention has a positive effect at any stage, whether to improve coverage over a fixed period or to reduce the time to reach a specific coverage; 2) a reasonable intervention frequency is crucial for LLMs to have a positive effect on SBTG. This work can better help understand when and how LLMs should be applied in SBTG and provide valuable suggestions for developers in practice.

Call graphs facilitate various tasks in software engineering. However, for the dynamic language Python, the complex language features and external library dependencies pose enormous challenges for building the call graphs of real projects. Some program analysis techniques used for call graph construction in other languages are impractical for Python. In this paper, we present STAR, a practical technique for the construction of Python static call graphs. We reformulate call graph construction as an entity identification task. STAR leverages inter-module summary and cross-project dependencies to construct a fine-grained entity knowledge base to identify the possible nodes and edges of the call graph in the code, and then construct the call graph. Our evaluation of three benchmarks shows that (1) STAR improves recall in three benchmarks compared to three baseline tools. Especially, STAR improves the recall of reachable nodes and reachable edges compared with the state-of-the-art tool by 11.3% and 9.8%, respectively; (2) STAR achieves comparable performance as three baseline tools in execution time and memory usage and is more efficient in large projects; (3) STAR can be effectively used for the task of detecting vulnerability propagation with real-world cases. We expect our results will attract more exploration of practical methods and improve the application of Python call graphs.

Code comment generation typically refers to the process of generating concise natural language descriptions for a piece of code, which facilitates program comprehension activities. Inline code comments, as a part of code comments, are also crucial for program comprehension. Recently, the emergence of large language models (LLMs) has significantly boosted the performance of natural language processing tasks. This naturally inspires us to explore the performance of the LLMs in the task of inline code comment generation. To this end, we evaluate open-source LLMs on a large-scale dataset and compare the results with the current state-of-the-art methods. Specifically, we explore the model performance in the following scenarios based on the widely used evaluation metrics (i.e. BLEU, Meteor, and ROUGE-L): (1) generation with simple instruction; (2) few-shot-guided generation with random examples selected from the database; (3) few-shot-guided generation with similar examples selected from the database; and (4) adopt the re-ranking strategy for the output of LLMs. Our findings reveal that: (1) under the simple instruction scenario, LLMs could not fully show the potential in the task of inline comment generation compared to the state-of-the-art models; (2) random few-shot leads to a slight improvement; (3) similar few-shot and re-ranking strategy could significantly enhance the performance of LLMs; and (4) for inline comment and code snippet pairs with different intents, why category achieves the best performance and what category achieves relatively poorer performance. That remains consistent across all four scenarios. Our findings shed light on future research directions for using LLMs in inline comment generation tasks.

As a fundamental component of software documentation, code comments could help developers comprehend and maintain programs. Several datasets of method header comments have been proposed in previous studies for machine learning-based code comment generation. As part of code comments, inline code comments are also crucial for code understanding activities. However, unlike method header comments written in a standard format and describing the whole method code, inline comments are often written in arbitrary formats by developers due to timelines pressures and different aspects of code snippets in the method are described. Currently, there is no large-scale dataset used for inline comments generation considering these. Hence, this naturally inspires us to explore whether we can construct a dataset to foster machine learning research that not only performs fine-grained noise-cleaning but conducts a taxonomy of inline comments. To this end, we first collect inline comments and code snippets from 8000 Java projects on GitHub. Then, we conduct a manual review to obtain heuristic rules, which could be used to clean the data noise in a fine-grained manner. As a result, we construct a large-scale benchmark dataset named ICG with 5,740,770 pairs of inline comments and code snippets. We then build a comprehensive taxonomy and conduct a statistical and manual analysis to explore the performances of different categories of inline comments, such as helpfulness in code understanding. After that, we provide and compare several baseline models to automatically generate inline comments, such as CodeBERT, to enhance the usability of the benchmark for researchers. The availability of our benchmark and baselines can help develop and validate new inline comment generation methods, which would also further facilitate code understanding activities.

Abstract Gradual typing integrates static and dynamic typing by introducing a dynamic type and a consistency relation. A problem of gradual type systems is that dynamic types can easily hide erroneous data flows since consistency relations are not transitive. Therefore, a more rigorous static check is required to reveal these hidden data flows statically. However, in order to preserve the expressiveness of gradually typed languages, static checks for gradually typed languages cannot simply reject programs with potentially erroneous data flows. By contrast, a more reasonable request is to show how these data flows can affect the execution of the program. In this paper, we propose and formalize Static Blame , a framework that can reveal hidden data flows for gradually typed programs and establish the correspondence between static-time data flows and runtime behavior. With this correspondence, we build a classification of potential errors detected from hidden data flows and formally characterize the possible impact of potential errors in each category on program execution, without simply rejecting the whole program. We implemented Static Blame on Grift, an academic gradually typed language, and evaluated the effectiveness of Static Blame by mutation analysis to verify our theoretical results. Our findings revealed that Static Blame exhibits a notable level of precision and recall in detecting type-related bugs. Furthermore, we conducted a manual classification to elucidate the reasons behind instances of failure. We also evaluated the performance of Static Blame, showing a quadratic growth in run time as program size increases.

Summary Natural language inference (NLI) is a task to infer the relationship between the premise and hypothesis sentences, whose models have essential applications in the many natural language processing (NLP) fields, for example, machine reading comprehension and recognizing textual entailment. Due to the data‐driven programming paradigm, bugs inevitably occur in NLI models during the application process, which calls for novel automatic testing techniques to deal with NLI testing challenges. The main difficulty in achieving automatic testing for NLI models is the oracle problem; that is, it may be too expensive to label NLI model inputs manually and hence be too challenging to verify the correctness of model outputs. To tackle the oracle problem, this study proposes a novel automatic testing method hybrid mutation driven testing (HMT), which extends the mutation idea applied in other NLP domains successfully. Specifically, as there are two sets of sentences, that is, premise and hypothesis, to be mutated, we propose four mutation operators to achieve the hybrid mutation strategy, which mutate the premise and the hypothesis sentences jointly or individually. We assume that the mutation would not affect the outputs; that is, if the original and mutated outputs are inconsistent, inconsistency bugs could be detected without knowing the true labels. To evaluate our method HMT, we conduct experiments on two widely used datasets with two advanced models and generate more than 520,000 mutations by applying our mutation operators. Our experimental results show that (a) our method, HMT, can effectively generate mutated testing samples, (b) our method can effectively trigger the inconsistency bugs of the NLI models, and (c) all four mutation operators can independently trigger inconsistency bugs.

Abstract Issue tracking systems are now prevalent in software development, which would help developers submit and discuss issues to solve development problems on software projects. Most previous studies have been conducted to analyze issue relations within projects, such as recommending similar or duplicate bug issues. However, along with the popularization of co‐developing through multiple projects, many issues are cross‐project correlated (CPC), that is, one issue is associated with another issue in a different project. When developers meet with CPC issues, it may primarily increase the difficulties of solving them because they need information from not only their projects but also other related projects that developers are not familiar with. Identifying a CPC issue as early as possible is a fundamental challenge for both managers and developers to allocate the resources for software maintenance and estimate the effort to solve it. This paper proposes 11 issue metrics of two groups to describe textual summary and reporters' activity, which can be extracted just after the issue was reported. We employ these 11 issue metrics to construct just‐in‐time (JIT) prediction models to identify CPC issues. To evaluate the effect of CPC issue prediction models, we conduct experiments on 16 open‐source data science and deep learning projects and compare our prediction model with two baseline models based on textual features (i.e., Term Frequency‐Inverse Document Frequency [TF‐IDF] and Word Embedding), which are commonly adopted by previous studies on issue prediction. The results show that the JIT prediction model based on issue metrics has significantly improved the performance of CPC issue prediction under two evaluation indicators, Matthew's correlation coefficient (MCC) and F1. In addition, we find that the prediction model is more suitable for large‐scale complex core projects in the open‐source ecosystem.

Python’s dynamic typing nature provides developers with powerful programming abstractions. However, many type-related bugs are accumulated in code bases of Python due to the misuse of dynamic typing. The goal of this article is to aid in the understanding of developers’ high-risk practices toward dynamic typing and the early detection of type-related bugs. We first formulate the rules of six types of risky dynamic typing-related practices (type smells for short) in Python. We then develop a rule-based tool named RUPOR, which builds an accurate type base to detect type smells. Our evaluation shows that RUPOR outperforms the existing type smell detection techniques (including the Large Language Models–based approaches, Mypy, and PYDYPE) on a benchmark of 900 Python methods. Based on RUPOR, we conduct an empirical study on 25 real-world projects. We find that type smells are significantly related to the occurrence of post-release faults. The fault-proneness prediction model built with type smell features slightly outperforms the model built without them. We also summarize the common patterns, including inserting type check to fix type smell bugs. These findings provide valuable insights for preventing and fixing type-related bugs in the programs written in dynamic-typed languages.

In recent years, dynamic languages such as Python have become popular due to their flexibility and productivity. The lack of static typing makes programs face the challenges of fixing type errors, early bug detection, and code understanding. To alleviate these issues, PEP 484 introduced optional type annotations for Python in 2014, but unfortunately, a large number of programs are still not annotated by developers. Annotation generation tools can utilize type inference techniques. However, several important aspects of type annotation generation are overlooked by existing works, such as in-depth effectiveness analysis, potential improvement exploration, and practicality evaluation. And it is unclear how far we have been and how far we can go. In this paper, we set out to comprehensively investigate the effectiveness of type inference tools for generating type annotations, applying three categories of state-of-the-art tools on a carefully-cleaned dataset. First, we use a comprehensive set of metrics and categories, finding that existing tools have different effectiveness and cannot achieve both high accuracy and high coverage. Then, we summarize six patterns to present the limitations in type annotation generation. Next, we implement a simple but effective tool to demonstrate that existing tools can be improved in practice. Finally, we conduct a controlled experiment showing that existing tools can reduce the time spent annotating types and determine more precise types, but cannot reduce subjective difficulty. Our findings point out the limitations and improvement directions in type annotation generation, which can inspire future work.

Background. Software testing is a critical activity for ensuring the quality and reliability of software systems. To evaluate the effectiveness of different test suites, researchers have developed a variety of metrics. Problem. However, comparing these metrics is challenging due to the lack of a standardized evaluation framework including comprehensive factors. As a result, researchers often focus on single factors (e.g., size), which finally leads to different or even contradictory conclusions. After comparing dozens of pieces of work in detail, we have found two main problems most troubling to our community: (1) researchers tend to oversimplify the description of the ground truth they use, and (2) data involving real defects is not suitable for analysis using traditional statistical indicators. Objective. We aim at scrutinizing the whole process of comparing test suites for our community. Method. To hit this aim, we propose a framework ASSENT (ev A luating te S t S uite E ffective N ess me T rics) to guide the follow-up research for evaluating a test suite effectiveness metric. ASSENT consists of three fundamental components: ground truth, benchmark test suites, and agreement indicator. Its functioning is as follows: first, users clarify the ground truth for determining the real order in effectiveness among test suites. Second, users generate a set of benchmark test suites and derive their ground truth order in effectiveness. Third, users use the metric to derive the order in effectiveness for the same test suites. Finally, users calculate the agreement indicator between the two orders derived by two metrics. Result. With ASSENT, we are able to compare the accuracy of different test suite effectiveness metrics. We apply ASSENT to evaluate representative test suite effectiveness metrics, including mutation score and code coverage metrics. Our results show that, based on the real faults, mutation score, and subsuming mutation score are the best metrics to quantify test suite effectiveness. Meanwhile, by using mutants instead of real faults, test effectiveness will be overestimated by more than 20% in values. Conclusion. We recommend that the standardized evaluation framework ASSENT should be used for evaluating and comparing test effectiveness metrics in the future work.

The emergence of build automation tools with dependency management features has significantly impacted software development. However, in the configuration process, improper settings of some configuration items, such as the dependency scope setting, may cause severe problems in the development process. Improper setting of dependency scope may cause problems such as missing dependencies and redundant dependencies, and may even spread the problem to the downstream of the software ecosystem.

Machine translation systems have penetrated our daily lives, providing translation services from source language to target language to millions of users online daily. Word Sense Disambiguation (WSD) is one of the essential functional requirements of machine translation systems, which aims to determine the exact sense of polysemes in the given context. Commercial machine translation systems (e.g., Google Translate) have been shown to fail in identifying the proper sense and consequently cause translation errors. However, to our knowledge, no prior studies focus on testing such WSD bugs for machine translation systems.

Python has become increasingly popular and widely used in many fields. Dynamic features of Python provide much convenience for developers. However, they can also cause many type-related bugs undetected until runtime, which increases the cost of maintenance. Static type checking is essential to find bugs early, and the introduction of gradual typing and type annotations makes it easier to perform static type analysis. However, it remains to be investigated how well gradual typing improves real bug detection. Therefore, we conducted a comprehensive study on three widely used checkers: MyPy, PyRight, and PyType. We used a benchmark containing 10 popular Python projects with 40 real type-related bugs. First, we performed static type checking on the projects with and without type annotations to evaluate the effectiveness of finding real bugs. Second, we manually analyzed the missing bugs and investigated the reasons. The results show that the three tools can detect 29 of the 40 studied bugs after annotating, while only 14 bugs are detected before annotating. We also found that type annotations can substantially improve the ability of static type checkers to detect real bugs. A detailed analysis of bugs missed by the checkers shows that: (i) the accuracy of type analysis is challenged when it comes to programs with complicated dynamic features, such as dynamically changing object’s attributes, even with annotations; (ii) the inaccurate type annotations can undermine the ability of static type checkers to detect real bugs; (iii) static type checkers have different checking strategies in some cases, which has an impact on real bug detection. Our study can not only enable developers to better understand static type checking and make better use of them but also guide future research.

Background. Code-line-level bugginess identification (CLBI) is a vital technique that can facilitate developers to identify buggy lines without expending a large amount of human effort. Most of the existing studies tried to mine the characteristics of source codes to train supervised prediction models, which have been reported to be able to discriminate buggy code lines amongst others in a target program. Problem. However, several simple and clear code characteristics, such as complexity of code lines, have been disregarded in the current literature. Such characteristics can be acquired and applied easily in an unsupervised way to conduct more accurate CLBI, which also can decrease the application cost of existing CLBI approaches by a large margin. Objective. We aim at investigating the status quo in the field of CLBI from the perspective of (1) how far we have really come in the literature, and (2) how far we have yet to go in the industry, by analyzing the performance of state-of-the-art (SOTA) CLBI approaches and tools, respectively. Method. We propose a simple heuristic baseline solution GLANCE (aimin G at contro L - AN d C ompl E x-statements) with three implementations (i.e., GLANCE-MD, GLANCE-EA, and GLANCE-LR). GLANCE is a two-stage CLBI framework: first, use a simple model to predict the potentially defective files; second, leverage simple code characteristics to identify buggy code lines in the predicted defective files. We use GLANCE as the baseline to investigate the effectiveness of the SOTA CLBI approaches, including natural language processing (NLP) based, model interpretation techniques (MIT) based, and popular static analysis tools (SAT). Result. Based on 19 open-source projects with 142 different releases, the experimental results show that GLANCE framework has a prediction performance comparable or even superior to the existing SOTA CLBI approaches and tools in terms of 8 different performance indicators. Conclusion. The results caution us that, if the identification performance is the goal, the real progress in CLBI is not being achieved as it might have been envisaged in the literature and there is still a long way to go to really promote the effectiveness of static analysis tools in industry. In addition, we suggest using GLANCE as a baseline in future studies to demonstrate the usefulness of any newly proposed CLBI approach.

Managing cross-project dependencies is tricky in modern software development. A primary way to manage dependencies is using dependency configuration files, which brings convenience to the entire software ecosystem, including developers, maintainers, and users. However, developers may introduce dependency smells if dependency configuration files are not well written and maintained. Dependency smells are recurring violations of dependency management in dependency configuration files and can potentially lead to severe consequences. This paper provides an in-depth look at three dependency smells, namely,Missing Dependency,Bloated Dependency, andVersion Constraint Inconsistencyin Python projects. First, we implement a tool calledPythonCross-projectDependency- PyCD to accurately extract dependency information from configuration files. The evaluation result on 212 Python projects shows that PyCD outperforms state-of-the-art tools. Then, we make an empirical study for three dependency smells in 132 Python projects to investigate the pervasiveness, causes, and evolution. The results show that: 1) dependency smells are prevalent in Python projects and exist inconsistently in different projects; 2) dependency smells are introduced into Python projects for different reasons, mainly due to the problems of synchronous update and collaborative development; and 3) dependency smells can be removed with different patterns according to different dependency smells. Furthermore, we report and get responses for 40 harmful dependency smell instances, 34 of which have been responded that these dependency smells do exist in the projects, and 10 instances are fixed or under process. The feedback from developers indicates that dependency smells can have a negative impact on project maintenance. Our study highlights that these dependency smells deserve the attention of developers.

Static analysis (SA) tools can generate useful static warnings to reveal the problematic code snippets in a software system without dynamically executing the corresponding source code. In the literature, static warnings are of paramount importance because they can easily indicate specific types of software defects in the early stage of a software development process, which accordingly reduces the maintenance costs by a substantial margin. Unfortunately, due to the conservative approximations of such SA tools, a large number of false positive (FP for short) warnings (i.e., they do not indicate real bugs) are generated, making these tools less effective. During the past two decades, therefore, many false positive mitigation (FPM for short) approaches have been proposed so that more accurate and critical warnings can be delivered to developers. This paper offers a detailed survey of research achievements on the topic of FPM. Given the collected 130 surveyed papers, we conduct a comprehensive investigation from five different perspectives. First, we reveal the research trends of this field. Second, we classify the existing FPM approaches into five different types and then present the concrete research progress. Third, we analyze the evaluation system applied to examine the performance of the proposed approaches in terms of studied SA tools, evaluation scenarios, performance indicators, and collected datasets, respectively. Fourth, we summarize the four types of empirical studies relating to SA warnings to exploit the insightful findings that are helpful to reduce FP warnings. Finally, we sum up 10 challenges unresolved in the literature from the aspects of systematicness, effectiveness, completeness, and practicability and outline possible research opportunities based on three emerging techniques in the future.

The label quality of defect data sets has a direct influence on the reliability of defect prediction models. In this paper, we conduct a systematic study of inconsistent defect labels in multi-version-project defect data sets, i.e., many instances having the same source code but different labels over multiple versions of a software project. First, we report the phenomena of inconsistent labels by real examples and analyze their essence in the context of defect prediction. Then, we uncover the causes that lead to the occurrence of inconsistent labels for the representative label collection approaches. Finally, we investigate the actual influence of inconsistent labels on defect prediction models. We find that inconsistent labels in general exist in six multi-version-project defect data sets (either widely used or the most up-to-date in the literature) collected by diverse label collection approaches. In particular, inconsistent labels in a training data set significantly reduce the prediction performance of a model, while inconsistent labels in a test data set can lead to a considerable evaluation bias on the real performance. Therefore, we recommend that: on the one hand, researchers leverage our findings to make targeted methodological improvements on existing defect label collection approaches to reduce the generation of inconsistent labels; on the other hand, practitioners detect and exclude inconsistent labels in defect data sets to avoid their potential negative influence on defect prediction.

The ellipsis of the syntactic structure is a common phenomenon in ordinary textual documents. Existing studies have found that despite syntactic ellipsis could help avoid repetition of normative documents, it could also, for example, lead to ambiguity and hamper the understandability of document contents. As a fundamental component of software, code comments are generally written by developers in a non-structured way just like normative documents. This naturally inspires us to explore whether syntactic ellipsis is also a common phenomenon in code comments and what potential negative effects would such ellipsis have on software tasks such as code/comments comprehension activities. Such explorations, in our opinion, are expected to facilitate the research on code comments and comments-related software tasks. To this end, we conduct the first large-scale study to explore the syntactic structure ellipsis problem of code comments, with a focus on Java inline comments. Specifically, we construct a data set of 1,000 Java projects with 1,307,457 inline comments and associated codes. Based on this data set, we first study the prevalence of syntactic structure ellipsis in inline comments. We find that syntactic structure ellipsis is quite common in inline comments where 83.6% comments have structure ellipsis (such as subject/predicate omissions). Then, we investigate the effects of syntactic structure ellipsis on code/comment understanding activities. As a result, we find that there indeed exists a negative relationship between them, with a medium effect size. Based on these findings, we further propose neural network based approaches to complete the ellipsis parts for the inline comments. With our approach, we could achieve: 1) a medium improvement in assisting code/comment understanding activities, and 2) a substantial improvement of 11.3% in comment-assisted code abbreviation extension task.

With the widespread application of machine learning (ML) software, especially in high-risk tasks, the concern about their unfairness has been raised towards both developers and users of ML software. The unfairness of ML software indicates the software behavior affected by the sensitive features (e.g., sex), which leads to biased and illegal decisions and has become a worthy problem for the whole software engineering community.

Optical character recognition (OCR) algorithms often run slow. They may take several seconds to recognize the texts on a GUI screen, which makes OCR-based widget localization in test automation unfriendly for use, especially on GPU-free computers. This paper first concludes a common type of widget text to be located in GUI testing: label text, which are short texts in widgets like buttons, menu items, and window titles. We then investigate the characteristics of texts on a GUI screen and introduce a fast GPU-independent Label Text Screening (LTS) technique to accelerate the OCR process for label text localization. The technique opens the black box of OCR engines and uses a combination of simple methods to avoid excessive text analysis on a screen as much as possible. Experiments show that, on the subject datasets, LTS reduces the average OCR-based label text localization time to a large extent. On 4k resolution GUI screens, it keeps the localization time below 0.5 seconds for over about 60% of cases without GPU support on a normal laptop computer. In contrast, the existing CPU-based approaches built on popular OCR engines Tesseract, PaddleOCR, and EasyOCR usually need over 2 seconds to achieve the same goal on the same platform. Even with GPU acceleration, they can hardly keep the analysis time in 1 second. We believe the proposed approach would be helpful for implementing OCR-based test automation tools.

Background. Mutation testing is a commonly used defect injection technique for evaluating the effectiveness of a test suite. However, it is usually computationally expensive. Therefore, many mutation reduction strategies, which aim to reduce the number of mutants, have been proposed. Problem. It is important to measure the ability of a mutation reduction strategy to maintain test suite effectiveness evaluation. However, existing evaluation indicators are unable to measure the “order-preserving ability”, i.e., to what extent the mutation score order among test suites is maintained before and after mutation reduction. As a result, misleading conclusions can be achieved when using existing indicators to evaluate the reduction effectiveness. Objective. We aim to propose evaluation indicators to measure the “order-preserving ability” of a mutation reduction strategy, which is important but missing in our community. Method. Given a test suite on a Software Under Test (SUT) with a set of original mutants, we leverage the test suite to generate a group of test suites that have a partial order relationship in defect detecting ability. When evaluating a reduction strategy, we first construct two partial order relationships among the generated test suites in terms of mutation score, one with the original mutants and another with the reduced mutants. Then, we measure the extent to which the partial order under the original mutants remains unchanged in the partial order under the reduced mutants. The more partial order is unchanged, the stronger the Order Preservation ( OP ) of the mutation reduction strategy is, and the more effective the reduction strategy is. Furthermore, we propose Effort-aware Relative Order Preservation ( EROP ) to measure how much gain a mutation reduction strategy can provide compared with a random reduction strategy. Result. The experimental results show that OP and EROP are able to efficiently measure the “order-preserving ability” of a mutation reduction strategy. As a result, they have a better ability to distinguish various mutation reduction strategies compared with the existing evaluation indicators. In addition, we find that Subsuming Mutant Selection (SMS) and Clustering Mutant Selection (CMS) are more effective than the other strategies under OP and EROP. Conclusion. We suggest, for the researchers, that OP and EROP should be used to measure the effectiveness of a mutant reduction strategy, and for the practitioners, that SMS and CMS should be given priority in practice.

Python is an interpreted programming language that has been widely used in many fields. The successful execution of a Python program depends on both the correctness of Python program and the correctness of Python interpreter. As an infrastructure software, there are many bugs in the Python interpreter. Exploring the bugs in Python interpreters can help developers and maintainers of Python interpreters detect and fix bugs and help users of Python avoid risks. In this article, we conduct an empirical study on the bugs in two mainstream Python interpreters: CPython and PyPy. By analyzing 25 958 fixed bugs, 18 824 revisions, 2 116 test cases, and root causes of randomly sampled 510 bugs, we have summarized the following findings.1)The distribution of bugs in the Python interpreter is so uneven that the vast majority of bugs are distributed in a few components and source files.2)The scales of the testing programs that reveal bugs are small.3)The fixing works seem to be not complicated since the number of modified source files and lines of code are limited; however, most bugs need a long time to be fixed; nearly 15% of the bugs need more than one year to fix.4)The priorities of bugs are independent of their locations, but they significantly correlate with duration of bugs.5)Semantic bugs are the most frequent root causes of bugs, and their proportion exceeds other types of root causes.These results could indicate some potential problems during the detecting and fixing of Python interpreter’s bugs, and provide some assistance to developers and maintainers of Python interpreters, users of Python, as well as researchers in related fields.

A very important function of an issue tracking system is to assign labels to issue reports, such as bug, feature, enhancement, etc., in order to categorize issues to facilitate various development activities. In practice, it is very common that an issue has multiple labels. However, current works are mainly based on single-label prediction, which are not suitable for just-in-time multi-labeling services, due to the low efficiency. Therefore, in this paper, we propose MULA, a just-in-time MUlti-LAbeling system, which learns and automatically assigns multiple labels to issue reports. We have built a dataset with 81,601 entries and 11 labels, as the first benchmark for this task, and implemented a GitHub app. To the best of our knowledge, this is the first work and tool for online multi-labeling GitHub issues based on their categories. We conduct a comprehensive empirical study, including comparisons with five commonly adopted labeling models that show the superiority of MULA, as well as an evaluation that shows high consistency between MULA’s suggestions and developers’ opinions.

Knowing the effectiveness of a test suite is essential for many activities such as assessing the test adequacy of code and guiding the generation of new test cases. Mutation testing is a commonly used defect injection technique for evaluating the effectiveness of a test suite. However, it is usually computationally expensive, as a large number of mutants (buggy versions) are needed to be generated from a production code under test and executed against the test suite. In order to reduce the expensive testing cost, recent studies proposed to use supervised models to predict the effectiveness of a test suite without executing the test suite against the mutants. Nonetheless, the training of such a supervised model requires labeled data, which still depends on the costly mutant execution. Furthermore, existing models are based on traditional supervised learning techniques, which assume that the training and testing data come from the same distribution. But, in practice, software systems are subject to considerable concept drifts, i.e., the same distribution assumption usually does not hold. This can lead to inaccurate predictions of a learned supervised model on the target code as time progresses. To tackle these problems, in this paper, we propose a Coverage-Based Unsupervised Approach (CBUA) for evaluating the effectiveness of a test suite. Given a production code under test, the corresponding mutants, and a test suite, CBUA first collects the coverage information of the mutated statements in the target production code under the execution of the test suite. Then, CBUA employs coverage to estimate the probability of each mutant being alive. As such, a mutation score is computed to evaluate the test suite effectiveness and the predicted labels (i.e., killed or alive) are obtained. The whole process only requires a one-time execution of the test suite against the target production code, without involving any mutant execution and any training data. CBUA can ensure the score monotonicity property (i.e., adding test cases to a test suite does not decrease its mutation score), which may be violated by a supervised approach. The experimental results show that CBUA is very competitive with the state-of-the-art supervised approaches in prediction accuracy. In particular, CBUA is shown to be more effective in finding mutants that are covered but not killed by a test suite, which is helpful in identifying the weaknesses in the current test suite and generating new test cases accordingly. Since CBUA is an easy-to-implement approach with a low cost, we suggest that it should be used as a baseline approach for comparison when any novel prediction approach is proposed in future studies.

The boom of DL technology leads to massive DL models built and shared, which facilitates the acquisition and reuse of DL models. For a given task, we encounter multiple DL models available with the same functionality, which are considered as candidates to achieve this task. Testers are expected to compare multiple DL models and select the more suitable ones w.r.t. the whole testing context. Due to the limitation of labeling effort, testers aim to select an efficient subset of samples to make an as precise rank estimation as possible for these models. To tackle this problem, we propose Sample Discrimination based Selection (SDS) to select efficient samples that could discriminate multiple models, i.e., the prediction behaviors (right/wrong) of these samples would be helpful to indicate the trend of model performance. To evaluate SDS, we conduct an extensive empirical study with three widely-used image datasets and 80 real world DL models. The experiment results show that, compared with state-of-the-art baseline methods, SDS is an effective and efficient sample selection method to rank multiple DL models.

LLVM is a widely adopted compiler tool-chain, and its quality is critical to the projects relying on it. The bugs in LLVM may lead to an unpredictably large impact on the community. Usually, tools in LLVM need to interact with each other to finish a particular task. This working style brings up a specific type of bugs that involve multiple tools, namely, tool-chain bugs, which complicates the bug detection and debugging. Though there exist prior works about LLVM bugs, none of them has investigated this particular type of bugs in-depth. Thus, in this paper, we conduct an empirical study of the LLVM tool-chain bugs, aiming to provide the first comprehensive understanding of these bugs. Overall, we find 1723 tool-chain bugs. We reveal frequently occurred tool combinations, six typical interaction reasons, as well as four commonly seen failure symptoms. Through the linkages between these bugs and their fixing commits, we identify six common root causes, and summarize debugging manners. Based on the above findings, we highlight three insights into testing and debugging these bugs.

Background. Self-admitted technical debt (SATD) is a special kind of technical debt that is intentionally introduced and remarked by code comments. Those technical debts reduce the quality of software and increase the cost of subsequent software maintenance. Therefore, it is necessary to find out and resolve these debts in time. Recently, many automatic approaches have been proposed to identify SATD. Problem. Popular IDEs support a number of predefined task annotation tags for indicating SATD in comments, which have been used in many projects. However, such clear prior knowledge is neglected by existing SATD identification approaches when identifying SATD. Objective. We aim to investigate how far we have really progressed in the field of SATD identification by comparing existing approaches with a simple approach that leverages the predefined task tags to identify SATD. Method. We first propose a simple heuristic approach that fuzzily Matches task Annotation Tags ( MAT ) in comments to identify SATD. In nature, MAT is an unsupervised approach, which does not need any data to train a prediction model and has a good understandability. Then, we examine the real progress in SATD identification by comparing MAT against existing approaches. Result. The experimental results reveal that: (1) MAT has a similar or even superior performance for SATD identification compared with existing approaches, regardless of whether non-effort-aware or effort-aware evaluation indicators are considered; (2) the SATDs (or non-SATDs) correctly identified by existing approaches are highly overlapped with those identified by MAT ; and (3) supervised approaches misclassify many SATDs marked with task tags as non-SATDs, which can be easily corrected by their combinations with MAT . Conclusion. It appears that the problem of SATD identification has been (unintentionally) complicated by our community, i.e., the real progress in SATD comments identification is not being achieved as it might have been envisaged. We hence suggest that, when many task tags are used in the comments of a target project, future SATD identification studies should use MAT as an easy-to-implement baseline to demonstrate the usefulness of any newly proposed approach.

Software projects are increasingly forming social-technical ecosystems within which individual projects rely on the infrastructures or functional components provided by other projects, leading to complex inter-dependencies. Through inter-project dependencies, a bug in an upstream project may have profound impact on a large number of downstream projects, resulting in cross-project bugs. This emerging type of bugs has brought new challenges in bug fixing due to their unclear influence on downstream projects. In this paper, we present an approach to estimating the impact of a cross-project bug within its ecosystem by identifying the affected downstream modules (classes/methods). Note that a downstream project that uses a buggy upstream function may not be affected as the usage does not satisfy the failure inducing preconditions. For a reported bug with the known root cause function and failure inducing preconditions, we first collect the candidate downstream modules that call the upstream function through an ecosystem-wide dependence analysis. Then, the paths to the call sites of the buggy upstream function are encoded as symbolic constraints. Solving the constraints, together with the failure inducing preconditions, identifies the affected downstream modules. Our evaluation of 31 existing upstream bugs on the scientific Python ecosystem containing 121 versions of 22 popular projects (with a total of 16 millions LOC) shows that the approach is highly effective: from the 25490 candidate downstream modules that invoke the buggy upstream functions, it identifies 1132 modules where the upstream bugs can be triggered, pruning 95.6% of the candidates. The technique has no false negatives and an average false positive rate of 7.9%. Only 49 downstream modules (out of the 1132 we found) were reported before to be affected.

Existing intrusive test automation techniques for touch screen applications (e.g., Appium and Sikuli) are difficult to work on many closed or uncommon systems, such as a GoPro. Being non-intrusive can largely extend the application scope of the test automation techniques. To this end, this paper presents RoScript, a truly non-intrusive test-script-driven robotic testing system for test automation of touch screen applications. RoScript leverages visual test scripts to express GUI actions on a touch screen application and uses a physical robot to drive automated test execution. To reduce the test script creation cost, a non-intrusive computer vision based technique is also introduced in RoScript to automatically record touch screen actions into test scripts from videos of human actions on the device under test. RoScript is applicable to touch screen applications running on almost arbitrary platforms, whatever the underlying operating systems or GUI frameworks are. We conducted experiments applying it to automate the testing of 21 touch screen applications on 6 different devices. The results show that RoScript is highly usable. In the experiments, it successfully automated 104 test scenarios containing over 650 different GUI actions on the subject applications. RoScript accurately performed GUI actions on over 90% of the test script executions and accurately recorded about 85% of human screen click actions into test code.

The dynamic typing discipline of Python allows developers to program at a high level of abstraction. However, type related bugs are commonly encountered in Python systems due to the lack of type declaration and static type checking. Especially, the misuse of dynamic typing discipline produces underlying bugs and increases maintenance efforts. In this paper, we introduce six types of dynamic typing related practices in Python programs, which are the common but potentially risky usage of dynamic typing discipline by developers. We also implement a tool named PYDYPE to detect them. Based on this tool, we conduct an empirical study on nine real-world Python systems (with the size of more than 460KLOC) to understand dynamic typing related practices. We investigate how widespread the dynamic typing related practices are, why they are introduced into the systems, whether their usage correlates with increased likelihood of bug occurring, and how developers fix dynamic typing related bugs. The results show that: (1) dynamic typing related practices exist inconsistently in different systems and Inconsistent Variable Types is most prevalent; (2) they are introduced into systems mainly during early development phase to promote development efficiency; (3) they have a significant positive correlation with bug occurring; (4) developers tend to add type checks or exception handling to fix dynamic typing related bugs. These results benefit future research in coding convention, language design, bug detection and fixing.

Blocking bugs are a severe type of bugs that prevent other bugs from being fixed. As software becomes increasingly complex and large, blocking bugs occur in many large-scale software, especially in software ecosystems. Blocking bugs may have a high negative impact on software development and maintenance. Usually, blocking bugs preventing more bugs should be more concerned. In this paper, we focus on a special type of blocking bugs that block at least two bugs, which we call Critical Blocking Bugs (CBBs).

Bug reports in a repository are generally organized line by line in a list-view, with their titles and other meta-data displayed. In this list-view, a concise and precise title plays an important role that enables project practitioners to quickly and correctly digest the core idea of the bug, without carefully reading the corresponding details. However, the quality of bug report titles varies in open-source communities, which may be due to the limited time and unprofessionalism of authors. To help report authors efficiently draft good-quality titles, we propose a method, named iTAPE, to automatically generate titles for their bug reports. iTAPE formulates title generation into a one-sentence summarization task. By properly tackling two domain-specific challenges (i.e. lacking off-the-shelf dataset and handling the low-frequency human-named tokens), iTAPE then generates titles using a Seq2Seq-based model. A comprehensive experimental study shows that iTAPE can obtain fairly satisfactory results, in terms of the comparison with three latest one-sentence summarization works, as well as the feedback from human evaluation.

With the increasing application of deep learning (DL) models in many safety-critical scenarios, effective and efficient DL testing techniques are much in demand to improve the quality of DL models. One of the major challenges is the data gap between the training data to construct the models and the testing data to evaluate them. To bridge the gap, testers aim to collect an effective subset of inputs from the testing contexts, with limited labeling effort, for retraining DL models.

For the test case prioritization problems, the average percent of faults detected (APFD) and its variant versions are widely used as metrics to evaluate prioritized test suite’s efficiency of fault detection. By a revisit of metrics for test case prioritization, we observe that APFD is only available for the scenarios where all test suites under evaluation contain the same number of test cases. Such a limitation is often overlooked, and lead to incorrect results when comparing fault detection efficiency of test suites with different sizes. Moreover, APFD cannot precisely illustrate the process of fault detection in the real world. Besides the APFD, most of its variants, including the NAPFD and the APFD[Formula: see text], have similar problems. This paper points out these limitations in detail by analyzing the physical explanation of APFD series metrics formally. In order to eliminate these limitations, we propose a series of improved metrics, including the relative average percent of faults detected (RAPFD) and the relative cost-cognizant weighted average percent of faults detected (RAPFD[Formula: see text]), to evaluate the efficiency of the test suite. Furthermore, for the scenario of parallel testing, a series of metrics including the relative average percent of faults detected in parallel testing ([Formula: see text]-RAPFD) and the relative cost-cognizant weighted average percent of faults detected in parallel testing ([Formula: see text]-RAPFD[Formula: see text]) are proposed too. All the proposed metrics refer to both the speed of fault detection and the constraint of the testing resource. A formal analysis and some examples show that all the proposed metrics provide much more precise illustrations of the fault detection process.

Generic programming is a key paradigm for developing reusable software components. The inherent support for generic constructs is therefore important in programming languages. As for C++, the generic construct, templates, has been supported since the language was first released. However, little is currently known about how C++ templates are actually used in developing real software. In this study, we conduct an experiment to investigate the use of templates in practice. We analyze 1,267 historical revisions of 50 open source systems, consisting of 566 million lines of C++ code, to collect the data of the practical use of templates. We perform statistical analyses on the collected data and produce many interesting results. We uncover the following important findings: (1) templates are practically used to prevent code duplication, but this benefit is largely confined to a few highly used templates; (2) function templates do not effectively replace C-style generics, and developers with a C background do not show significant preference between the two language constructs; (3) developers seldom convert dynamic polymorphism to static polymorphism by using CRTP (Curiously Recursive Template Pattern); (4) the use of templates follows a power-law distribution in most cases, and C++ developers who prefer using templates are those without other language background; (5) C developer background seems to override C++ project guidelines. These findings are helpful not only for researchers to understand the tendency of template use but also for tool builders to implement better tools to support generic programming.

Software systems becomes increasingly complex for the wide use of social-software-development platforms, such as GitHub. Complicated inter-dependencies within ecosystems impose new challenges in resolving the blocking bug pairs, in which the upstream bugs prevent the downstream bugs to be fixed. Generally, the downstream bugs cannot be fixed until the upstream bugs are fixed in blocking bug pairs, which keeps the downstream developers waiting for a long time. However, previous research found that some blocking pairs could be "broken" through a workaround, i.e., a temporary solution proposed by the downstream developers before the upstream bugs get fixed. In this paper, we propose an approach to describe and predict the breakability of the blocking bug pairs. Our goal is to help the ecosystem developers to predict whether the given blocking bug pair can be broken. We evaluate our approach on two real world ecosystems: Mozilla Firefox and Netbeans, and have the following two main findings. First, the participants within the blocking bug pair and the vitality of the downstream bug have much impact on the breakability. Second, we can build breakability predication models with the accuracy of approximate 80%. Moreover, for predicting unbreakable pairs which affect the downstream projects seriously, our approach achieves a precision over 92%. It can be used to effectively remind the upstream developers to fix these bugs as quickly as possible.

Performance problems widely exist in modern software systems. Existing performance optimization techniques, including profiling-based and pattern-based techniques, usually fail to consider the architectural impacts among methods that easily slow down the overall system performance. This paper contributes a new approach, named Speedoo, to identify groups of methods that should be treated together and deserve high priorities for performance optimization. The uniqueness of Speedoo is to measure and rank the performance optimization opportunities of a method based on 1) the architectural impact and 2) the optimization potential. For each highly ranked method, we locate a respective Optimization Space based on 5 performance patterns generalized from empirical observations. The top ranked optimization spaces are suggested to developers as potential optimization opportunities. Our evaluation on three real-life projects has demonstrated that 18.52% to 42.86% of methods in the top ranked optimization spaces indeed undertook performance optimization in the projects. This outperforms one of the state-of-the-art profiling tools YourKit by 2 to 3 times. An important implication of this study is that developers should treat methods in an optimization space together as a group rather than as individuals in performance optimization. The proposed approach can provide guidelines and reduce developers' manual effort.

Code smells are considered to have negative impacts on software evolution and maintenance. Many researchers have conducted studies to investigate these effects and correlations. However, because code smells constantly change in the evolution, understanding these changes and the correlation between them and the operations of source code files is helpful for developers in maintenance. In this paper, on four popular Java projects with 58 release versions, we conduct an extensive empirical study to investigate the correlation between code smells and basic operations of source code files. We find that, the density of code smells decreases with the software evolution. The files containing smells have a higher likelihood to be modified while smells are not strongly correlated with adding or removing files. Furthermore, some certain smells have significant impact on file changes. These findings are helpful for developers to understand the evolution of code smells and better focus on quality assurance.

Background. Recent years have seen an increasing interest in cross-project defect prediction (CPDP), which aims to apply defect prediction models built on source projects to a target project. Currently, a variety of (complex) CPDP models have been proposed with a promising prediction performance. Problem. Most, if not all, of the existing CPDP models are not compared against those simple module size models that are easy to implement and have shown a good performance in defect prediction in the literature. Objective. We aim to investigate how far we have really progressed in the journey by comparing the performance in defect prediction between the existing CPDP models and simple module size models. Method. We first use module size in the target project to build two simple defect prediction models, ManualDown and ManualUp, which do not require any training data from source projects. ManualDown considers a larger module as more defect-prone, while ManualUp considers a smaller module as more defect-prone. Then, we take the following measures to ensure a fair comparison on the performance in defect prediction between the existing CPDP models and the simple module size models: using the same publicly available data sets, using the same performance indicators, and using the prediction performance reported in the original cross-project defect prediction studies. Result. The simple module size models have a prediction performance comparable or even superior to most of the existing CPDP models in the literature, including many newly proposed models. Conclusion. The results caution us that, if the prediction performance is the goal, the real progress in CPDP is not being achieved as it might have been envisaged. We hence recommend that future studies should include ManualDown/ManualUp as the baseline models for comparison when developing new CPDP models to predict defects in a complete target project.

Continuous Integration (CI) has become a good practice of software development in recent years. As an essential part of CI, build creates software from source code. Predicting build outcome help developers to review and fix bugs before building to save time. However, we are missing objective evidence of practical factors affecting build result. Travis CI provides a hosted, distributed continuous integration service used to build and test software projects hosted at GitHub. The TravisTorrent is a dataset which deeply analyzes source code, process and dependency status of projects hosting on Travis CI. We use this dataset to investigate which factors may impact a build result. We first preprocess TravisTorrent data to extract 27 features. We then analyze the correlation between these features and the result of a build. Finally, we build four prediction models to predict the result of a build and perform a horizontal analysis. We found that in our study, the number of commits in a build (git_num_all_built_commits) is the most import factor that has significant impact on the build result, and SVM performs best in the four of the prediction models we used.

GitHub has fostered complicated and enormous software ecosystems, in which projects depend on and co-evolve with each other. An error in an upstream project may affect its downstream projects through inter-dependencies, forming crossproject bugs. Though the upstream developers should fix the bugs on their side, proposing a workaround, i.e., a temporary solution in the downstream project is a common practice for the downstream developers. In this study, we empirically investigated the characteristics of downstream workarounds in the scientific Python ecosystem. Combining the statistical comparisons and manual inspection, we have the following three main findings. First, in general, the workarounds and the corresponding upstream fixes are significantly different in code size and code structure. Second, there are three kinds of crossproject bugs that the downstream developers usually work around. Last, four types of common patterns are identified from the investigated workarounds. The findings of this study lead to better understanding of cross-project bugs and the practices of developers in software ecosystems.

GitHub, a popular social-software-development platform, has fostered a variety of software ecosystems where projects depend on one another and practitioners interact with each other. Projects within an ecosystem often have complex inter-dependencies that impose new challenges in bug reporting and fixing. In this paper, we conduct an empirical study on cross-project correlated bugs, i.e., causally related bugs reported to different projects, focusing on two aspects: 1) how developers track the root causes across projects, and 2) how the downstream developers coordinate to deal with upstream bugs. Through manual inspection of bug reports collected from the scientific Python ecosystem and an online survey with developers, this study reveals the common practices of developers and the various factors in fixing cross-project bugs. These findings provide implications for future software bug analysis in the scope of ecosystem, as well as shed light on the requirements of issue trackers for such bugs.

Software has been changing during its whole life cycle. Therefore, identification of source code changes becomes a key issue in software evolution analysis. However, few current change analysis research focus on dynamic language software. In this paper, we pay attention to the fine-grained source code changes of Python software. We implement an automatic tool named PyCT to extract 77 kinds of fine-grained source code change types from commit history information. We conduct an empirical study on ten popular Python projects from five domains, with 132294 commits, to investigate the characteristics of dynamic software source code changes. Analyzing the source code changes in four aspects, we distill 11 findings, which are summarized into two insights on software evolution: change prediction and fault code fix. In addition, we provide direct evidence on how developers use and change dynamic features. Our results provide useful guidance and insights for improving the understanding of source code evolution of dynamic language software.

We propose a novel type inference technique for Python programs. Type inference is difficult for Python programs due to their heavy dependence on external APIs and the dynamic language features. We observe that Python source code often contains a lot of type hints such as attribute accesses and variable names. However, such type hints are not reliable. We hence propose to use probabilistic inference to allow the beliefs of individual type hints to be propagated, aggregated, and eventually converge on probabilities of variable types. Our results show that our technique substantially outperforms a state-of-the-art Python type inference engine based on abstract interpretation.

Nowadays concurrent programming is in large demand. The inherent support for concurrency is therefore increasingly important in programming languages. As for C++, an abundance of standard concurrency constructs have been supported since C++11. However, to date there is little work investigating how these constructs are actually used in developing real software. In this paper, we perform an empirical study to investigate the adoption of C++ concurrency constructs in open-source applications, with the goal to provide insightful information for practitioners to use concurrency constructs efficiently. To this end, we analyze 127 open-source applications that adopt C++ concurrency constructs, comprising 34 million lines of C++ code, to conduct the experiment. The experimental results show that: (1) to implement concurrency code, thread-based constructs are significantly more often used than atomics-based constructs and task-based constructs; (2) to manage synchronization, lock-based constructs are significantly more often used than lock-free constructs and blocking constructs; (3) among the key thread-based constructs and task-based constructs (i.e. mutex, promise, and future), there is not a construct significantly more commonly misused than others; (4) small-size applications introduce concurrency constructs more intensively and more quickly than medium-size applications and large-size applications; and (5) an increasing use of standard concurrency constructs does not result in a substantially decreasing use of unstandardized concurrency constructs. Based on these findings, we make actionable suggestions for language designers, developers, and novices to assist them in designing and using C++ concurrency constructs.

The interactions between anti-patterns are claimed to affect maintenance. However, little work has been conducted to examine how anti-patterns interact. In this paper, we aim to investigate which pairs of anti-patterns tend to be coupled, i.e., interact with each other. We employ Fisher's exact test and Wilcoxon rank-sum test to identify coupled anti-patterns in the same class and coupled classes. Analyzing the relationships amongst 10 kinds of anti-patterns in five open-source projects, our results show that 1) several kinds of anti-patterns tend to be coupled, but some are conflicting, 2) the effect of anti-patterns on their dependent or co-changed ones are significant but small, 3) in ArgoUML, Xalan and Xerces-J, the classes infected with dependent anti-patterns are mostly (69.9% ~ 100%) modified in maintenance activities. Our findings offer empirical evidences for the existence of anti-pattern interactions, which provides valuable implications for practitioners and researchers.

The dynamic features of programming languages are useful constructs that bring developers convenience and flexibility, but they are also perceived to lead to difficulties in software maintenance.Figuring out whether the use of dynamic features affects maintenance is significant for both researchers and practitioners, yet little work has been done to investigate it.In this paper, we conduct an empirical study to explore whether program source code files using dynamic features are more change-prone and whether particular categories of dynamic features are more correlated to change-proneness than others.To this end, we statically analyze historical data from 4 to 7 years of the development of seven open-source systems.We employ Fisher and Mann-Whitney hypothetical test methods, along with logistic regression model to solve three research questions.The results show that: (1) files with dynamic features are more change-prone, (2) files with a higher number of dynamic features are more change-prone, and (3) Introspection is shown to be more correlated to change-proneness than the other three categories in most systems.This innovative work can give some inspirations and references to researchers who are always focusing their eyes on how and why the dynamic features are used.For practitioners, we suggest them to be wary of files with dynamic features because they are more likely to be the subject of their maintenance effort.

C++ libraries provide an abundance of reusable components for writing high-quality programs and are thus widely adopted by software developers.However, to date there is little work investigating how these libraries are actually used in real software.In this paper, we perform an empirical study to investigate the adoption of C++ standard libraries in open-source applications, with the goal to provide actionable information for developers to help them employ libraries more efficiently.To this end, we analyze 379 historical revisions of 30 applications, containing 149 million lines of C++ code, to conduct the experiment.The experimental results show that: (1) three standard libraries (i.e.Containers Library, Utilities Library, and Strings Library) are significantly more often used than other libraries; (2) the new libraries of C++11 (i.e.Regular Expressions Library, Atomic Operations Library, and Thread Support Library) are significantly less often used than the formerlyestablished libraries; (3) the deprecated library constructs (i.e. auto pointers, function objects, and array I/O operations) are not used at a declining frequency; and (4) applications with a larger size do not adopt libraries more frequently.Based on these results, we propose four suggestions, which could help developers learn and use C++ libraries in an efficient way.

There has been a long debate on which programming language can help write better object-oriented programs.However, to date little response is given to this issue with empirical evidence.In this paper, we perform a comparative study on C++, C#, and Java programs by using object-oriented metrics, which comprise measures for class size, complexity, coupling, cohesion, inheritance, encapsulation, polymorphism, and reusability.Our experiment is conducted on 78 tasks in Rosetta Code, a code repository providing solutions to the same programming tasks in different languages.The experimental results show that: (1) C++ classes are significantly larger than C# and Java classes in size, but their complexity does not differ significantly; (2) C# classes are significantly more likely to be coupled than C++ and Java classes through inter-class method invocations instead of direct data access; (3) C# and Java classes tend to be more cohesive than C++ classes; (4) C# and Java significantly outperform C++ in building deep inheritance trees; and (5) programs written in C++, C#, and Java do not show a significant difference in class encapsulation, polymorphism, and reusability.These findings could help practitioners choose suitable languages to develop object-oriented systems. 1

Background. Slice-based cohesion metrics leverage program slices with respect to the output variables of a module to quantify the strength of functional relatedness of the elements within the module. Although slice-based cohesion metrics have been proposed for many years, few empirical studies have been conducted to examine their actual usefulness in predicting fault-proneness. Objective. We aim to provide an in-depth understanding of the ability of slice-based cohesion metrics in effort-aware post-release fault-proneness prediction, i.e. their effectiveness in helping practitioners find post-release faults when taking into account the effort needed to test or inspect the code. Method. We use the most commonly used code and process metrics, including size, structural complexity, Halstead's software science, and code churn metrics, as the baseline metrics. First, we employ principal component analysis to analyze the relationships between slice-based cohesion metrics and the baseline metrics. Then, we use univariate prediction models to investigate the correlations between slice-based cohesion metrics and post-release fault-proneness. Finally, we build multivariate prediction models to examine the effectiveness of slice-based cohesion metrics in effort-aware post-release fault-proneness prediction when used alone or used together with the baseline code and process metrics. Results. Based on open-source software systems, our results show that: 1) slice-based cohesion metrics are not redundant with respect to the baseline code and process metrics; 2) most slice-based cohesion metrics are significantly negatively related to post-release fault-proneness; 3) slice-based cohesion metrics in general do not outperform the baseline metrics when predicting post-release fault-proneness; and 4) when used with the baseline metrics together, however, slice-based cohesion metrics can produce a statistically significant and practically important improvement of the effectiveness in effort-aware post-release fault-proneness prediction. Conclusion. Slice-based cohesion metrics are complementary to the most commonly used code and process metrics and are of practical value in the context of effort-aware post-release fault-proneness prediction.

Python is widely used for web programming and GUI development. Due to the dynamic features of Python, Python programs may contain various unlimited errors. Dynamic slicing extracts those statements from a program which affect the variables in a slicing criterion with a particular input. Dynamic slicing of Python programs is essential for program debugging and fault location. In this paper, we propose an approach of dynamic slicing for Python programs which combines static analysis and dynamic tracing of the Python byte code. It precisely handles the dynamic features of Python, such as dynamic typing of variables, heavy usage of first-class objects, and dynamic modifications of classes and instances. Finally, we evaluate our approach on several Python programs. Experimental results show that the whole dynamic slicing for each subject program spends at most about 13 seconds on the average and costs at most 7.58 mb memory space overhead. Furthermore, the average slice ratio of Python source code ranges from 9.26% to 59.42%. According to it, our dynamic slicing approach can be effectively and efficiently performed. To the best of our knowledge, it is the first one of dynamic slicing for Python programs.

Background. The extent of the potentially confounding effect of class size in the fault prediction context is not clear, nor is the method to remove the potentially confounding effect, or the influence of this removal on the performance of fault-proneness prediction models.Objective. We aim to provide an in-depth understanding of the effect of class size on the true associations between object-oriented metrics and fault-proneness.Method. We first employ statistical methods to examine the extent of the potentially confounding effect of class size in the fault prediction context. After that, we propose a linear regression-based method to remove the potentially confounding effect. Finally, we empirically investigate whether this removal could improve the prediction performance of fault-proneness prediction models.Results. Based on open-source software systems, we found: (a) the confounding effect of class size on the associations between object-oriented metrics and fault-proneness in general exists; (b) the proposed linear regression-based method can effectively remove the confounding effect; and (c) after removing the confounding effect, the prediction performance of fault prediction models with respect to both ranking and classification can in general be significantly improved.Conclusion. We should remove the confounding effect of class size when building fault prediction models.

With the popularity of social network and the increasing number of Web Services, making individual service recommendation has been a hot research spot nowadays. In this paper, we present a service recommendation algorithm named as URPC-Rec (User Relationships & Preferences Clustering and Recommendation), which first clusters users based on their history behaviors such as the services they ever invoked, and then makes personalized recommendations for users considering both the clustering results and user basic information and relationships, such as gender, age, occupation, preference tags, etc. The case study indicates that URPC-Rec can effectively reduce the dimensionality of sparse matrix, and partially solve the cold-start problem of recommendation systems. The comprehensive experiment shows that URPC-Rec algorithm with user relationships and references has better recommending result than the one without user information and the collaborative filtering approach.

Combinatorial testing has been widely used in practice. People usually assume all test cases in combinatorial test suite will run completely. However, in many scenarios where combinatorial testing is needed, for example the regression testing, the entire combinatorial test suite is not run completely as a result of test resource constraints. To improve the efficiency of testing, combinatorial test case prioritization technique is required. For the scenario of regression testing, this paper proposes a new cost-cognizant combinatorial test case prioritization technique, which takes both combination weights and test costs into account. Here we propose a series of metrics with physical meaning, which assess the combinatorial coverage efficiency of test suite, to guide the prioritization of combinatorial test cases. And two heuristic test case prioritization algorithms, which are based on total and additional techniques respectively, are utilized in our technique. Simulation experimental results illustrate some properties and advantages of proposed technique.

Python is a powerful dynamically typed programming language. Dynamic typing brings great flexibility for programming. However, lack of static type checking, it is hard to detect some bugs before run time. We present a constraint framework based on Python’s structural equivalence type system. The framework does not introduce any new language features, thus without losing benefits of Python’s dynamic typing. Constraints are extracted from source code via static analysis and are used to check bugs, such as passing wrong parameters to function. A case study shows how to use the framework to check validity of function calls.