Large Language Models (LLMs) have gained significant traction in software engineering for automating tasks such as unit test generation. Most existing studies prioritize code coverage as the primary metric for enhancing test suite effectiveness. However, prior research has shown that although code coverage can reach approximately 80%, the mutation score, which generally exhibits a stronger correlation with defect detection effectiveness, attains only about 35%. This gap highlights the need to enhance test suite effectiveness guided by mutation score rather than code coverage. Recent studies, including MuTAP and Mu tGe n, explored the use of survived mutants to enhance test suite effectiveness. However, their evaluations were limited to simple standalone methods that rely on built-in functions and standard libraries. Non-standalone methods, which depend on other classes and involve complex user-defined types, are more intricate and commonly found in real-world projects. The limited contextual information and basic repair mechanisms in their prompt designs make it unclear whether their performance can generalize to non-standalone methods. Moreover, the two studies rely on existing language-specific, rule-based mutation techniques, which require specific configurations and incur additional costs when adapting to other programming languages. To bridge this gap, we propose a novel, fully automatic LLM-based approach to enhance test suite effec-tiveness, guided by survived mutants. The approach augments initial test suites by integrating mutation testing with test case generation. It takes focal method information as input and generates test cases targeting survived mutants identified from applying the initial test suites. Our approach incorporates multiple prompt techniques, rich contextual information, and an advanced repair mechanism to effectively generate test cases for non-standalone methods. The evaluation covers 1,035 focal methods, categorized as standalone or non-standalone. On average, the mutation score increases by 16.04% for standalone methods and 8.11% for non-standalone methods. We validate the practical impact of augmented test suites in LLM-based code generation. After test suite augmentation, pass@1 decreased by 0.3152 and 0.1772 on average for standalone and non-standalone methods, respectively, indicating the effectiveness of our approach in reducing false positives caused by insufficient test cases in code generation evaluation.

Background . Code coverage and mutation score serve as pivotal test effectiveness metrics used to assess a test suite’s ability to uncover actual defects. However, prior research has produced inconsistent or even conflicting findings regarding their correlation with defect detection capability, particularly concerning the impact of test suite size. Problem. The extent of the potentially confounding effect of test suite size in test effectiveness evaluation context is not clear, nor is the method to remove the potentially confounding effect, or the influence of this removal on the performance of test suite optimization. Objective . Our goal is to deeply understand how test suite size affects the true relationship between test effectiveness metrics and a test suite’s ability to detect actual defects. Method. We first employ statistical methods to examine the extent of the potentially confounding effect of test suite size in the context of test effectiveness evaluation. After that, we propose a linear regression-based method to remove the potentially confounding effect of test suite size. Finally, we empirically explore the impact of this removal method on test suite optimization. Result. Our experimental results, based on the Defects4J defect dataset, uncovers that: (1) the confounding effect of test suite size on the associations between test effectiveness metrics and defect detection capability in general exists; (2) the proposed linear regression-based method can effectively remove the confounding effect; and (3) after removing the confounding effect, mutation score demonstrates superior effectiveness in predicting test suite effectiveness, while statement coverage is the least effective metric. Furthermore, both coverage-based and mutation-based test suite reduction exhibit enhanced cost-effectiveness in defect detection, and there is a marginal improvement in the speed of defect detection for coverage-based test case prioritization. Conclusion . When using test effectiveness metrics to assess test suite effectiveness, it is crucial to eliminate the influence of test suite size.

Machine Learning (ML) software employs statistical algorithms to perform high-stake tasks in our daily lives, whose results are usually discriminatory due to protected features (e.g., gender), i.e., one part (called privileged, e.g., male) may be more likely to obtain beneficial decisions than the other part (called unprivileged, e.g., female). In alleviating the unfairness, developers have obtained widely held beliefs about the tradeoff between performance and fairness for ML software. Surprisingly, recent research on feature engineering suggests that enlarging the feature set is the perfect way to kill two birds with one stone, i.e., achieving both higher performance and fairness. However, the experiments used in the prior study did not remove the effect of protected features, which have been suggested to be excluded in both industrial applications and academic studies. As a result, the study did not fully explore the tradeoff between performance and fairness. In this article, we first conduct an empirical study to replicate this prior study after excluding the protected features and observe that there is still a tradeoff between performance and fairness with enlarging the features, i.e., more features are not perfect, which would lead to higher performance and lower fairness. Due to more features causing more collection and pre-processing budgets, we aim to search for an effective alternative. Inspired by the “less is more” principle, we propose a novel feature ranking method, Hybrid-importance and Early-validation based Feature Ranking (HEFR) , to find an efficient subset to replace the full feature set with comparable performance and fairness. Our method, HEFR, employs hybrid feature importances to combine performance and fairness and conducts early validation to check the effectiveness of hybrid importances. We conduct experiments on seven datasets and three classifiers to evaluate our method with five baselines. The results have shown that (a) HEFR is efficient for ML software feature engineering: applying HEFR to choose about 10% of features would construct ML software with better or comparable performance and fairness, and (b) HEFR is actionable with small dataset sizes: applying HEFR with only 10% data size would still help choose the proper feature subset.

The rapid evolution of machine learning model training has outpaced the development of corresponding measurement and testing tools, leading to two significant challenges. Firstly, developers of deep learning frameworks struggle to identify operators that fail to meet precision criteria, as these issues may only manifest in a few data points. Secondly, model trainers lack effective methods to estimate precision loss caused by operators during training. To address these issues, we introduce a Pythonic framework inspired by common network layer definitions in deep learning. Our framework includes two new layers: the Fuzz Layer and the Check Layer, designed to aid in measurement and testing. The Fuzz Layer introduces minor perturbations to tensor inputs for any deterministic layer under testing (LUT). The Check Layer then measures precision by analyzing the differences before and after the perturbation. This approach estimates a lower bound of the maximal relative error and alerts developers or trainers of potential bugs if the difference exceeds a pre-defined tolerance. Additionally, Check Layers can be used independently to conduct precision tests for specific layers, ensuring the precision of operators during runtime. Despite the additional memory and time requirements, this runtime testing ensures proper training of the original model. We demonstrate the utility of our framework, FuMi, through two experiments. First, we tested 21 torch operators across nine popular machine learning models using PyTorch for various tasks, finding that the conv2d and linear operators often fail to meet precision requirements. Second, to showcase the generalizability of our framework, we tested the ATTENTION operator. By comparing different implementations of state-of-the-art ATTENTION operators, we found that the maximum relative error of the ATTENTION operator is not less than 1%, which is 13 times larger than that measured by Predoo (a unit test tool). This framework provides a robust solution for identifying precision issues in deep learning models, ensuring reliable and accurate model training.

Compilers are prone to bugs that can have severe consequences for downstream applications. Accurately identifying and localizing compiler faults poses unique challenges due to the inherent complexity and large scale of modern compiler infrastructures. Existing studies have proposed various techniques to construct passing and failing executions by generating witness test programs from bug-inducing test cases or by producing adversarial compilation configurations for the same test program. These executions are then leveraged to apply spectrum-based fault localization (SBFL) techniques for isolating compiler faults, yielding promising results. Recently, Yang et al. revisited SBFL-based techniques and showed that a simple yet widely adopted debugging practice—treating files modified in bug-inducing commits (BICs) as potential fault candidates—can surprisingly outperform SBFL-based techniques on the most critical localization metrics. Moreover, they further demonstrated that BIC-based and SBFL-based techniques are highly complementary, as they tend to localize different subsets of compiler faults. Consequently, effectively integrating these two sources of information to improve compiler fault localization remains an open and largely unexplored challenge. To address this problem, we propose DUALTRACK, a hybrid approach that integrates dynamic execution information from SBFL with historical information derived from BICs. DUALTRACKemploys a two-layer framework that first prioritizes files modified in bug-inducing commits and then refines their rankings using suspiciousness scores computed by SBFL formulas. An evaluation on 120 real-world compiler bugs from GCC and LLVM shows that DUALTRACK successfully identifies 52% of faulty files at the Top-1 rank, demonstrating a substantial improvement over existing state-of-the-art compiler fault localization techniques.

Ensuring the correctness of debugger toolchains is of paramount importance, as they play a vital role in understanding and resolving programming errors during software development. Bugs hidden within these toolchains can significantly mislead developers. Unfortunately, comprehensive testing of debugger toolchains is lacking due to the absence of effective test oracles. Existing studies on debugger toolchain validation have primarily focused on validating the debug information within optimized executables by comparing the traces between debugging optimized and unoptimized executables (i.e., different executables) in the debugger, under the assumption that the traces obtained from debugging unoptimized executables serve as a reliable oracle. However, these techniques suffer from inherent limitations, as compiler optimizations can drastically alter source code elements, variable representations, and instruction order, rendering the traces obtained from debugging different executables incomparable and failing to uncover bugs in debugger toolchains when debugging unoptimized executables. To address these limitations, we propose a novel concept called Cross-Level Debugging (CLD) for validating the debugger toolchain. CLD compares the traces obtained from debugging the same executable using source-level and instruction-level strategies within the same debugger. The core insight of CLD is that the execution traces obtained from different debugging levels for the same executable should adhere to specific relationships, regardless of whether the executable is generated with or without optimization. We formulate three key relations in CLD: reachability preservation of program locations, order preservation for reachable program locations, and value consistency at program locations, which apply to traces at different debugging levels. We implement Devil, a practical framework that employs these relations for debugger toolchain validation. We evaluate the effectiveness of Devil using two widely used production debugger toolchains, GDB and LLDB. Ultimately, Devil successfully identified 27 new bug reports, of which 18 have been confirmed and 12 have been fixed by developers.

Ensuring the reliability of the Rust compiler is of paramount importance, given increasing adoption of Rust for critical systems development, due to its emphasis on memory and thread safety. However, generating valid test programs for the Rust compiler poses significant challenges, given Rust's complex syntax and strict requirements. With the growing popularity of large language models (LLMs), much research in software testing has explored using LLMs to generate test cases. Still, directly using LLMs to generate Rust programs often results in a large number of invalid test cases. Existing studies have indicated that test cases triggering historical compiler bugs can assist in software testing. Our investigation into Rust compiler bug issues supports this observation. Inspired by existing work and our empirical research, we introduce a bracket-based masking and filling strategy called clozeMask. The clozeMask strategy involves extracting test code from historical issue reports, identifying and masking code snippets with specific structures, and using an LLM to fill in the masked portions for synthesizing new test programs. This approach harnesses the generative capabilities of LLMs while retaining the ability to trigger Rust compiler bugs. It enables comprehensive testing of the compiler's behavior, particularly exploring edge cases. We implemented our approach as a prototype ClozeMaster. ClozeMaster has identified 27 confirmed bugs for rustc and mrustc, of which 10 have been fixed by developers. Furthermore, our experimental results indicate that ClozeMaster outperforms existing fuzzers in terms of code coverage and effectiveness.

Code-line-Ievel defect prediction (CLDP) is an effective technique to incorporate comprehensive measures for buggy line identification to optimize efforts in Software Quality Assurance activities. Most CLDP methods either consider the textual information of the code or rely merely on file-level label information, which have not fully leveraged the essential information in the CLDP context, with historical code-line-level labels being incredibly overlooked in their application. Due to the vast number of code lines and the sparsity of the tokens they contain, leveraging historical code-line-level label information remains a significant challenge. To address this issue, we propose a novel CLDP method, Spectrum infOrmation and caUsality aNalysis based coDe-line-level defect prediction (SOUND). SOUND incorporates two key ideas: (a) it introduces a spectrum information perspective, utilizing labels from historical defective lines to quantify the contribution of tokens to line-level defects, and (b) it applies causal analysis to obtain a more systematic and comprehensive understanding of the causal relationships between tokens and defects. After conducting a comprehensive study involving 142 releases across 19 software projects, the experimental results demonstrate that our method significantly outperforms existing state-of-the-art (SOTA) CLDP baseline methods in terms of its ability to rank defective lines under three indicators, IFA, Recall@Top20%LOC, and Effort@Top20%Recall. Notably, in terms of IFA, our method achieves a score of 0 in most cases, indicating that the first line in the ranking list generated by our method is actually defective, significantly enhancing its practicality.

Abstract Researchers have recently achieved significant advances in deep learning techniques, which in turn has substantially advanced other research disciplines, such as natural language processing, image processing, speech recognition, and software engineering. Various deep learning techniques have been successfully employed to facilitate software engineering tasks, including code generation, software refactoring, and fault localization. Many studies have also been presented in top conferences and journals, demonstrating the applications of deep learning techniques in resolving various software engineering tasks. However, although several surveys have provided overall pictures of the application of deep learning techniques in software engineering, they focus more on learning techniques, that is, what kind of deep learning techniques are employed and how deep models are trained or fine-tuned for software engineering tasks. We still lack surveys explaining the advances of subareas in software engineering driven by deep learning techniques, as well as challenges and opportunities in each subarea. To this end, in this study, we present the first task-oriented survey on deep learning-based software engineering. It covers twelve major software engineering subareas significantly impacted by deep learning techniques. Such subareas spread out through the whole lifecycle of software development and maintenance, including requirements engineering, software development, testing, maintenance, and developer collaboration. As we believe that deep learning may provide an opportunity to revolutionize the whole discipline of software engineering, providing one survey covering as many subareas as possible in software engineering can help future research push forward the frontier of deep learning-based software engineering more systematically. For each of the selected subareas, we highlight the major advances achieved by applying deep learning techniques with pointers to the available datasets in such a subarea. We also discuss the challenges and opportunities concerning each of the surveyed software engineering subareas.

Satisfiability Modulo Theories (SMT) solvers are widely used for program analysis and other applications that require automated reasoning. Rewrite systems, as crucial integral components of SMT solvers, are responsible for simplifying and transforming formulas to optimize the solving process. The effectiveness of an SMT solver heavily depends on the robustness of its rewrite system, making its validation crucial. Despite ongoing advancements in SMT solver testing, rewrite system validation remains largely unexplored. Our empirical analysis reveals that developers invest significant effort in ensuring the correctness and reliability of rewrite systems. However, existing testing techniques do not adequately address this aspect. In this paper, we introduce Aries , a novel technique designed to validate SMT solver rewrite systems. First, Aries employs mimetic mutation , a targeted strategy that actively reshapes input formulas to provoke and diversify rewrite opportunities. By aligning mutated terms with known rewrite patterns, Aries can conduct a thorough exploration of the rewrite space in the following phase. Second, Aries utilizes deductive rewriting , leveraging generative equality saturation to effectively explore rewrite space and produce semantically equivalent mutants for the purpose of validation. We implemented Aries as a practical validation tool and evaluated it on leading SMT solvers, including Z3 and cvc5. Our experiments demonstrate that Aries effectively identifies bugs, with 27 new issues detected, of which 22 have been confirmed or fixed by developers. Most of these issues involve the rewrite systems, highlighting Aries ’s strength in exploring the rewrite space.

Deep learning (DL) models have proven to be highly successful and are now essential to our everyday routines. However, DL models, like traditional software, inevitably contain bugs that affect their performance in real-world scenarios. Effective software engineering techniques are necessary to ensure their dependability. In recent years, fault localization methods for DL models have gained significant attention as a valuable tool for improving the reliability of DL models. Owing to the data-driven programming paradigm, traditional fault localization techniques are challenging to apply directly to DL programs. Previous studies have shown that neuron errors within models can lead to abnormal behavior, and they fix the DL model errors from the perspective of neurons. Nonetheless, there remains a significant gap between the DL program statement and model errors. To tackle this problem, this paper proposes a novel fault localization method for DL models, named wei G hted s U sp I ciousness an D balanc E d agg R egation ( \(\mathsf{GUIDER}\) ) that revisits the idea and challenge of spectrum-based fault localization in the context of DL models. For pre-trained DL models, \(\mathsf{GUIDER}\) utilizes neuron coverage information and test case confidence to compute weighted neuron suspiciousness values and employs balanced aggregation methods to elevate these values from the neuron level to the layer level, which establishes a bridge between the DL model and the DL program, facilitating the developers’ debugging process. We evaluate \(\mathsf{GUIDER}\) using 161 real model bugs collected from StackOverflow and five state-of-the-art fault localization methods for DL models as baselines. The results indicate that (a) our method successfully localizes 67% of the model bugs by ranking the buggy layer to the first place (i.e., top- \(1\) ), significantly outperforming all five baselines, and (b) our method maintains an acceptable time overhead compared with all baseline methods.

The co-evolution of production and test code (PT co-evolution) has received increasing attention in recent years. However, we found that existing work did not comprehensively study various PT co-evolution scenarios, such as the qualification and persistence of their effects on software. Inspired by technical debt (TD), we refer to TD generated during the co-evolution between production and test code as PT co-evolution technical debt (PTCoTD). To better understand PT co-evolution, we first conducted an exploratory study on its characteristics on 15 open-source projects, finding unbalanced PT co-evolution prevalent and summarizing five potential PT flaws. Then we proposed an approach to identify and quantify PTCoTDs of these flaw patterns, considering evolutionary and structural relationships. We also built prediction models to describe cost trajectories and rank all PTCoTDs to prioritize expensive ones. The evaluation on the 15 projects shows that our approach can identify PTCoTDs that deserve attention. The identified PTCoTDs account for about half of the project's total maintenance costs, and the cost proportion of the expensive Top-5 is 1.8x more than the file proportion they contain. Almost all covered maintenance costs persist as PTCoTD in the future, with an average increase of 6.8% between the last two releases. Our approach also accurately predicts the costs of PTCoTD with an average prediction deviation of only 8.3%. Our study provides valuable insights into PT co-evolution scenarios and their effects, which can guide practices and inspire future work on software testing and maintenance.

Android app developers currently mainly attempt to merge all functions into one app to fit different types of devices. However, this “one-size-fits-all” strategy can introduce various problems to both developers and end-users, such as slower download speed, and a larger attack surface. To resolve this issue, Google promotes the App Bundle framework and requires all new apps must adopt this framework after August 2021. The app bundle framework allows developers to organize their apps in modules. As a new framework, building an app bundle can be time-consuming and error-prone for developers. To fill this gap, in this paper, we discuss how developers build app bundles in practice. By investing in over 200,000 apps from Google Play, we find that 30% of apps have already adopted app bundles. The adoption ratio of large-size apps is even higher than 90%. We also find hands-on programming practices for building feature modules and dynamic assets in app bundles. This study also finds 12 common design practices, which assist developers in building app bundles.

In the rapidly advancing field of software development, the demand for practical code translation tools has surged, driven by the need for interoperability across different programming environments. Existing learning-based approaches often need help with low-resource programming languages that lack sufficient parallel code corpora for training. To address these limitations, we propose a novel training framework that begins with monolingual seed corpora, generating parallel datasets via back-translation and incorporating compiler feedback to optimize the translation model.As a case study, we apply our method to train a code translation model for a new-born low-resource programming language, Cangjie. We also construct a parallel test dataset forJava-to-Cangjietranslation and test cases to evaluate the effectiveness of our approach. Experimental results demonstrate that compiler feedback greatly enhances syntactical correctness, semantic accuracy, and test pass rates of the translatedCangjiecode. These findings highlight the potential of our method to support code translation in low-resource settings, expanding the capabilities of learning-based models for programming languages with limited data availability.

Compilation optimization bugs are prevalent and can significantly affect the correctness of software products, posing serious challenges to software development. Identifying and localizing these bugs are critical tasks for compiler developers. However, the intricate nature and extensive scale of modern compilers make it difficult to pinpointing the root causes of such bugs. Previous research has introduced innovative techniques that generatewitness test programs–tests that pass–by mutating bug-triggering test cases, highlighting the importance of this problem and demonstrating the effectiveness of such approaches. Nevertheless, existing techniques based on witness test programs generation suffer from inherent limitations. Specifically, they do not guarantee the successful creation of witness test programs via mutation and are often time-consuming, typically requiring extensive iterations to produce a valid witness test program. In this study, we present Odfl, a simple yet effective approach for automatically isolating compiler optimization faults by introducing the concept ofdifferentiated compilation configurations. The core insight behind Odfl is that modifying compilation settings such as disabling fine-grained compilation flags in GCC or reducing the number of fine-grained compilation passes in LLVM, can suppress the manifestation of compiler bugs triggered by the same test program. Through adjusting these settings, Odfl creates differentiated compilation configuration that produce multiple compiler executions with distinct pass/-fail outcomes. We utilize these differentiated configurations to collect both passing and failing compiler coverage, and then applySpectrum-Based Fault Localization (SBFL)techniques to rank compiler source files based on their suspiciousness. Our evaluation of 60 GCC and 50 LLVM compiler bugs demonstrates that Odfl substantially outperforms state-of-the-art compiler fault localization techniques in terms of both effectiveness and efficiency. Notably, Odfl achieves over 90% improvement in accurately ranking the top-1 faulty source files compared to three existing techniques–DiWi, RecBi, and LLM4CBI–and reduces fault localization time by more than 99% on average.

Mutation testing aims to simulate real-world defects, but existing tools often struggle to replicate method invocation defects accurately. To address this, we propose MIN (Method INvocation mutator), which uses a mapping strategy to pair method names with corresponding values, ensuring that methods share argument and return types. This method enhances the feasibility and realism of mutants by considering factors such as library methods, access control, inheritance, and static methods. Experimental results show that integrating MIN into Major (a popular mutation tool) improves semantic similarity to real defects by 11%, increases mutant set diversity to 97.5%, and reduces undetected faults by 38.5%. Furthermore, MIN’s performance rivals that of state-of-the-art machine learning-based mutators like CodeBERT, with a 10x speed advantage over CodeBERT and 4x over DeepMutation in generating compilable mutants. These findings demonstrate that MIN can significantly enhance defect simulation and improve the efficiency of mutation testing.

In the wake of developments in the field of Natural Language Processing, Question Answering (QA) software has penetrated our daily lives. Due to the data-driven programming paradigm, QA software inevitably contains bugs, i.e., misbehaving in real-world applications. Current testing techniques for testing QA software include two folds, reference-based testing and metamorphic testing.

Code comment generation typically refers to the process of generating concise natural language descriptions for a piece of code, which facilitates program comprehension activities. Inline code comments, as a part of code comments, are also crucial for program comprehension. Recently, the emergence of large language models (LLMs) has significantly boosted the performance of natural language processing tasks. This naturally inspires us to explore the performance of the LLMs in the task of inline code comment generation. To this end, we evaluate open-source LLMs on a large-scale dataset and compare the results with the current state-of-the-art methods. Specifically, we explore the model performance in the following scenarios based on the widely used evaluation metrics (i.e. BLEU, Meteor, and ROUGE-L): (1) generation with simple instruction; (2) few-shot-guided generation with random examples selected from the database; (3) few-shot-guided generation with similar examples selected from the database; and (4) adopt the re-ranking strategy for the output of LLMs. Our findings reveal that: (1) under the simple instruction scenario, LLMs could not fully show the potential in the task of inline comment generation compared to the state-of-the-art models; (2) random few-shot leads to a slight improvement; (3) similar few-shot and re-ranking strategy could significantly enhance the performance of LLMs; and (4) for inline comment and code snippet pairs with different intents, why category achieves the best performance and what category achieves relatively poorer performance. That remains consistent across all four scenarios. Our findings shed light on future research directions for using LLMs in inline comment generation tasks.

As a fundamental component of software documentation, code comments could help developers comprehend and maintain programs. Several datasets of method header comments have been proposed in previous studies for machine learning-based code comment generation. As part of code comments, inline code comments are also crucial for code understanding activities. However, unlike method header comments written in a standard format and describing the whole method code, inline comments are often written in arbitrary formats by developers due to timelines pressures and different aspects of code snippets in the method are described. Currently, there is no large-scale dataset used for inline comments generation considering these. Hence, this naturally inspires us to explore whether we can construct a dataset to foster machine learning research that not only performs fine-grained noise-cleaning but conducts a taxonomy of inline comments. To this end, we first collect inline comments and code snippets from 8000 Java projects on GitHub. Then, we conduct a manual review to obtain heuristic rules, which could be used to clean the data noise in a fine-grained manner. As a result, we construct a large-scale benchmark dataset named ICG with 5,740,770 pairs of inline comments and code snippets. We then build a comprehensive taxonomy and conduct a statistical and manual analysis to explore the performances of different categories of inline comments, such as helpfulness in code understanding. After that, we provide and compare several baseline models to automatically generate inline comments, such as CodeBERT, to enhance the usability of the benchmark for researchers. The availability of our benchmark and baselines can help develop and validate new inline comment generation methods, which would also further facilitate code understanding activities.

Abstract Gradual typing integrates static and dynamic typing by introducing a dynamic type and a consistency relation. A problem of gradual type systems is that dynamic types can easily hide erroneous data flows since consistency relations are not transitive. Therefore, a more rigorous static check is required to reveal these hidden data flows statically. However, in order to preserve the expressiveness of gradually typed languages, static checks for gradually typed languages cannot simply reject programs with potentially erroneous data flows. By contrast, a more reasonable request is to show how these data flows can affect the execution of the program. In this paper, we propose and formalize Static Blame , a framework that can reveal hidden data flows for gradually typed programs and establish the correspondence between static-time data flows and runtime behavior. With this correspondence, we build a classification of potential errors detected from hidden data flows and formally characterize the possible impact of potential errors in each category on program execution, without simply rejecting the whole program. We implemented Static Blame on Grift, an academic gradually typed language, and evaluated the effectiveness of Static Blame by mutation analysis to verify our theoretical results. Our findings revealed that Static Blame exhibits a notable level of precision and recall in detecting type-related bugs. Furthermore, we conducted a manual classification to elucidate the reasons behind instances of failure. We also evaluated the performance of Static Blame, showing a quadratic growth in run time as program size increases.

Abstract Deep learning is a powerful technique to boost application performance in various fields, including face recognition, image classification, natural language understanding, and recommendation system. With the rapid increase in the computing power of mobile devices, developers can embed deep learning models into their apps for building more competitive products with more accurate and faster responses. Although there are several works of adversarial attacks against deep learning models in apps, they all need information about the models' internals (i.e., structures and weights) or need to modify the models. In this paper, we propose an effective black‐box approach by training substitute models to spoof the deep learning systems inside the apps. We evaluate our approach on 10 real‐world deep‐learning apps from Google Play to perform black‐box adversarial attacks. Through the study, we find three factors that can affect the performance of attacks. Our approach can reach a relatively high attack success rate of 66.60% on average. Compared with other adversarial attacks on mobile deep learning models, in terms of the average attack success rates, our approach outperforms its counterparts by 27.63%.

Abstract Background Software defect prediction (SDP) is a topic actively researched in the software engineering community. Within‐project defect prediction (WPDP) involves using labeled modules from previous versions of the same project to train classifiers. Over time, many defect prediction models have been evaluated under the WPDP scenario. Problem Data duplication poses a significant challenge in current WPDP evaluation procedures. Unchanged modules, characterized by identical executable source code, are frequently present in both target and source versions during experimentation. However, it is still unclear how and to what extent the presence of unchanged modules affects the performance assessment of WPDP models and the comparison of multiple WPDP models. Method In this paper, we provide a method to detect and remove unchanged modules from defect datasets and unveil the impact of data duplication in WPDP on model evaluation. Results The experiments conducted on 481 target versions from 62 projects provide evidence that data duplication significantly affects the reported performance values of individual learners in WPDP. However, when ranking multiple WPDP models based on prediction performance, the impact of removing unchanged instances is not substantial. Nevertheless, it is important to note that removing unchanged instances does have a slight influence on the selection of models with better generalization. Conclusion We recommend that future WPDP studies take into consideration the removal of unchanged modules from target versions when evaluating the performance of their models. This practice will enhance the reliability and validity of the results obtained in WPDP research, leading to improved understanding and advancements in defect prediction models.

Abstract Because defects in software modules (e.g., classes) might lead to product failure and financial loss, software defect prediction enables us to better understand and control software quality. Software development is a dynamic evolutionary process that may result in data distributions (e.g., defect characteristics) varying from version to version. In this case, effective cross‐version defect prediction (CVDP) is not easy to achieve. In this paper, we aim to investigate whether the defect prediction method of the threshold‐based active learning (TAL) can tackle the problem of the different data distribution between successive versions. Our TAL method includes two stages. At the active learning stage, a committee of investigated metrics is constructed to vote on the unlabeled modules of the current version. We pick up the unlabeled module with the median of voting scores to domain experts. The domain experts test and label the selected unlabeled module. Then, we merge the selected labeled module and the remaining modules with pseudo‐labels from the current version into the labeled modules of the prior version to form enhanced training data. Based on the training data, we derive the metric thresholds used for the next iteration. At the defect prediction stage, the iterations stop when a predefined threshold is reached. Finally, we use the cutoff threshold of voting scores, that is, 50%, to predict the defect‐prone of the remaining unlabeled modules. We evaluate the TAL method on 31 versions of 10 projects with three prevalent performance indicators. The results show that TAL outperforms the baseline methods, including three variations methods, two common supervised methods, and the state‐of‐the‐art method Hybrid Active Learning and Kernel PCA (HALKP). The results indicate that TAL can effectively address the different data distribution between successive versions. Furthermore, to keep the cost of extensive testing low in practice, selecting 5% of candidate modules from the current version is sufficient for TAL to achieve a good performance of defect prediction.

Summary Natural language inference (NLI) is a task to infer the relationship between the premise and hypothesis sentences, whose models have essential applications in the many natural language processing (NLP) fields, for example, machine reading comprehension and recognizing textual entailment. Due to the data‐driven programming paradigm, bugs inevitably occur in NLI models during the application process, which calls for novel automatic testing techniques to deal with NLI testing challenges. The main difficulty in achieving automatic testing for NLI models is the oracle problem; that is, it may be too expensive to label NLI model inputs manually and hence be too challenging to verify the correctness of model outputs. To tackle the oracle problem, this study proposes a novel automatic testing method hybrid mutation driven testing (HMT), which extends the mutation idea applied in other NLP domains successfully. Specifically, as there are two sets of sentences, that is, premise and hypothesis, to be mutated, we propose four mutation operators to achieve the hybrid mutation strategy, which mutate the premise and the hypothesis sentences jointly or individually. We assume that the mutation would not affect the outputs; that is, if the original and mutated outputs are inconsistent, inconsistency bugs could be detected without knowing the true labels. To evaluate our method HMT, we conduct experiments on two widely used datasets with two advanced models and generate more than 520,000 mutations by applying our mutation operators. Our experimental results show that (a) our method, HMT, can effectively generate mutated testing samples, (b) our method can effectively trigger the inconsistency bugs of the NLI models, and (c) all four mutation operators can independently trigger inconsistency bugs.

Abstract Background Line‐level software defect prediction (LL‐SDP) serves as a valuable tool for developers to detect defective lines with minimal human effort. Recently, GLANCE was proposed as a readily implementable baseline for assessing the efficacy of newly proposed LL‐SDP models. Problem While DeepLineDP, a cutting‐edge LL‐SDP model rooted in deep learning, has demonstrated state‐of‐the‐art performance, it has not yet been compared against GLANCE. Objective We aim to empirically compare DeepLineDP with GLANCE to obtain a comprehensive understanding of how deep learning contributes to solving the LL‐SDP challenge. Method We compare GLANCE against DeepLineDP to assess the extent to which DeepLineDP surpasses GLANCE in predicting defective files and identifying problematic lines. In order to obtain a reliable conclusion, we use the same dataset and performance metrics utilized by DeepLineDP. Result Our experimental findings indicate that DeepLineDP does not outperform GLANCE in LL‐SDP. This suggests that the application of deep learning, in this context, does not yield the anticipated significant improvements. Conclusion This finding underscores the need for further research in deep learning‐based LL‐SDP to attain the state‐of‐the‐art performance that remains elusive for less advanced techniques.

In recent years, dynamic languages such as Python have become popular due to their flexibility and productivity. The lack of static typing makes programs face the challenges of fixing type errors, early bug detection, and code understanding. To alleviate these issues, PEP 484 introduced optional type annotations for Python in 2014, but unfortunately, a large number of programs are still not annotated by developers. Annotation generation tools can utilize type inference techniques. However, several important aspects of type annotation generation are overlooked by existing works, such as in-depth effectiveness analysis, potential improvement exploration, and practicality evaluation. And it is unclear how far we have been and how far we can go. In this paper, we set out to comprehensively investigate the effectiveness of type inference tools for generating type annotations, applying three categories of state-of-the-art tools on a carefully-cleaned dataset. First, we use a comprehensive set of metrics and categories, finding that existing tools have different effectiveness and cannot achieve both high accuracy and high coverage. Then, we summarize six patterns to present the limitations in type annotation generation. Next, we implement a simple but effective tool to demonstrate that existing tools can be improved in practice. Finally, we conduct a controlled experiment showing that existing tools can reduce the time spent annotating types and determine more precise types, but cannot reduce subjective difficulty. Our findings point out the limitations and improvement directions in type annotation generation, which can inspire future work.

Background. Software testing is a critical activity for ensuring the quality and reliability of software systems. To evaluate the effectiveness of different test suites, researchers have developed a variety of metrics. Problem. However, comparing these metrics is challenging due to the lack of a standardized evaluation framework including comprehensive factors. As a result, researchers often focus on single factors (e.g., size), which finally leads to different or even contradictory conclusions. After comparing dozens of pieces of work in detail, we have found two main problems most troubling to our community: (1) researchers tend to oversimplify the description of the ground truth they use, and (2) data involving real defects is not suitable for analysis using traditional statistical indicators. Objective. We aim at scrutinizing the whole process of comparing test suites for our community. Method. To hit this aim, we propose a framework ASSENT (ev A luating te S t S uite E ffective N ess me T rics) to guide the follow-up research for evaluating a test suite effectiveness metric. ASSENT consists of three fundamental components: ground truth, benchmark test suites, and agreement indicator. Its functioning is as follows: first, users clarify the ground truth for determining the real order in effectiveness among test suites. Second, users generate a set of benchmark test suites and derive their ground truth order in effectiveness. Third, users use the metric to derive the order in effectiveness for the same test suites. Finally, users calculate the agreement indicator between the two orders derived by two metrics. Result. With ASSENT, we are able to compare the accuracy of different test suite effectiveness metrics. We apply ASSENT to evaluate representative test suite effectiveness metrics, including mutation score and code coverage metrics. Our results show that, based on the real faults, mutation score, and subsuming mutation score are the best metrics to quantify test suite effectiveness. Meanwhile, by using mutants instead of real faults, test effectiveness will be overestimated by more than 20% in values. Conclusion. We recommend that the standardized evaluation framework ASSENT should be used for evaluating and comparing test effectiveness metrics in the future work.

Large language models (LLMs) have demonstrated impressive capabilities across various natural language processing (NLP) tasks, such as machine translation, question answering, summarization, and so on. Additionally, LLMs are also highly valuable in supporting software engineering tasks, particularly in the field of code generation. Automatic code generation is a process of automatically generating source code or executable code based on given specifications or requirements, improving developer productivity. In this study, we perform a systematic empirical assessment to the quality of code generation usingChatGPT, a recent state-of-the-art product LLM. We leverage 728 algorithm problems in five languages (i.e., C, C++, Java, Python, and JavaScript) and 18 CWEs with 54 code scenarios for the code generation task. Our evaluation encompasses a comprehensive analysis of code snippets generated byChatGPT, focusing on three critical aspects: correctness, complexity, and security. We also specifically investigateChatGPT’s ability to engage in multi-round fixing process (i.e.,ChatGPT’s dialog ability, chatting between users andChatGPTfor fixing generated buggy code) of facilitating code generation. By delving into the generated code and examining the experimental results, this work provides valuable insights into the performance ofChatGPTin tackling code generation tasks over the three critical aspects. The experimental results demonstrate that (1)ChatGPTis better at generating functionally correct code for problems before 2021 in different languages than problems after 2021 with 48.14% advantage in Accepted rate on judgment platform, butChatGPT’s ability to directly fix erroneous code with multi-round fixing process to achieve correct functionality is relatively weak; (2) the distribution of cyclomatic and cognitive complexity levels for code snippets in different languages varies. Furthermore, the multi-round fixing process withChatGPTgenerally preserves or increases the complexity levels of code snippets; (3) in algorithm scenarios with languages of C, C++, and Jave, and CWE scenarios with languages of C and Python3, the code generated byChatGPThas relevant vulnerabilities. However, the multi-round fixing process for vulnerable code snippets demonstrates promising results, with more than 89% of vulnerabilities successfully addressed; and (4) code generation may be affected byChatGPT’s non-determinism factor, resulting in variations of code snippets in functional correctness, complexity, and security. Overall, our findings uncover potential issues and limitations that arise in theChatGPT-based code generation and lay the groundwork for improving AI and LLM-based code generation techniques.

In [1, p. 4967], a figure citation is incorrect and “Fig. 3(c)” should be “Fig. 1(c)” in the left column, the fourth line from the bottom. It is corrected below.

Unit testing is critical to the software development process, ensuring the correctness of basic programming units in a program (e.g., a method). Search-based software testing (SBST) is an automated approach to generating test cases. SBST generates test cases with genetic algorithms by specifying the coverage criterion (e.g., branch coverage). However, a good test suite must have different properties, which cannot be captured using an individual coverage criterion. Therefore, the state-of-the-art approach combines multiple criteria to generate test cases. Since combining multiple coverage criteria brings multiple objectives for optimization, it hurts the test suites’ coverage for certain criteria compared with using the single criterion. To cope with this problem, we propose a novel approach namedsmart selection. Based on the coverage correlations among criteria and the subsumption relationships among coverage goals, smart selection selects a subset of coverage goals to reduce the number of optimization objectives and avoid missing any properties of all criteria. We conduct experiments to evaluate smart selection on$400$Java classes with three state-of-the-art genetic algorithms under the$2$-minute budget. On average, smart selection outperforms combining all goals on$65.1\%$of the classes having significant differences between the two approaches. Secondly, we conduct experiments to verify our assumptions about coverage criteria relationships. Furthermore, we assess the coverage performance of smart selection under varying budgets of$5$,$8$, and$10$minutes and explore its effect on bug detection, confirming the advantage of smart selection over combining all goals.

Background: Bug dependencies refer to the link relationships between bugs and related issues, which are commonly observed in software evolution. It has been found that bugs with bug dependencies often take longer time to be resolved than other bugs without any dependencies. Despite the potential impact of bug dependencies on bug-fixing time, previous studies use traditional metrics without considering bug dependencies to build bug-fixing time prediction models. As a result, there is currently little empirical evidence to support the use of bug dependencies in improving prediction accuracy. Aims: We aim to conduct a comprehensive empirical study to investigate the value of considering bug dependencies for bug-fixing time prediction. Method: We define a set of bug dependency metrics based on bug dependencies. We first investigate the correlation between bug dependency metrics and bug-fixing time to investigate whether bugs with more complex dependencies are more time-consuming to be fixed. Next, we employ principal component analysis to study whether bug dependency metrics capture additional dimensions of a bug compared to traditional metrics. Finally, we build multivariate prediction models to explore whether considering bug dependencies can improve the effectiveness of bug-fixing time prediction. Results: The experimental results suggest that: (1) bugs with more complex dependencies require more time to be fixed; (2) bug dependency metrics are complementary to traditional metrics; (3) considering bug dependencies can improve the effectiveness of bug-fixing time prediction. Conclusions: These findings highlight the importance of considering bug dependencies in bug-fixing time prediction, and provide valuable insights into the potential impact of bug dependencies on software development processes.

SMT solvers check the satisfiability of logic formulas over first-order theories, which have been utilized in a rich number of critical applications, such as software verification, test case generation, and program synthesis. Bugs hidden in SMT solvers would severely mislead those applications and further cause severe consequences. Therefore, ensuring the reliability and robustness of SMT solvers is of critical importance. Although many approaches have been proposed to test SMT solvers, it is still a challenge to discover bugs effectively. To tackle such a challenge, we conduct an empirical study on the historical bug-triggering formulas in SMT solvers' bug tracking systems. We observe that the historical bug-triggering formulas contain valuable skeletons (i.e., core structures of formulas) as well as associated atomic formulas which can cast significant impacts on formulas' ability in triggering bugs. Therefore, we propose a novel approach that utilizes the skeletons extracted from the historical bug-triggering formulas and enumerates atomic formulas under the guidance of association rules derived from historical formulas. In this study, we realized our approach as a practical fuzzing tool HistFuzz and conducted extensive testing on the well-known SMT solvers Z3 and cvc5. To date, HistFuzz has found 111 confirmed new bugs for Z3 and cvc5, of which 108 have been fixed by the developers. More notably, out of the confirmed bugs, 23 are soundness bugs and invalid model bugs found in the solvers' default mode, which are essential for SMT solvers. In addition, our experiments also demonstrate that HistFuzz outperforms the state-of-the-art SMT solver fuzzers in terms of achieved code coverage and effectiveness.

The emergence of build automation tools with dependency management features has significantly impacted software development. However, in the configuration process, improper settings of some configuration items, such as the dependency scope setting, may cause severe problems in the development process. Improper setting of dependency scope may cause problems such as missing dependencies and redundant dependencies, and may even spread the problem to the downstream of the software ecosystem.

Machine translation systems have penetrated our daily lives, providing translation services from source language to target language to millions of users online daily. Word Sense Disambiguation (WSD) is one of the essential functional requirements of machine translation systems, which aims to determine the exact sense of polysemes in the given context. Commercial machine translation systems (e.g., Google Translate) have been shown to fail in identifying the proper sense and consequently cause translation errors. However, to our knowledge, no prior studies focus on testing such WSD bugs for machine translation systems.

Python has become increasingly popular and widely used in many fields. Dynamic features of Python provide much convenience for developers. However, they can also cause many type-related bugs undetected until runtime, which increases the cost of maintenance. Static type checking is essential to find bugs early, and the introduction of gradual typing and type annotations makes it easier to perform static type analysis. However, it remains to be investigated how well gradual typing improves real bug detection. Therefore, we conducted a comprehensive study on three widely used checkers: MyPy, PyRight, and PyType. We used a benchmark containing 10 popular Python projects with 40 real type-related bugs. First, we performed static type checking on the projects with and without type annotations to evaluate the effectiveness of finding real bugs. Second, we manually analyzed the missing bugs and investigated the reasons. The results show that the three tools can detect 29 of the 40 studied bugs after annotating, while only 14 bugs are detected before annotating. We also found that type annotations can substantially improve the ability of static type checkers to detect real bugs. A detailed analysis of bugs missed by the checkers shows that: (i) the accuracy of type analysis is challenged when it comes to programs with complicated dynamic features, such as dynamically changing object’s attributes, even with annotations; (ii) the inaccurate type annotations can undermine the ability of static type checkers to detect real bugs; (iii) static type checkers have different checking strategies in some cases, which has an impact on real bug detection. Our study can not only enable developers to better understand static type checking and make better use of them but also guide future research.

SMT solvers are utilized to check the satisfiability of logic formulas and have been applied in various crucial domains, including software verification, test case generation, and program synthesis. However, bugs hidden in SMT solvers can lead to severe consequences, causing erroneous results in these domains. Therefore, ensuring the reliability and robustness of SMT solvers is of critical importance. Despite several testing approaches proposed for SMT solvers, generating effective test formulas to comprehensively test SMT solvers remains a challenge. To address this challenge, in this study, we propose to port large language models (LLMs) to generate SMT formulas for fuzzing solvers. Specifically, the study presents a novel retrain-finetune pipeline to unleash the potential of language models to generate effective SMT formulas and improve their generation performance through data augmentation. We implemented our approach as a practical fuzzing tool, named LasT,and then extensively tested the state-of-the-art SMT solvers, namely Z3, cvc5, and Bitwuzla. To date, Last has successfully uncovered 65 genuine bugs for the solvers, of which 45 have been fixed by the developers.

Ensuring the correctness of code coverage profilers is crucial, given the widespread adoption of code coverage for various software engineering tasks. Existing validation techniques, such as differential testing and metamorphic testing, have shown effectiveness in uncovering bugs in coverage profilers. However, these techniques have limitations as they primarily rely on homogeneous sources, i.e., different coverage profilers or the profilers themselves, for validation. In this paper, we propose Decov, a novel heterogeneous testing technique, to validate coverage profilers using the information provided by debuggers as a heterogeneous source. Coverage profilers record execution counts for each source line in the program, while debuggers monitor hit counts for each source line when running the program in debug mode. Our key insight is that the execution counts obtained from coverage profilers should align with the hit counts monitored by debuggers, without conflicts. Decov constructs multiple heterogeneous relations and utilizes them to uncover bugs in coverage profilers. Through experiments on Gcov and LLVM-cov, two widely used code coverage profilers, we discovered 21 new bug reports, with 19 of them directly confirmed by developers. Notably, developers have resolved 5 bugs in the latest trunk version. Decov serves as a simple yet effective coverage profiler validator and offers a complementary approach to existing techniques.

Compiler testing is crucially important as compiler is the fundamental infrastructure in software development. One common compiler testing practice leverages a random program generator such as Csmith to generate a huge number of test programs to stress-test compilers. Each of the test programs will be compiled to different executables at different optimization levels and then their outputs will be compared against each other to differentially test compilers. However, the execution time of different test programs varies a lot. Therefore, in practice, developers often set a time limit, such as 60 or 300 seconds, to control the execution of different executables. If the execution exceeds the time limit, it will be terminated. Nevertheless, it is still unclear which time limit is more suitable for compiler testing in this context. We therefore perform the first empirical analysis to investigate how different time limits afffect the efficiency of compiler testing. We found that a time limit of 0.1 seconds can achieve the maximum benefits for compiler testing with the randomly generated test programs. At the same time, we found that 12% test programs requires more than 300 seconds for the execution and these test programs with long-execution-time (LET) consume more than 90% of the entire testing resources which makes compiler testing not cost-effectiveness. We thus propose a framework named ELECT to automatically identify and exclude LET test programs for boosting compiler testing. Our extensive experiments on two popular compilers GCC and LLVM have shown that ELECT can significantly improve the cost-effectiveness of compiler testing as it can respectively detect about 19% and 10% more bugs than the two baseline approaches under the same testing time budget. Besides, ELECT can respectively save about 12% and 38% time than the two other approaches for detecting the same number of bugs.

Many studies have explored the methods of deriving thresholds of object-oriented (i.e. OO) metrics. Unsupervised methods are mainly based on the distributions of metric values, while supervised methods principally rest on the relationships between metric values and defect-proneness of classes. The objective of this study is to empirically examine whether there are effective threshold values of OO metrics by analyzing existing threshold derivation methods with a large-scale meta-analysis. Based on five representative threshold derivation methods (i.e. VARL, ROC, BPP, MFM, and MGM) and 3268 releases from 65 Java projects, we first employ statistical meta-analysis and sensitivity analysis techniques to derive thresholds for 62 OO metrics on the training data. Then, we investigate the predictive performance of five candidate thresholds for each metric on the validation data to explore which of these candidate thresholds can be served as the threshold. Finally, we evaluate their predictive performance on the test data. The experimental results show that 26 of 62 metrics have the threshold effect and the derived thresholds by meta-analysis achieve promising results of GM values and significantly outperform almost all five representative (baseline) thresholds.

Background. Code-line-level bugginess identification (CLBI) is a vital technique that can facilitate developers to identify buggy lines without expending a large amount of human effort. Most of the existing studies tried to mine the characteristics of source codes to train supervised prediction models, which have been reported to be able to discriminate buggy code lines amongst others in a target program. Problem. However, several simple and clear code characteristics, such as complexity of code lines, have been disregarded in the current literature. Such characteristics can be acquired and applied easily in an unsupervised way to conduct more accurate CLBI, which also can decrease the application cost of existing CLBI approaches by a large margin. Objective. We aim at investigating the status quo in the field of CLBI from the perspective of (1) how far we have really come in the literature, and (2) how far we have yet to go in the industry, by analyzing the performance of state-of-the-art (SOTA) CLBI approaches and tools, respectively. Method. We propose a simple heuristic baseline solution GLANCE (aimin G at contro L - AN d C ompl E x-statements) with three implementations (i.e., GLANCE-MD, GLANCE-EA, and GLANCE-LR). GLANCE is a two-stage CLBI framework: first, use a simple model to predict the potentially defective files; second, leverage simple code characteristics to identify buggy code lines in the predicted defective files. We use GLANCE as the baseline to investigate the effectiveness of the SOTA CLBI approaches, including natural language processing (NLP) based, model interpretation techniques (MIT) based, and popular static analysis tools (SAT). Result. Based on 19 open-source projects with 142 different releases, the experimental results show that GLANCE framework has a prediction performance comparable or even superior to the existing SOTA CLBI approaches and tools in terms of 8 different performance indicators. Conclusion. The results caution us that, if the identification performance is the goal, the real progress in CLBI is not being achieved as it might have been envisaged in the literature and there is still a long way to go to really promote the effectiveness of static analysis tools in industry. In addition, we suggest using GLANCE as a baseline in future studies to demonstrate the usefulness of any newly proposed CLBI approach.

When encountering unfamiliar APIs, developers tend to seek help from API tutorials and Stack Overflow (SO). API tutorials help developers understand the API knowledge in a general context, while SO often explains the API knowledge in a specific programming task. Thus, tutorials and SO posts together can provide more API knowledge. However, it is non-trivial to retrieve API knowledge from both API tutorials and SO posts based on natural language queries. Two major problems are irrelevant API knowledge in two different resources and the lexical gap between the queries and documents. In this article, we regard a fragment in tutorials and a Question and Answering (Q&A) pair in SO as a knowledge item (KI). We generate ⟨ API, FRA ⟩ pairs (FRA stands for fragment) from tutorial fragments and APIs and build ⟨ API, QA ⟩ pairs based on heuristic rules of SO posts. We fuse ⟨ API, FRA ⟩ pairs and ⟨ API, QA ⟩ pairs to generate API knowledge (AK for short) datasets, where each data item is an ⟨ API, KI ⟩ pair. We propose a novel approach, called PLAN, to automatically retrieve API knowledge from both API tutorials and SO posts based on natural language queries. PLAN contains three main stages: (1) API knowledge modeling, (2) query mapping, and (3) API knowledge retrieving. It first utilizes a deep-transfer-metric-learning-based relevance identification (DTML) model to effectively find relevant ⟨ API, KI ⟩ pairs containing two different knowledge items (⟨ API, QA ⟩ pairs and ⟨ API, FRA ⟩ pairs) simultaneously. Then, PLAN generates several potential APIs as a way to reduce the lexical gap between the query and ⟨ API, KI ⟩ pairs. According to potential APIs, we can select relevant ⟨ API, KI ⟩ pairs to generate potential results. Finally, PLAN returns a list of ranked ⟨ API, KI ⟩ pairs that are related to the query. We evaluate the effectiveness of PLAN with 270 queries on Java and Android AK datasets containing 10,072 ⟨ API, KI ⟩ pairs. Our experimental results show that PLAN is effective and outperforms the state-of-the-art approaches. Our user study further confirms the effectiveness of PLAN in locating useful API knowledge.

Managing cross-project dependencies is tricky in modern software development. A primary way to manage dependencies is using dependency configuration files, which brings convenience to the entire software ecosystem, including developers, maintainers, and users. However, developers may introduce dependency smells if dependency configuration files are not well written and maintained. Dependency smells are recurring violations of dependency management in dependency configuration files and can potentially lead to severe consequences. This paper provides an in-depth look at three dependency smells, namely,Missing Dependency,Bloated Dependency, andVersion Constraint Inconsistencyin Python projects. First, we implement a tool calledPythonCross-projectDependency- PyCD to accurately extract dependency information from configuration files. The evaluation result on 212 Python projects shows that PyCD outperforms state-of-the-art tools. Then, we make an empirical study for three dependency smells in 132 Python projects to investigate the pervasiveness, causes, and evolution. The results show that: 1) dependency smells are prevalent in Python projects and exist inconsistently in different projects; 2) dependency smells are introduced into Python projects for different reasons, mainly due to the problems of synchronous update and collaborative development; and 3) dependency smells can be removed with different patterns according to different dependency smells. Furthermore, we report and get responses for 40 harmful dependency smell instances, 34 of which have been responded that these dependency smells do exist in the projects, and 10 instances are fixed or under process. The feedback from developers indicates that dependency smells can have a negative impact on project maintenance. Our study highlights that these dependency smells deserve the attention of developers.

Static analysis (SA) tools can generate useful static warnings to reveal the problematic code snippets in a software system without dynamically executing the corresponding source code. In the literature, static warnings are of paramount importance because they can easily indicate specific types of software defects in the early stage of a software development process, which accordingly reduces the maintenance costs by a substantial margin. Unfortunately, due to the conservative approximations of such SA tools, a large number of false positive (FP for short) warnings (i.e., they do not indicate real bugs) are generated, making these tools less effective. During the past two decades, therefore, many false positive mitigation (FPM for short) approaches have been proposed so that more accurate and critical warnings can be delivered to developers. This paper offers a detailed survey of research achievements on the topic of FPM. Given the collected 130 surveyed papers, we conduct a comprehensive investigation from five different perspectives. First, we reveal the research trends of this field. Second, we classify the existing FPM approaches into five different types and then present the concrete research progress. Third, we analyze the evaluation system applied to examine the performance of the proposed approaches in terms of studied SA tools, evaluation scenarios, performance indicators, and collected datasets, respectively. Fourth, we summarize the four types of empirical studies relating to SA warnings to exploit the insightful findings that are helpful to reduce FP warnings. Finally, we sum up 10 challenges unresolved in the literature from the aspects of systematicness, effectiveness, completeness, and practicability and outline possible research opportunities based on three emerging techniques in the future.

The label quality of defect data sets has a direct influence on the reliability of defect prediction models. In this paper, we conduct a systematic study of inconsistent defect labels in multi-version-project defect data sets, i.e., many instances having the same source code but different labels over multiple versions of a software project. First, we report the phenomena of inconsistent labels by real examples and analyze their essence in the context of defect prediction. Then, we uncover the causes that lead to the occurrence of inconsistent labels for the representative label collection approaches. Finally, we investigate the actual influence of inconsistent labels on defect prediction models. We find that inconsistent labels in general exist in six multi-version-project defect data sets (either widely used or the most up-to-date in the literature) collected by diverse label collection approaches. In particular, inconsistent labels in a training data set significantly reduce the prediction performance of a model, while inconsistent labels in a test data set can lead to a considerable evaluation bias on the real performance. Therefore, we recommend that: on the one hand, researchers leverage our findings to make targeted methodological improvements on existing defect label collection approaches to reduce the generation of inconsistent labels; on the other hand, practitioners detect and exclude inconsistent labels in defect data sets to avoid their potential negative influence on defect prediction.

Code coverage has been widely used as the basis for various software quality assurance techniques. Therefore, it is of great importance to ensure that coverage profilers provide reliable code coverage. However, it is challenging to validate the correctness of the code coverage generated due to the lack of an effective oracle. In this paper, we propose an effective approach based on control flow constraint solving to test coverage profilers and have implemented a coverage bug hunting tool, DOG (finD cOverage buGs). Our core idea is to leverage inherent control flow features to generate control flow constraints that the resulting coverage statistics should respect. If DOG identifies any unsatisfiable constraints, it signifies the presence of incorrect coverage statistics. In such cases, DOG provides detailed diagnostic information about the suspicious coverage statistics for manual inspection. Compared with the state-of-the-art works, DOG has the following prominent advantages: (1) wide applicability: DOG eliminates the need for multiple coverage profilers (as required by differential testing) and program variants (as needed in metamorphic testing), making it highly versatile; (2) unique testing capability: DOG effectively analyzes and utilizes relationships among available coverage statistics, boosting its testing capabilities; and (3) enhanced interpretability: DOG provides clear control flow explanations for incorrect code coverage, enabling the localization of suspicious coverage areas. During our testing period with DOG, we successfully identified and reported 27 bugs in Gcov and llvm-cov, both widely-used coverage profilers. Of these, 17 bugs have been confirmed (11 have been fixed), 3 were deemed expected behaviors by developers, and 7 remain unresolved. Remarkably, 21 out of 24 unexpected bugs had been latent for over two and a half years, and nearly half of the coverage bugs (10 out of 24) were undetectable by state-of-the-art coverage profiler validators. These results demonstrate the effectiveness and importance of using DOG to improve the reliability of code coverage profilers.

The ellipsis of the syntactic structure is a common phenomenon in ordinary textual documents. Existing studies have found that despite syntactic ellipsis could help avoid repetition of normative documents, it could also, for example, lead to ambiguity and hamper the understandability of document contents. As a fundamental component of software, code comments are generally written by developers in a non-structured way just like normative documents. This naturally inspires us to explore whether syntactic ellipsis is also a common phenomenon in code comments and what potential negative effects would such ellipsis have on software tasks such as code/comments comprehension activities. Such explorations, in our opinion, are expected to facilitate the research on code comments and comments-related software tasks. To this end, we conduct the first large-scale study to explore the syntactic structure ellipsis problem of code comments, with a focus on Java inline comments. Specifically, we construct a data set of 1,000 Java projects with 1,307,457 inline comments and associated codes. Based on this data set, we first study the prevalence of syntactic structure ellipsis in inline comments. We find that syntactic structure ellipsis is quite common in inline comments where 83.6% comments have structure ellipsis (such as subject/predicate omissions). Then, we investigate the effects of syntactic structure ellipsis on code/comment understanding activities. As a result, we find that there indeed exists a negative relationship between them, with a medium effect size. Based on these findings, we further propose neural network based approaches to complete the ellipsis parts for the inline comments. With our approach, we could achieve: 1) a medium improvement in assisting code/comment understanding activities, and 2) a substantial improvement of 11.3% in comment-assisted code abbreviation extension task.

With the widespread application of machine learning (ML) software, especially in high-risk tasks, the concern about their unfairness has been raised towards both developers and users of ML software. The unfairness of ML software indicates the software behavior affected by the sensitive features (e.g., sex), which leads to biased and illegal decisions and has become a worthy problem for the whole software engineering community.

Unit testing is a critical part of software development process, ensuring the correctness of basic programming units in a program (e.g., a method). Search-based software testing (SBST) is an automated approach to generating test cases. SBST generates test cases with genetic algorithms by specifying the coverage criterion (e.g., branch coverage). However, a good test suite must have different properties, which cannot be captured by using an individual coverage criterion. Therefore, the state-of-the-art approach combines multiple criteria to generate test cases. As combining multiple coverage criteria brings multiple objectives for optimization, it hurts the test suites’ coverage for certain criteria compared with using the single criterion. To cope with this problem, we propose a novel approach named smart selection. Based on the coverage correlations among criteria and the coverage goals’ subsumption relationships, smart selection selects a subset of coverage goals to reduce the number of optimization objectives and avoid missing any properties of all criteria. We conduct experiments to evaluate smart selection on 400 Java classes with three state-of-the-art genetic algorithms. On average, smart selection outperforms combining all goals on of the classes having significant differences between the two approaches.

Code optimization is an essential feature for compilers and almost all software products are released by compiler optimizations. Consequently, bugs in code optimization will inevitably cast significant impact on the correctness of software systems. Locating optimization bugs in compilers is challenging as compilers typically support a large amount of optimization configurations. Although prior studies have proposed to locate compiler bugs via generating witness test programs, they are still time-consuming and not effective enough. To address such limitations, we propose an automatic bug localization approach, ODFL, for locating compiler optimization bugs via differentiating finer-grained options in this study. Specifically, we first disable the fine-grained options that are enabled by default under the bug-triggering optimization levels independently to obtain bug-free and bug-related fine-grained options. We then configure several effective passing and failing optimization sequences based on such fine-grained options to obtain multiple failing and passing compiler coverage. Finally, such generated coverage information can be utilized via Spectrum-Based Fault Localization formulae to rank the suspicious compiler files. We run ODFL on 60 buggy GCC compilers from an existing benchmark. The experimental results show that ODFL significantly outperforms the state-of-the-art compiler bug isolation approach RecBi in terms of all the evaluated metrics, demonstrating the effectiveness of ODFL. In addition, ODFL is much more efficient than RecBi as it can save more than 88% of the time for locating bugs on average.

Heterogeneous defect prediction (HDP) refers to the defect prediction across projects with different metrics. Most existing HDP methods map source and target data into a common metric space where each dimension has no actual meaning, which weakens their interpretability. Besides, HDP always suffers from the class imbalance problem. For deficiencies of current HDP methods, we intend to propose a novel HDP approach that can reduce the heterogeneity of source and target data and deal with imbalanced data while retaining the actual meaning for each dimension of constructed common metric space. We propose an Aligned Metric Representation based Balanced Multiset Ensemble learning (BMEL+ AMR) approach for HDP. AMR consists of shared, source-specific, and target-specific metrics. It is built by learning the translation from shared to specific metrics and reducing the distribution difference. To deal with imbalanced data, we design BMEL that constructs multiple balanced subsets for source data and produces an aggregated classifier for predicting labels of target data. Experimental results on 22 public projects indicate that (1) among all competing methods, BMEL+AMR achieves the best performance on all indicators except Popt, followed by AMR; (2) compared with AMR, the introduction of BMEL improves the performance on non-effort-aware indicators statistically significantly except F1-score; compared with BMEL, the introduction of AMR improves the performance throughout all indicators statistically significantly. BMEL+AMR can effectively improve HDP performance by eliminating heterogeneity and dealing with imbalanced data, and AMR is helpful to explain the prediction model.

The computational methods of protein-protein interaction sites prediction can effectively avoid the shortcomings of high cost and time in traditional experimental approaches. However, the serious class imbalance between interface and non-interface residues on the protein sequences limits the prediction performance of these methods. This work therefore proposed a new strategy, NearMiss-based under-sampling for unbalancing datasets and Random Forest classification (NM-RF), to predict protein interaction sites. Herein, the residues on protein sequences were represented by the PSSM-derived features, hydropathy index (HI) and relative solvent accessibility (RSA). In order to resolve the class imbalance problem, an under-sampling method based on NearMiss algorithm is adopted to remove some non-interface residues, and then the random forest algorithm is used to perform binary classification on the balanced feature datasets. Experiments show that the accuracy of NM-RF model reaches 87.6% and 84.3% on Dtestset72 and PDBtestset164 respectively, which demonstrate the effectiveness of the proposed NM-RF method in differentiating the interface or non-interface residues.

Background. Mutation testing is a commonly used defect injection technique for evaluating the effectiveness of a test suite. However, it is usually computationally expensive. Therefore, many mutation reduction strategies, which aim to reduce the number of mutants, have been proposed. Problem. It is important to measure the ability of a mutation reduction strategy to maintain test suite effectiveness evaluation. However, existing evaluation indicators are unable to measure the “order-preserving ability”, i.e., to what extent the mutation score order among test suites is maintained before and after mutation reduction. As a result, misleading conclusions can be achieved when using existing indicators to evaluate the reduction effectiveness. Objective. We aim to propose evaluation indicators to measure the “order-preserving ability” of a mutation reduction strategy, which is important but missing in our community. Method. Given a test suite on a Software Under Test (SUT) with a set of original mutants, we leverage the test suite to generate a group of test suites that have a partial order relationship in defect detecting ability. When evaluating a reduction strategy, we first construct two partial order relationships among the generated test suites in terms of mutation score, one with the original mutants and another with the reduced mutants. Then, we measure the extent to which the partial order under the original mutants remains unchanged in the partial order under the reduced mutants. The more partial order is unchanged, the stronger the Order Preservation ( OP ) of the mutation reduction strategy is, and the more effective the reduction strategy is. Furthermore, we propose Effort-aware Relative Order Preservation ( EROP ) to measure how much gain a mutation reduction strategy can provide compared with a random reduction strategy. Result. The experimental results show that OP and EROP are able to efficiently measure the “order-preserving ability” of a mutation reduction strategy. As a result, they have a better ability to distinguish various mutation reduction strategies compared with the existing evaluation indicators. In addition, we find that Subsuming Mutant Selection (SMS) and Clustering Mutant Selection (CMS) are more effective than the other strategies under OP and EROP. Conclusion. We suggest, for the researchers, that OP and EROP should be used to measure the effectiveness of a mutant reduction strategy, and for the practitioners, that SMS and CMS should be given priority in practice.

Software refactoring is one of the most significant practices in software maintenance as the quality of software design tends to deteriorate during software evolution. But, refactoring software is a very challenging task as it requires a holistic view of the entire software system. To this end, recent studies introduced search-based algorithms to facilitate software refactoring. However, they still have the following major limitations: 1) the searched solutions may violate the design principles as their fitness functions do not directly reflect the degree of software’s compliance with design principles; 2) most approaches start the searching process from a completely random initial population, which may lead to unoptimal solutions. In this article, we aim to develop effective search-based refactoring approach to recommend better refactoring activities for developers which can improve the degree of software’s compliance with design principles as well as the software design quality. We proposeDEPICTER, a design-principle guided and heuristic-rule constrained software refactoring recommendation approach. In particular,DEPICTERuses non-dominated sorting genetic algorithm (NSGA)-II genetic algorithm and employs design-principle metrics as fitness functions. Besides,DEPICTERleverages heuristic rules to improve the quality of initial population for subsequent generic evolution. Our evaluations, based on four widely used systems, show thatDEPICTERis effective for guiding the development of better refactoring models in practice.

Knowing the effectiveness of a test suite is essential for many activities such as assessing the test adequacy of code and guiding the generation of new test cases. Mutation testing is a commonly used defect injection technique for evaluating the effectiveness of a test suite. However, it is usually computationally expensive, as a large number of mutants (buggy versions) are needed to be generated from a production code under test and executed against the test suite. In order to reduce the expensive testing cost, recent studies proposed to use supervised models to predict the effectiveness of a test suite without executing the test suite against the mutants. Nonetheless, the training of such a supervised model requires labeled data, which still depends on the costly mutant execution. Furthermore, existing models are based on traditional supervised learning techniques, which assume that the training and testing data come from the same distribution. But, in practice, software systems are subject to considerable concept drifts, i.e., the same distribution assumption usually does not hold. This can lead to inaccurate predictions of a learned supervised model on the target code as time progresses. To tackle these problems, in this paper, we propose a Coverage-Based Unsupervised Approach (CBUA) for evaluating the effectiveness of a test suite. Given a production code under test, the corresponding mutants, and a test suite, CBUA first collects the coverage information of the mutated statements in the target production code under the execution of the test suite. Then, CBUA employs coverage to estimate the probability of each mutant being alive. As such, a mutation score is computed to evaluate the test suite effectiveness and the predicted labels (i.e., killed or alive) are obtained. The whole process only requires a one-time execution of the test suite against the target production code, without involving any mutant execution and any training data. CBUA can ensure the score monotonicity property (i.e., adding test cases to a test suite does not decrease its mutation score), which may be violated by a supervised approach. The experimental results show that CBUA is very competitive with the state-of-the-art supervised approaches in prediction accuracy. In particular, CBUA is shown to be more effective in finding mutants that are covered but not killed by a test suite, which is helpful in identifying the weaknesses in the current test suite and generating new test cases accordingly. Since CBUA is an easy-to-implement approach with a low cost, we suggest that it should be used as a baseline approach for comparison when any novel prediction approach is proposed in future studies.

The boom of DL technology leads to massive DL models built and shared, which facilitates the acquisition and reuse of DL models. For a given task, we encounter multiple DL models available with the same functionality, which are considered as candidates to achieve this task. Testers are expected to compare multiple DL models and select the more suitable ones w.r.t. the whole testing context. Due to the limitation of labeling effort, testers aim to select an efficient subset of samples to make an as precise rank estimation as possible for these models. To tackle this problem, we propose Sample Discrimination based Selection (SDS) to select efficient samples that could discriminate multiple models, i.e., the prediction behaviors (right/wrong) of these samples would be helpful to indicate the trend of model performance. To evaluate SDS, we conduct an extensive empirical study with three widely-used image datasets and 80 real world DL models. The experiment results show that, compared with state-of-the-art baseline methods, SDS is an effective and efficient sample selection method to rank multiple DL models.

Developers often use learning resources such as API tutorials and Stack Overflow (SO) to learn how to use an unfamiliar API. An API tutorial can be divided into a number of consecutive units that describe the same topic, denoted as tutorial fragments. We consider a tutorial fragment explaining the API usage knowledge as a relevant fragment of the API. Discovering relevant tutorial fragments of APIs can facilitate API understanding and learning. However, existing approaches, based on supervised or unsupervised approaches, often suffer from either high manual efforts or lack of consideration of the relevance information. In this paper, we propose a novel approach, called SO2RT, to detect relevant tutorial fragments of APIs based on SO posts. SO2RT first automatically extracts relevant and irrelevant 〈API,QA〉 pairs based on heuristic rules of SO, and constructs 〈API, FRA〉 pairs (FRA stands out fragment) by using tutorial fragments and APIs. SO2RT then trains a semi-supervised transfer learning based detection model, which can transfer the API usage knowledge in SO Q&A pairs to tutorial fragments by utilizing the easy-to-extract relevance of 〈API, QA〉 pairs. Finally, relevant fragments of APIs can be discovered by consulting the trained model. In this way, the effort for labeling the relevance between tutorial fragments and APIs can be reduced. We evaluate SO2RT on Java and Android datasets containing 21,008 〈API, QA〉 pairs. Experimental results show that SO2RT improves the state-of-the-art approaches in terms of F-Measure on both datasets. Our user study further confirms the effectiveness of SO2RT in practice.

Protein-protein interactions play essential roles in various biological progresses. Identifying protein interaction sites can facilitate researchers to understand life activities and therefore will be helpful for drug design. However, the number of experimental determined protein interaction sites is far less than that of protein sites in protein-protein interaction or protein complexes. Therefore, the negative and positive samples are usually imbalanced, which is common but bring result bias on the prediction of protein interaction sites by computational approaches. In this work, we presented three imbalance data processing strategies to reconstruct the original dataset, and then extracted protein features from the evolutionary conservation of amino acids to build a predictor for identification of protein interaction sites. On a dataset with 10,430 surface residues but only 2,299 interface residues, the imbalance dataset processing strategies can obviously reduce the prediction bias, and therefore improve the prediction performance of protein interaction sites. The experimental results show that our prediction models can achieve a better prediction performance, such as a prediction accuracy of 0.758, or a high F-measure of 0.737, which demonstrated the effectiveness of our method.

Pathogenicity-related studies are of great importance in understanding the pathogenesis of complex diseases and improving the level of clinical medicine. This work proposed a bioinformatics scheme to analyze cancer-related gene mutations, and try to figure out potential genes associated with diseases from the protein domain-domain interaction network. Herein, five measures of the principle of centrality lethality had been adopted to implement potential correlation analysis, and prioritize the significance of genes. This method was further applied to KEGG pathway analysis by taking the malignant melanoma as an example. The experimental results show that 25 domains can be found, and 18 of them have high potential to be pathogenically important related to malignant melanoma. Finally, a web-based tool, named Human Cancer Related Domain Interaction Network Analyzer, is developed for potential pathogenic genes prioritization for 26 types of human cancers, and the analysis results can be visualized and downloaded online.

Background. Self-admitted technical debt (SATD) is a special kind of technical debt that is intentionally introduced and remarked by code comments. Those technical debts reduce the quality of software and increase the cost of subsequent software maintenance. Therefore, it is necessary to find out and resolve these debts in time. Recently, many automatic approaches have been proposed to identify SATD. Problem. Popular IDEs support a number of predefined task annotation tags for indicating SATD in comments, which have been used in many projects. However, such clear prior knowledge is neglected by existing SATD identification approaches when identifying SATD. Objective. We aim to investigate how far we have really progressed in the field of SATD identification by comparing existing approaches with a simple approach that leverages the predefined task tags to identify SATD. Method. We first propose a simple heuristic approach that fuzzily Matches task Annotation Tags ( MAT ) in comments to identify SATD. In nature, MAT is an unsupervised approach, which does not need any data to train a prediction model and has a good understandability. Then, we examine the real progress in SATD identification by comparing MAT against existing approaches. Result. The experimental results reveal that: (1) MAT has a similar or even superior performance for SATD identification compared with existing approaches, regardless of whether non-effort-aware or effort-aware evaluation indicators are considered; (2) the SATDs (or non-SATDs) correctly identified by existing approaches are highly overlapped with those identified by MAT ; and (3) supervised approaches misclassify many SATDs marked with task tags as non-SATDs, which can be easily corrected by their combinations with MAT . Conclusion. It appears that the problem of SATD identification has been (unintentionally) complicated by our community, i.e., the real progress in SATD comments identification is not being achieved as it might have been envisaged. We hence suggest that, when many task tags are used in the comments of a target project, future SATD identification studies should use MAT as an easy-to-implement baseline to demonstrate the usefulness of any newly proposed approach.

Software projects are increasingly forming social-technical ecosystems within which individual projects rely on the infrastructures or functional components provided by other projects, leading to complex inter-dependencies. Through inter-project dependencies, a bug in an upstream project may have profound impact on a large number of downstream projects, resulting in cross-project bugs. This emerging type of bugs has brought new challenges in bug fixing due to their unclear influence on downstream projects. In this paper, we present an approach to estimating the impact of a cross-project bug within its ecosystem by identifying the affected downstream modules (classes/methods). Note that a downstream project that uses a buggy upstream function may not be affected as the usage does not satisfy the failure inducing preconditions. For a reported bug with the known root cause function and failure inducing preconditions, we first collect the candidate downstream modules that call the upstream function through an ecosystem-wide dependence analysis. Then, the paths to the call sites of the buggy upstream function are encoded as symbolic constraints. Solving the constraints, together with the failure inducing preconditions, identifies the affected downstream modules. Our evaluation of 31 existing upstream bugs on the scientific Python ecosystem containing 121 versions of 22 popular projects (with a total of 16 millions LOC) shows that the approach is highly effective: from the 25490 candidate downstream modules that invoke the buggy upstream functions, it identifies 1132 modules where the upstream bugs can be triggered, pruning 95.6% of the candidates. The technique has no false negatives and an average false positive rate of 7.9%. Only 49 downstream modules (out of the 1132 we found) were reported before to be affected.

With the increasing application of deep learning (DL) models in many safety-critical scenarios, effective and efficient DL testing techniques are much in demand to improve the quality of DL models. One of the major challenges is the data gap between the training data to construct the models and the testing data to evaluate them. To bridge the gap, testers aim to collect an effective subset of inputs from the testing contexts, with limited labeling effort, for retraining DL models.

Cloud computing has promoted the success of big data applications such as medical data analyses. With the abundant resources provisioned by cloud platforms, the quality of service (QoS) of services that process big data could be boosted significantly. However, due to unstable network or fake advertisement, the QoS published by service providers is not always trusted. Therefore, it becomes a necessity to evaluate the service quality in a trustable way, based on the services' historical QoS records. However, the evaluation efficiency would be low and cannot meet users' quick response requirement, if all the records of a service are recruited for quality evaluation. Moreover, it may lead to `Lagging Effect' or low evaluation accuracy, if all the records are treated equally, as the invocation contexts of different records are not exactly the same. In view of these challenges, a novel approach named Partial Historical Records-based service evaluation approach (Partial-HR) is put forward in this paper. In Partial-HR, each historical QoS record is weighted based on its service invocation context. Afterwards, only partial important records are employed for quality evaluation. Finally, a group of experiments are deployed to validate the feasibility of our proposal, in terms of evaluation accuracy and efficiency.

Generic programming is a key paradigm for developing reusable software components. The inherent support for generic constructs is therefore important in programming languages. As for C++, the generic construct, templates, has been supported since the language was first released. However, little is currently known about how C++ templates are actually used in developing real software. In this study, we conduct an experiment to investigate the use of templates in practice. We analyze 1,267 historical revisions of 50 open source systems, consisting of 566 million lines of C++ code, to collect the data of the practical use of templates. We perform statistical analyses on the collected data and produce many interesting results. We uncover the following important findings: (1) templates are practically used to prevent code duplication, but this benefit is largely confined to a few highly used templates; (2) function templates do not effectively replace C-style generics, and developers with a C background do not show significant preference between the two language constructs; (3) developers seldom convert dynamic polymorphism to static polymorphism by using CRTP (Curiously Recursive Template Pattern); (4) the use of templates follows a power-law distribution in most cases, and C++ developers who prefer using templates are those without other language background; (5) C developer background seems to override C++ project guidelines. These findings are helpful not only for researchers to understand the tendency of template use but also for tool builders to implement better tools to support generic programming.

Spectrum-based fault localization (SBFL) techniques can automatically localize software faults. They employ the program spectrum, such as code coverage profile with test verdicts, to rank the program entities based on their code suspiciousness. In the past decades, researchers have proposed many approaches to optimize these techniques; however, the program structure, which can influence their performance, is not taken into consideration in developing and improving these techniques. In this article, we identify and analyze the effect of the program structure on the application of SBFL techniques. We observe that some specific program structures may introduce structure bias to code suspiciousness and negatively influence the output of SBFL techniques. To mitigate these effects and improve the performance of fault localization, we propose Delta4Ts, a structure-aware technique. Delta4Ts references debugging history to alleviate the impact of structure bias in the calculation of code suspiciousness. It reasons from the observable suspicious value towards the desired suspicious value and the impact of structure bias. To evaluate Delta4Ts under practical constraints, we conduct a controlled experiment using nine widely-studied SBFL formulae on 12 C programs and 6 Java programs. The experiment results show that Delta4Ts can significantly improve the accuracy of the studied SBFL formulae by an average of 34.8% on 12 C programs and 30.6% on 6 Java programs, and improve more on subject programs associated with more history versions or having larger code sizes.

Reliable code coverage tools are critically important as it is heavily used to facilitate many quality assurance activities, such as software testing, fuzzing, and debugging. However, little attention has been devoted to assessing the reliability of code coverage tools. In this study, we propose a randomized differential testing approach to hunting for bugs in the most widely used C code coverage tools. Specifically, by generating random input programs, our approach seeks for inconsistencies in code coverage reports produced by different code coverage tools, and then identifies inconsistencies as potential code coverage bugs. To effectively report code coverage bugs, we addressed three specific challenges: (1) How to filter out duplicate test programs as many of them triggering the same bugs in code coverage tools; (2) how to automatically reduce large test programs to much smaller ones that have the same properties; and (3) how to determine which code coverage tools have bugs? The extensive evaluations validate the effectiveness of our approach, resulting in 42 and 28 confirmed/fixed bugs for gcov and llvm-cov, respectively. This case study indicates that code coverage tools are not as reliable as it might have been envisaged. It not only demonstrates the effectiveness of our approach, but also highlights the need to continue improving the reliability of code coverage tools. This work opens up a new direction in code coverage validation which calls for more attention in this area.

Code coverage as the primitive dynamic program behavior information, is widely adopted to facilitate a rich spectrum of software engineering tasks, such as testing, fuzzing, debugging, fault detection, reverse engineering, and program understanding. Thanks to the widespread applications, it is crucial to ensure the reliability of the code coverage profilers. Unfortunately, due to the lack of research attention and the existence of testing oracle problem, coverage profilers are far away from being tested sufficiently. Bugs are still regularly seen in the widely deployed profilers, like gcov and llvm-cov, along with gcc and llvm, respectively. This paper proposes Cod, an automated self-validator for effectively uncovering bugs in the coverage profilers. Starting from a test program (either from a compiler's test suite or generated randomly), Cod detects profiler bugs with zero false positive using a metamorphic relation in which the coverage statistics of that program and a mutated variant are bridged. We evaluated Cod over two of the most well-known code coverage profilers, namely gcov and llvm-cov. Within a four-month testing period, a total of 196 potential bugs (123 for gcov, 73 for llvm-cov) are found, among which 23 are confirmed by the developers.

BACKGROUND: The recognition of protein interaction sites is of great significance in many biological processes, signaling pathways and drug designs. However, most sites on protein sequences cannot be defined as interface or non-interface sites because only a small part of protein interactions had been identified, which will cause the lack of prediction accuracy and generalization ability of predictors in protein interaction sites prediction. Therefore, it is necessary to effectively improve prediction performance of protein interaction sites using large amounts of unlabeled data together with small amounts of labeled data and background knowledge today. RESULTS: In this work, three semi-supervised support vector machine-based methods are proposed to improve the performance in the protein interaction sites prediction, in which the information of unlabeled protein sites can be involved. Herein, five features related with the evolutionary conservation of amino acids are extracted from HSSP database and Consurf Sever, i.e., residue spatial sequence spectrum, residue sequence information entropy and relative entropy, residue sequence conserved weight and residual Base evolution rate, to represent the residues within the protein sequence. Then three predictors are built for identifying the interface residues from protein surface using three types of semi-supervised support vector machine algorithms. CONCLUSION: The experimental results demonstrated that the semi-supervised approaches can effectively improve prediction performance of protein interaction sites when unlabeled information is involved into the predictors and one of them can achieve the best prediction performance, i.e., the accuracy of 70.7%, the sensitivity of 62.67% and the specificity of 78.72%, respectively. With comparison to the existing studies, the semi-supervised models show the improvement of the predication performance.

Abstract Conformal parameterization for surfaces into various parameter domains is a fundamental task in computer graphics. Prior research on discrete Ricci flow provided us with promising inspirations from methods derived via Riemannian geometry, which is rigorous in theory and effective inpractice. In this paper, we propose a unified conformal parameterization approachfor turning triangle meshes into planar and spherical domains using discrete Calabi flow onpiecewise linear metric. We incorporate edge‐flipping surgery to guarantee convergence as well as other significant improvements including approximate Newton's method, optimal step‐lengths, priority embedding and boundary customizing, which achieve better performance and functionality with robustness and accuracy.

Software systems becomes increasingly complex for the wide use of social-software-development platforms, such as GitHub. Complicated inter-dependencies within ecosystems impose new challenges in resolving the blocking bug pairs, in which the upstream bugs prevent the downstream bugs to be fixed. Generally, the downstream bugs cannot be fixed until the upstream bugs are fixed in blocking bug pairs, which keeps the downstream developers waiting for a long time. However, previous research found that some blocking pairs could be "broken" through a workaround, i.e., a temporary solution proposed by the downstream developers before the upstream bugs get fixed. In this paper, we propose an approach to describe and predict the breakability of the blocking bug pairs. Our goal is to help the ecosystem developers to predict whether the given blocking bug pair can be broken. We evaluate our approach on two real world ecosystems: Mozilla Firefox and Netbeans, and have the following two main findings. First, the participants within the blocking bug pair and the vitality of the downstream bug have much impact on the breakability. Second, we can build breakability predication models with the accuracy of approximate 80%. Moreover, for predicting unbreakable pairs which affect the downstream projects seriously, our approach achieves a precision over 92%. It can be used to effectively remind the upstream developers to fix these bugs as quickly as possible.

Accurate software defect prediction could help software practitioners allocate test resources to defect-prone modules effectively and efficiently. In the last decades, much effort has been devoted to build accurate defect prediction models, including developing quality defect predictors and modeling techniques. However, current widely used defect predictors such as code metrics and process metrics could not well describe how software modules change over the project evolution, which we believe is important for defect prediction. In order to deal with this problem, in this paper, we propose to use the Historical Version Sequence of Metrics (HVSM) in continuous software versions as defect predictors. Furthermore, we leverage Recurrent Neural Network (RNN), a popular modeling technique, to take HVSM as the input to build software prediction models. The experimental results show that, in most cases, the proposed HVSM-based RNN model has significantly better effort-aware ranking effectiveness than the commonly used baseline models.

Recommending appropriate product items to the target user is becoming the key to ensure continuous success of E-commerce. Today, many E-commerce systems adopt various recommendation techniques, e.g., Collaborative Filtering (abbreviated as CF)-based technique, to realize product item recommendation. Overall, the present CF recommendation can perform very well, if the target user owns similar friends (user-based CF), or the product items purchased and preferred by target user own one or more similar product items (item-based CF). While due to the sparsity of big rating data in E-commerce, similar friends and similar product items may be both absent from the user-product purchase network, which lead to a big challenge to recommend appropriate product items to the target user. Considering the challenge, we put forward a Structural Balance Theory-based Recommendation (i.e., SBT-Rec) approach. In the concrete, (I) user-based recommendation: we look for target user's “enemy” (i.e., the users having opposite preference with target user); afterwards, we determine target user's “possible friends”, according to “enemy's enemy is a friend” rule of Structural Balance Theory, and recommend the product items preferred by “possible friends” of target user to the target user. (II) likewise, for the product items purchased and preferred by target user, we determine their “possibly similar product items” based on Structural Balance Theory and recommend them to the target user. At last, the feasibility of SBT-Rec is validated, through a set of experiments deployed on MovieLens-1M dataset.

Background. Recent years have seen an increasing interest in cross-project defect prediction (CPDP), which aims to apply defect prediction models built on source projects to a target project. Currently, a variety of (complex) CPDP models have been proposed with a promising prediction performance. Problem. Most, if not all, of the existing CPDP models are not compared against those simple module size models that are easy to implement and have shown a good performance in defect prediction in the literature. Objective. We aim to investigate how far we have really progressed in the journey by comparing the performance in defect prediction between the existing CPDP models and simple module size models. Method. We first use module size in the target project to build two simple defect prediction models, ManualDown and ManualUp, which do not require any training data from source projects. ManualDown considers a larger module as more defect-prone, while ManualUp considers a smaller module as more defect-prone. Then, we take the following measures to ensure a fair comparison on the performance in defect prediction between the existing CPDP models and the simple module size models: using the same publicly available data sets, using the same performance indicators, and using the prediction performance reported in the original cross-project defect prediction studies. Result. The simple module size models have a prediction performance comparable or even superior to most of the existing CPDP models in the literature, including many newly proposed models. Conclusion. The results caution us that, if the prediction performance is the goal, the real progress in CPDP is not being achieved as it might have been envisaged. We hence recommend that future studies should include ManualDown/ManualUp as the baseline models for comparison when developing new CPDP models to predict defects in a complete target project.

GitHub has fostered complicated and enormous software ecosystems, in which projects depend on and co-evolve with each other. An error in an upstream project may affect its downstream projects through inter-dependencies, forming crossproject bugs. Though the upstream developers should fix the bugs on their side, proposing a workaround, i.e., a temporary solution in the downstream project is a common practice for the downstream developers. In this study, we empirically investigated the characteristics of downstream workarounds in the scientific Python ecosystem. Combining the statistical comparisons and manual inspection, we have the following three main findings. First, in general, the workarounds and the corresponding upstream fixes are significantly different in code size and code structure. Second, there are three kinds of crossproject bugs that the downstream developers usually work around. Last, four types of common patterns are identified from the investigated workarounds. The findings of this study lead to better understanding of cross-project bugs and the practices of developers in software ecosystems.

Background: Many relevancy filters have been proposed to select training data for building cross-project defect prediction (CPDP) models. However, up to now, there is no consensus about which relevancy filter is better for CPDP. Goal: In this paper, we conduct a thorough experiment to compare nine relevancy filters proposed in the recent literature. Method: Based on 33 publicly available data sets, we compare not only the retaining ratio of the original training data and the overlapping degree among the retained data but also the prediction performance of the resulting CPDP models under the ranking and classification scenarios. Results: In terms of retaining ratio and overlapping degree, there are important differences among these filters. According to the defect prediction performance, global filter always stays in the first level. Conclusions: For practitioners, it appears that there is no need to filter source project data, as this may lead to better defect prediction results.

Background: An increasing research effort has devoted to just-in-time (JIT) defect prediction. A recent study by Yang et al. at FSE'16 leveraged individual change metrics to build unsupervised JIT defect prediction model. They found that many unsupervised models performed similarly to or better than the state-of-the-art supervised models in effort-aware JIT defect prediction. Goal: In Yang et al.'s study, code churn (i.e. the change size of a code change) was neglected when building unsupervised defect prediction models. In this study, we aim to investigate the effectiveness of code churn based unsupervised defect prediction model in effort-aware JIT defect prediction. Methods: Consistent with Yang et al.'s work, we first use code churn to build a code churn based unsupervised model (CCUM). Then, we evaluate the prediction performance of CCUM against the state-of-the-art supervised and unsupervised models under the following three prediction settings: cross-validation, time-wise cross-validation, and cross-project prediction. Results: In our experiment, we compare CCUM against the state-of-the-art supervised and unsupervised JIT defect prediction models. Based on six open-source projects, our experimental results show that CCUM performs better than all the prior supervised and unsupervised models. Conclusions: The result suggests that future JIT defect prediction studies should use CCUM as a baseline model for comparison when a novel model is proposed.

GitHub, a popular social-software-development platform, has fostered a variety of software ecosystems where projects depend on one another and practitioners interact with each other. Projects within an ecosystem often have complex inter-dependencies that impose new challenges in bug reporting and fixing. In this paper, we conduct an empirical study on cross-project correlated bugs, i.e., causally related bugs reported to different projects, focusing on two aspects: 1) how developers track the root causes across projects, and 2) how the downstream developers coordinate to deal with upstream bugs. Through manual inspection of bug reports collected from the scientific Python ecosystem and an online survey with developers, this study reveals the common practices of developers and the various factors in fixing cross-project bugs. These findings provide implications for future software bug analysis in the scope of ecosystem, as well as shed light on the requirements of issue trackers for such bugs.

In practice, the data owners of source projects may need to share data without disclosing sensitive information. Therefore, privacy-preserving data-sharing becomes an important topic in cross-company defect prediction (CCDP). In this context, the challenge is how to achieve a high privacy-preserving level while ensuring the utility of the shared privatized data for CCDP. CLIFF&MORPH is a recently proposed state-of-the-art privacy-preserving data-sharing algorithm for CCDP. It has been reported that the CLIFF&MORPH CCDP model produces a promising defect prediction performance. However, we find that ManualDown, a simple (unsupervised) module size model, built on the target projects has a comparable or even better defect prediction performance. Since ManualDown does not require any source project data to build the model, it is free of the privacy-preserving data-sharing challenges for CCDP. This means that, for practitioners, the motivation of applying privacy-preserving data-sharing algorithms to CCDP could not be well justified if the utility challenge is not addressed. We analyze the implications of our findings and outline the directions for future research. In particular, we strongly suggest that future studies at least use ManualDown as a baseline model for comparison to help develop practical privacy-preserving data-sharing algorithms for CCDP.

Continuous Integration (CI) has become a disruptive innovation in software development: with proper tool support and adoption, positive effects have been demonstrated for pull request throughput and scaling up of project sizes. As any other innovation, adopting CI implies adapting existing practices in order to take full advantage of its potential, and "best practices" to that end have been proposed. Here we study the adaptation and evolution of code writing and submission, issue and pull request closing, and testing practices as TRAVIS CI is adopted by hundreds of established projects on GITHUB. To help essentialize the quantitative results, we also survey a sample of GITHUB developers about their experiences with adopting TRAVIS CI. Our findings suggest a more nuanced picture of how GITHUB teams are adapting to, and benefiting from, continuous integration technology than suggested by prior work.

In recent years, it has been shown that fault prediction models could effectively guide test effort allocation in finding faults if they have a high enough fault prediction accuracy (Norm(Popt) > 0.78). However, it is often difficult to achieve such a high fault prediction accuracy in practice. As a result, fault-prediction-model-guided allocation (FPA) methods may be not applicable in real development environments. To attack this problem, in this paper, we propose a new type of test effort allocation strategy: reliability-growth-model-guided allocation (RGA) method. For a given project release V, RGA attempts to predict the optimal test effort allocation for V by learning the fault distribution information from the previous releases. Based on three open-source projects, we empirically investigate the cost-effectiveness of three test effort allocation strategies for finding faults: RGA, FPA, and structural-complexity-guided allocation (SCA) method. The experimental results show that RGA shows a promising performance in finding faults when compared with SCA and FPA.

A dependence cluster is a set of mutually inter-dependent program elements. Prior studies have found that large dependence clusters are prevalent in software systems. It has been suggested that dependence clusters have potentially harmful effects on software quality. However, little empirical evidence has been provided to support this claim. The study presented in this paper investigates the relationship between dependence clusters and software quality at the function-level with a focus on effort-aware fault-proneness prediction. The investigation first analyzes whether or not larger dependence clusters tend to be more fault-prone. Second, it investigates whether the proportion of faulty functions inside dependence clusters is significantly different from the proportion of faulty functions outside dependence clusters. Third, it examines whether or not functions inside dependence clusters playing a more important role than others are more fault-prone. Finally, based on two groups of functions (i.e., functions inside and outside dependence clusters), the investigation considers a segmented fault-proneness prediction model. Our experimental results, based on five well-known open-source systems, show that (1) larger dependence clusters tend to be more fault-prone; (2) the proportion of faulty functions inside dependence clusters is significantly larger than the proportion of faulty functions outside dependence clusters; (3) functions inside dependence clusters that play more important roles are more fault-prone; (4) our segmented prediction model can significantly improve the effectiveness of effort-aware fault-proneness prediction in both ranking and classification scenarios. These findings help us better understand how dependence clusters influence software quality.

Unsupervised models do not require the defect data to build the prediction models and hence incur a low building cost and gain a wide application range. Consequently, it would be more desirable for practitioners to apply unsupervised models in effort-aware just-in-time (JIT) defect prediction if they can predict defect-inducing changes well. However, little is currently known on their prediction effectiveness in this context. We aim to investigate the predictive power of simple unsupervised models in effort-aware JIT defect prediction, especially compared with the state-of-the-art supervised models in the recent literature. We first use the most commonly used change metrics to build simple unsupervised models. Then, we compare these unsupervised models with the state-of-the-art supervised models under cross-validation, time-wise-cross-validation, and across-project prediction settings to determine whether they are of practical value. The experimental results, from open-source software systems, show that many simple unsupervised models perform better than the state-of-the-art supervised models in effort-aware JIT defect prediction.

Nowadays concurrent programming is in large demand. The inherent support for concurrency is therefore increasingly important in programming languages. As for C++, an abundance of standard concurrency constructs have been supported since C++11. However, to date there is little work investigating how these constructs are actually used in developing real software. In this paper, we perform an empirical study to investigate the adoption of C++ concurrency constructs in open-source applications, with the goal to provide insightful information for practitioners to use concurrency constructs efficiently. To this end, we analyze 127 open-source applications that adopt C++ concurrency constructs, comprising 34 million lines of C++ code, to conduct the experiment. The experimental results show that: (1) to implement concurrency code, thread-based constructs are significantly more often used than atomics-based constructs and task-based constructs; (2) to manage synchronization, lock-based constructs are significantly more often used than lock-free constructs and blocking constructs; (3) among the key thread-based constructs and task-based constructs (i.e. mutex, promise, and future), there is not a construct significantly more commonly misused than others; (4) small-size applications introduce concurrency constructs more intensively and more quickly than medium-size applications and large-size applications; and (5) an increasing use of standard concurrency constructs does not result in a substantially decreasing use of unstandardized concurrency constructs. Based on these findings, we make actionable suggestions for language designers, developers, and novices to assist them in designing and using C++ concurrency constructs.

The interactions between anti-patterns are claimed to affect maintenance. However, little work has been conducted to examine how anti-patterns interact. In this paper, we aim to investigate which pairs of anti-patterns tend to be coupled, i.e., interact with each other. We employ Fisher's exact test and Wilcoxon rank-sum test to identify coupled anti-patterns in the same class and coupled classes. Analyzing the relationships amongst 10 kinds of anti-patterns in five open-source projects, our results show that 1) several kinds of anti-patterns tend to be coupled, but some are conflicting, 2) the effect of anti-patterns on their dependent or co-changed ones are significant but small, 3) in ArgoUML, Xalan and Xerces-J, the classes infected with dependent anti-patterns are mostly (69.9% ~ 100%) modified in maintenance activities. Our findings offer empirical evidences for the existence of anti-pattern interactions, which provides valuable implications for practitioners and researchers.

In order to identify vulnerable software components, developers can take software metrics as predictors or use text mining techniques to build vulnerability prediction models. A recent study reported that text mining based models have higher recall than software metrics based models. However, this conclusion was drawn without considering the sizes of individual components which affects the code inspection effort to determine whether a component is vulnerable. In this paper, we investigate the predictive power of these two kinds of prediction models in the context of effort-aware vulnerability prediction. To this end, we use the same data sets, containing 223 vulnerabilities found in three web applications, to build vulnerability prediction models. The experimental results show that: (1) in the context of effort-aware ranking scenario, text mining based models only slightly outperform software metrics based models, (2) in the context of effort-aware classification scenario, text mining based models perform similarly to software metrics based models in most cases, and (3) most of the effect sizes (i.e. the magnitude of the differences) between these two kinds of models are trivial. These results suggest that, from the viewpoint of practical application, software metrics based models are comparable to text mining based models. Therefore, for developers, software metrics based models are practical choices for vulnerability prediction, as the cost to build and apply these models is much lower.

Software bug localization aiming to determine the locations needed to be fixed for a bug report is one of the most tedious and effort consuming activities in software debugging. Learning-to-rank (LR) is the state-of-the-art approach proposed by Ye et al. to recommending relevant files for bug localization. Ye et al.'s experimental results show that the LR approach significantly outperforms previous bug localization approaches in terms of "precision" and "accuracy". However, this evaluation does not take into account the influence of the size of the recommended files on the efficiency in detecting bugs. In practice, developers will generally spend more code inspection effort to detect bugs if larger files are recommended. In this paper, we use six large-scale open-source Java projects to evaluate the LR approach in the context of effort-aware bug localization. Our results, surprisingly, show that, when taking into account the code inspection effort to detect bugs, the LR approach is similar to or even worse than the standard VSM (Vector Space Model), a naïve IR-based bug localization approach.

C++ libraries provide an abundance of reusable components for writing high-quality programs and are thus widely adopted by software developers.However, to date there is little work investigating how these libraries are actually used in real software.In this paper, we perform an empirical study to investigate the adoption of C++ standard libraries in open-source applications, with the goal to provide actionable information for developers to help them employ libraries more efficiently.To this end, we analyze 379 historical revisions of 30 applications, containing 149 million lines of C++ code, to conduct the experiment.The experimental results show that: (1) three standard libraries (i.e.Containers Library, Utilities Library, and Strings Library) are significantly more often used than other libraries; (2) the new libraries of C++11 (i.e.Regular Expressions Library, Atomic Operations Library, and Thread Support Library) are significantly less often used than the formerlyestablished libraries; (3) the deprecated library constructs (i.e. auto pointers, function objects, and array I/O operations) are not used at a declining frequency; and (4) applications with a larger size do not adopt libraries more frequently.Based on these results, we propose four suggestions, which could help developers learn and use C++ libraries in an efficient way.

There has been a long debate on which programming language can help write better object-oriented programs.However, to date little response is given to this issue with empirical evidence.In this paper, we perform a comparative study on C++, C#, and Java programs by using object-oriented metrics, which comprise measures for class size, complexity, coupling, cohesion, inheritance, encapsulation, polymorphism, and reusability.Our experiment is conducted on 78 tasks in Rosetta Code, a code repository providing solutions to the same programming tasks in different languages.The experimental results show that: (1) C++ classes are significantly larger than C# and Java classes in size, but their complexity does not differ significantly; (2) C# classes are significantly more likely to be coupled than C++ and Java classes through inter-class method invocations instead of direct data access; (3) C# and Java classes tend to be more cohesive than C++ classes; (4) C# and Java significantly outperform C++ in building deep inheritance trees; and (5) programs written in C++, C#, and Java do not show a significant difference in class encapsulation, polymorphism, and reusability.These findings could help practitioners choose suitable languages to develop object-oriented systems. 1

Background. Slice-based cohesion metrics leverage program slices with respect to the output variables of a module to quantify the strength of functional relatedness of the elements within the module. Although slice-based cohesion metrics have been proposed for many years, few empirical studies have been conducted to examine their actual usefulness in predicting fault-proneness. Objective. We aim to provide an in-depth understanding of the ability of slice-based cohesion metrics in effort-aware post-release fault-proneness prediction, i.e. their effectiveness in helping practitioners find post-release faults when taking into account the effort needed to test or inspect the code. Method. We use the most commonly used code and process metrics, including size, structural complexity, Halstead's software science, and code churn metrics, as the baseline metrics. First, we employ principal component analysis to analyze the relationships between slice-based cohesion metrics and the baseline metrics. Then, we use univariate prediction models to investigate the correlations between slice-based cohesion metrics and post-release fault-proneness. Finally, we build multivariate prediction models to examine the effectiveness of slice-based cohesion metrics in effort-aware post-release fault-proneness prediction when used alone or used together with the baseline code and process metrics. Results. Based on open-source software systems, our results show that: 1) slice-based cohesion metrics are not redundant with respect to the baseline code and process metrics; 2) most slice-based cohesion metrics are significantly negatively related to post-release fault-proneness; 3) slice-based cohesion metrics in general do not outperform the baseline metrics when predicting post-release fault-proneness; and 4) when used with the baseline metrics together, however, slice-based cohesion metrics can produce a statistically significant and practically important improvement of the effectiveness in effort-aware post-release fault-proneness prediction. Conclusion. Slice-based cohesion metrics are complementary to the most commonly used code and process metrics and are of practical value in the context of effort-aware post-release fault-proneness prediction.

Python is widely used for web programming and GUI development. Due to the dynamic features of Python, Python programs may contain various unlimited errors. Dynamic slicing extracts those statements from a program which affect the variables in a slicing criterion with a particular input. Dynamic slicing of Python programs is essential for program debugging and fault location. In this paper, we propose an approach of dynamic slicing for Python programs which combines static analysis and dynamic tracing of the Python byte code. It precisely handles the dynamic features of Python, such as dynamic typing of variables, heavy usage of first-class objects, and dynamic modifications of classes and instances. Finally, we evaluate our approach on several Python programs. Experimental results show that the whole dynamic slicing for each subject program spends at most about 13 seconds on the average and costs at most 7.58 mb memory space overhead. Furthermore, the average slice ratio of Python source code ranges from 9.26% to 59.42%. According to it, our dynamic slicing approach can be effectively and efficiently performed. To the best of our knowledge, it is the first one of dynamic slicing for Python programs.

Background. The extent of the potentially confounding effect of class size in the fault prediction context is not clear, nor is the method to remove the potentially confounding effect, or the influence of this removal on the performance of fault-proneness prediction models.Objective. We aim to provide an in-depth understanding of the effect of class size on the true associations between object-oriented metrics and fault-proneness.Method. We first employ statistical methods to examine the extent of the potentially confounding effect of class size in the fault prediction context. After that, we propose a linear regression-based method to remove the potentially confounding effect. Finally, we empirically investigate whether this removal could improve the prediction performance of fault-proneness prediction models.Results. Based on open-source software systems, we found: (a) the confounding effect of class size on the associations between object-oriented metrics and fault-proneness in general exists; (b) the proposed linear regression-based method can effectively remove the confounding effect; and (c) after removing the confounding effect, the prediction performance of fault prediction models with respect to both ranking and classification can in general be significantly improved.Conclusion. We should remove the confounding effect of class size when building fault prediction models.

Coverage-based fault localization techniques leverage the coverage information to identify the faulty elements of a program. However, these techniques can be adversely affected by coincidental correctness, which occurs when the defect is executed but no failure is revealed. In this paper, we propose a clustering-based strategy to identify coincidental correctness in fault localization. The insight behind this strategy is that tests in the same cluster have similar behaviors. Thus a passed test in a cluster with many failed tests is highly possible to be coincidentally correct because it has the potential to execute the faulty elements as those failed ones do. We evaluated this technique from two aspects: the ability to identify coincidental correctness and the effectiveness to improve fault localization. The experimental results show that our strategy can alleviate the coincidental correctness problem and improve the effectiveness of fault localization.

Many feature subset selection (FSS) algorithms have been proposed, but not all of them are appropriate for a given feature selection problem. At the same time, so far there is rarely a good way to choose appropriate FSS algorithms for the problem at hand. Thus, FSS algorithm automatic recommendation is very important and practically useful. In this paper, a meta learning based FSS algorithm automatic recommendation method is presented. The proposed method first identifies the data sets that are most similar to the one at hand by the k-nearest neighbor classification algorithm, and the distances among these data sets are calculated based on the commonly-used data set characteristics. Then, it ranks all the candidate FSS algorithms according to their performance on these similar data sets, and chooses the algorithms with best performance as the appropriate ones. The performance of the candidate FSS algorithms is evaluated by a multi-criteria metric that takes into account not only the classification accuracy over the selected features, but also the runtime of feature selection and the number of selected features. The proposed recommendation method is extensively tested on 115 real world data sets with 22 well-known and frequently-used different FSS algorithms for five representative classifiers. The results show the effectiveness of our proposed FSS algorithm recommendation method.

Cluster filtering is a kind of test selection technique, which saves human efforts for result inspection by reducing test size and finding maximum failures. Cluster sampling strategies play a key role in the cluster filtering technique. A good sampling strategy can greatly improve the failure detection capability. In this paper, we propose a new cluster sampling strategy called execution-spectra-based sampling (ESBS). Different from the existing sampling strategies, ESBS iteratively selects test cases from each cluster. In each iteration process, ESBS selects the test case that has the maximum possibility to be a failed test. For each test, its suspiciousness is computed based on the execution spectra information of previous passed and failed test cases selected from the same cluster. The new sampling strategy ESBS is evaluated experimentally and the results show that it is more effective than existing sampling strategies in most cases.

For scalability, many side-effect analysis methods choose inclusion-based context-insensitive (IBCI) pointer analysis as their basis. However, such a pointer analysis is known to be imprecise, which often results in over-conservative side-effect sets. In this paper, we present a lightweight approach that exploits lazy access path resolving to improve the precision of side-effect analysis under IBCI pointer analysis. The approach partly represents and propagates side-effects in the access path form with the help of interstatement must aliases. All access paths can finally be resolved to the accessed locations, but during the side-effect propagation phase, an access path will never be resolved as long as it could be mapped to another access path in the caller. Since in inclusion-based points-to analysis, points-to sets of variables in the callers tend to be smaller than the ones in the callees, such lazy resolving mechanism can lead to more precision. The experimental results show that the lazy access path resolving approach is effective in improving the precision of IBCI pointer analysis based side-effect analysis methods.

Previous research shows that class size can influence the associations between object-oriented (OO) metrics and fault-proneness and therefore proposes that it should be controlled as a confounding variable when validating OO metrics on fault-proneness. Otherwise, their true associations may be distorted. However, it has not been determined whether this practice is equally applicable to other external quality attributes. In this paper, we use three size metrics, two of which are available during the high-level design phase, to examine the potentially confounding effect of class size on the associations between OO metrics and change-proneness. The OO metrics that are investigated include cohesion, coupling, and inheritance metrics. Our results, based on Eclipse, indicate that: 1) The confounding effect of class size on the associations between OO metrics and change-proneness, in general, exists, regardless of whichever size metric is used; 2) the confounding effect of class size generally leads to an overestimate of the associations between OO metrics and change-proneness; and 3) for many OO metrics, the confounding effect of class size completely accounts for their associations with change-proneness or results in a change of the direction of the associations. These results strongly suggest that studies validating OO metrics on change-proneness should also consider class size as a confounding variable.

Web site success is significantly associated with navigability, an important attribute of usability that denotes the ease with which users find desired information as they move through a Web site. Navigable Web sites allow users to form a mental model of the type and location of information in the Web site and an expectation of where and to what a particular hyperlink will lead. Existing navigability measures are based mainly on the static hyperlink structure of a Web site. Such measures, however, have two main drawbacks: 1) the effect on navigability of a hyperlink structure cannot be well characterized and 2) the effect on navigability of the navigation aids (such as the "Back" button provided by a browser) is ignored. In this paper, we abstract a dynamic Web surfing behavior as a Markov model which synthesizes typical surfing actions. Based on this model, we propose a novel navigability measure MNav. The experimental results show that MNav can be efficiently computed and it provides an effective and useful measurement of Web site navigability.

In the last decade, empirical studies on object-oriented design metrics have shown some of them to be useful for predicting the fault-proneness of classes in object-oriented software systems. This research did not, however, distinguish among faults according to the severity of impact. It would be valuable to know how object-oriented design metrics and class fault-proneness are related when fault severity is taken into account. In this paper, we use logistic regression and machine learning methods to empirically investigate the usefulness of object-oriented design metrics, specifically, a subset of the Chidamber and Kemerer suite, in predicting fault-proneness when taking fault severity into account. Our results, based on a public domain NASA data set, indicate that 1) most of these design metrics are statistically related to fault-proneness of classes across fault severity, and 2) the prediction capabilities of the investigated metrics greatly depend on the severity of faults. More specifically, these design metrics are able to predict low severity faults in fault-prone classes better than high severity faults in fault-prone classes.

A large number of cohesion measures based on method-attribute references have been proposed. However, virtually no attention has been paid to the abstract representation that objectively depicts the relationships among the members of a class. Specially, the flow dependence relationship among attributes, the indirect and potential dependence relationships among class members, and the direction of method-attribute references are ignored. To address this problem, we first identifies four types of basic dependence relationships and uses a class member dependence graph to represent all dependences among the members of a class. Then, a dependence relationships based measure for measuring the class cohesiveness is proposed. Finally, we compare our class cohesion measure with typical cohesion measures.

Abstract Chae et al. believe that the improved CBMC measures class cohesion from the viewpoint of the usage criteria of instance variables and that it cannot allow meaningful interpretations about classes. This paper exemplifies that it characterizes the interaction patterns better than the original one does and that it could also be used as a guideline for quality evaluation so as to enable the restructuring of poorly designed classes. Copyright © 2003 John Wiley & Sons, Ltd.

Classes are the basic modules in object-oriented (OO) software, which consist of attributes and methods. Thus, in an OO environment, cohesion mainly concerns how tight the attributes and methods of classes are. This paper discusses the relationships between attributes and attributes, attributes and methods, and methods and methods of a class based on dependence analysis. Then we discuss the properties of these relationships. According to these properties, this paper proposes a novel approach to measuring class cohesion. Our approach overcomes the limitations of previous class cohesion measures, which consider only one or two of the three relationships in a class. We also prove that this measure satisfies the properties that a good measurement should have.

Class cohesion could be used to evaluate the design quality of classes, to develop test measures for object-oriented software and to restructure poorly designed classes. Among a number of class cohesion measures proposed in the last decade, Chae's measure is based on the structure of the reference graph of a class, which overcomes the limitations of most class cohesion measures. However, it only considers the patterns of interactions among the members of a class partly and hence does not satisfy monotonicity, which may cause the measurement results to be inconsistent with intuition in some cases. This paper first analyzes the limitations of typical cohesion measures for classes in detail, and then proposes an improved cohesion measure ICBMC. Finally, this paper exemplifies the advantages and applications of ICBMC.

Abstract Although H. S. Chae's class cohesion measure considers not only the number of interactions, but also the patterns of the interactions among the constitute members of a class (which overcomes the limitations of previous class cohesion measures) it, however, only partly considers the patterns of interactions, and might cause the measuring results to be inconsistent with intuition in some cases. This paper discusses the demerits and proposes constructive amendments to Chae's cohesion measure. Copyright © 2001 John Wiley & Sons, Ltd.

Reengineering legacy systems written in conventional procedural languages to equivalent OO systems makes software more maintainable and reliable. In this paper a method for extracting objects from legacy Ada 83 systems using module features is proposed. First, we develop metrics to measure module cohesion. Then, effects on cohesion from changing module components are discussed and rules on how to extract inheritance relations among objects are given. Finally, an object identification tool in APAUS is described.