While Large Language Models (LLMs) have substantially improved the functional correctness of code translation, the critical dimension of execution efficiency remains overlooked.We present TRACE, the first benchmark to explicitly assess efficiency in LLM-translated code.TRACE includes 1,000 efficiency-critical tasks across C++, Java, and Python, each augmented with stress tests that reveal efficiency disparities often overlooked by small-scale tests.Using TRACE, we conduct an extensive evaluation of 28 representative LLMs and highlight several key insights: 1) Correctness and efficiency are often misaligned: the correctness leader Claude-Sonnet-4-Think achieves only moderate time efficiency, outperformed by smaller open-source LLMs such as Qwen2.5-Coder-14B-Instruct.2) Inefficiency is both prevalent and patterned: 23.5% of correct translations suffer from notable inefficiency, mainly arising from algorithm implementation discrepancy (11.9%), language construct mismatch (66.4%), and resource management inefficiency (21.7%).3) Inference-time prompt strategies bring only modest improvements, indicating that simple prompting alone is insufficient to improve translation efficiency.Together, our results establish execution efficiency as an essential dimension of code translation and position TRACE as a principled foundation for efficiency-oriented evaluation.Our

Smart contracts have revolutionized the way transactions are executed, offering decentralized and immutable frameworks. The immutability of smart contracts poses significant risks when vulnerabilities exist in their code, leading to financial losses. Despite advancements in using deep learning for smart contract vulnerability detection (SCVD), existing methods struggle with the complex logic and intricate semantics embedded within smart contract code. Large Language Models (LLMs) have shown promise in providing deeper insights into smart contract logic. However, LLMs, such as GPT follow a decoder-only architecture and are trained in an unsupervised manner rather than learning specific labels. In the SCVD task, these LLMs have difficulty in capturing information related to vulnerabilities, leading to very low accuracy. Therefore, we propose CodeXplain, a novel SCVD approach that leverages the deep insights into code from LLM and the supervised learning capabilities of deep learning models to set the latest advance and performance. In particular, we deeply analyze 14 types of dangerous and common smart contract vulnerabilities. Based on the rationale of these vulnerabilities, nine perspective prompts are introduced to guide LLMs in generating code explanations that contribute to SCVD. Then, we propose a CodeT5-based semantic fusion module integrating smart contract code and code explanations. Finally, the performance of SCVD is improved by performing supervised learning on trusted labels. Experimental results on 3,544 real-world smart contracts demonstrate that CodeXplain outperforms 16 state-of-the-art SCVD methods, achieving an F1-score of 94.12% and an accuracy of 93.88%, surpassing all baselines.

GUI test migration aims to produce test cases with events and assertions to test specific functionalities of a target app. Existing migration approaches typically focus on the widget-mapping paradigm that maps widgets from source apps to target apps. However, since different apps may implement the same functionality in different ways, direct mapping may result in incomplete or buggy test cases, thus significantly impacting the effectiveness of testing the target functionality and the practical applicability of migration approaches. In this article, we propose a new migration paradigm (i.e., the abstraction-concretization paradigm) that first abstracts the test logic for the target functionality and then utilizes this logic to generate the concrete GUI test case. Furthermore, we introduce MACdroid , the first approach that migrates GUI test cases based on this paradigm. Specifically, we propose an abstraction technique that utilizes source test cases from source apps targeting the same functionality to extract a general test logic for that functionality. Then, we propose a concretization technique that utilizes the general test logic to guide an LLM in generating the corresponding GUI test case (including events and assertions) for the target app. We evaluate MACdroid on two widely used datasets (including 31 apps, 34 functionalities, and 123 test cases). On the FrUITeR dataset, the test cases generated by MACdroid successfully test 64% of the target functionalities, improving the baselines by 191%. On the Lin dataset, MACdroid successfully tests 75% of the target functionalities, outperforming the baselines by 42%. These results underscore the effectiveness of MACdroid in GUI test migration.

Deep neural networks (DNNs) are one of the most effective methods available for detecting smart contract vulner-abilities (SCVs). The performance of current DNN approaches relies heavily on a large number of training samples and labels. The semi-supervised learning (SSL) trains the pseudo-labeling mechanism and performs label propagation on unlabeled data, thereby mitigating this problem. However, these approaches are not suitable for tasks related to Smart Contract Vulnerability Detection (SCVD) due to their limitations in capturing subtle faults, which are the primary causes of SCVD. We believe that subtle faults are contained in the correlation feature, which encompasses both commonalities among vulnerable contracts and differences between vulnerable and non-vulnerable contracts. Therefore, we propose a correlation-driven SSL method called Jupiter to solve this limitation. Specifically, Jupiter incorporates a contrastive learning module that conducts pairwise comparisons between smart contracts and captures correlation features. These correlation features facilitate the separation of feature distributions in vulnerable and non-vulnerable contracts. Then, a support vector machine with a built-in RBF kernel function is utilized to establish a decision boundary between the two types of contract distributions. We employ the distance from data to the decision boundary as a confidence score and propagate high-quality pseudo-labels to each unlabeled data surpassing the threshold. This process is iterated, where the labeled dataset is constructed using a combination of pseudo-labeled and reallabeled data to be re-input into the contrastive learning model until all data is labeled. Finally, the combination of semantic information and correlation features accurately detects SCVs.By conducting an empirical evaluation on a large-scale realworld dataset comprising over 40,000 smart contracts, we compare the performance of 6 state-of-the-art SSL methods and 5 state-of-the-art SCVD methods. Our findings demonstrate the effectiveness of our proposed method, Jupiter, in two key aspects: (1) Jupiter achieves optimal performance across all SSL methods, outperforming them by 18.89% to 28.42% in terms of F1- score; (2) Current state-of-the-art SCVD methods fail to deliver satisfactory results when only small amounts of labeled data are available. Specifically, these methods achieve F1-scores ranging from 49.72% to 50.98% when utilizing just 10% of the labeled data. In contrast, under the same conditions, Jupiter outperforms all baselines, achieving an F1-score of 89.28%, which represents an improvement of 75.12% to 79.56%.

Modern software development often introduces bug inducing commits (BICs) that can degrade performance or cause crashes. Swift localization of BICs is crucial but challenging due to the entanglement among multiple kinds of information overlooked by existing methods that treat these elements independently. Understanding this entanglement is promising but faces two key challenges: (1) the entanglement representation problem, since simply concatenating diverse data types fails to capture their entanglement effectively; (2) the large input size problem, as identifying BICs requires analyzing a vast set of commits, making simultaneous processing infeasible. To address these challenges, we propose BICSleuth, a framework that encodes the entanglement for effective BIC localization through three stages. First, a small-model-based ranker efficiently narrows down commits despite large input sizes. Second, an LLM-based discriminator deepens the understanding of the entanglement through selective information integration. Third, a reranking strategy combines insights from both stages to enhance localization accuracy. Evaluated on a BIC dataset constructed from Defects4J v2.0.0, BICSleuth outperforms four state-of-the-art approaches, achieving 148.9% of the Mean Reciprocal Rank compared to the best spectrum-based baseline and 507.1% of the MRR of the best IR-based method. Additionally, BICSleuth ranks the BIC first in 70.0% of projects and within the top five in 84.6%. The results demonstrate that BICSleuth effectively leverages the entanglement for BIC localization, with all stages contributing to its success.

Formal verification is a crucial means to assure software quality. Regrettably, the manual composition of verification scripts proves to be both laborious and time-consuming. In response, researchers have put forth automated theorem prover approaches; however, these approaches still grapple with several limitations. These limitations encompass insufficient handling of lengthy proof steps, difficulty in aligning the various components of a Coq program with the requirements and constraints of the proof goal, and inefficiencies. To surmount these limitations, we present Gpass, a goal-adaptive neural theorem prover based on deep learning technology. Firstly, we design a unique sequence encoder for Gpass that completely scans previous proof tactics through multiple sliding windows and provides information related to the current proof step. Secondly, Gpass incorporates a goal-adaptive feature integration module to align the reasoning process with the requirements of the proof goal. Finally, we devise a parameter selection method based on loss values and loss slopes to procure parameter sets with diverse distributions, thereby facilitating the exploration of various proof tactics. Experimental results demonstrate that Gpass attains better performance on the extensive CoqGym benchmark and proves 11.03%-96.37% more theorems than the prior work most closely related to ours. We find that the orthogonality between Gpass and CoqHammer proves their complementary capabilities, and together they prove a total of 3,774 theorems, which is state-of-the-art performance. In addition, we propose an efficiency optimisation approach that allows Gpass to achieve performance beyond Diva at one-sixth of the parameter sets.

C++ is a system-level programming language for modern software development, which supports multiple programming paradigms, including object-oriented, generic, and functional programming. The intrinsic complexity of these paradigms and their interactions grants C++ powerful expressiveness while posing significant challenges for compilers in correctly implementing its type system. A type system encompasses various aspects such as type inference, type checking, subtyping, type conversions, generics, scoping, and binding. However, systematic testing of the type systems of C++ compilers remains largely underexplored in existing studies. In this work, we present TyMut, the first approach specifically designed to test the C++ type system. TyMut is a mutation-based compiler fuzzer equipped with advanced type-driven mutation operators, carefully crafted to target intricate type-related features such as template generics, type conversions, and inheritance. Beyond differential testing, TyMut introduces enhanced test oracles through a must analysis that partially confirms the validity of generated programs. Specifically, mutation operators are classified into well-formed and not-well-formed : Programs generated by well-formed mutation operators are valid and must be accepted by compilers. Programs generated by not-well-formed operators are validated against a set of well-formedness rules . Any violation indicates the program is invalid and must be rejected. For programs that pass the rules but lack a definitive oracle, TyMut applies differential testing to identify behavioral inconsistencies across compilers. The testing campaign took about 32 hours to generate and test 250584 programs. The must analysis provides definite test oracles for nearly 80% of all generated programs. TyMut uncovered 102 bugs in the recent versions of GCC and Clang, with 56 confirmed as new bugs by compiler developers. Among the confirmed bugs, 26 of them cause compiler crashes, and more than 50% cause miscompilation. Additionally, 7 of them had remained hidden for over 20 years, 22 for over 10 years, and 39 for over 5 years. One long-standing bug discovered by TyMut was later confirmed as the root cause of a real-world issue in TensorFlow. Before submitting this paper, 13 bugs were fixed, most of which were fixed within 60 days. Notably, some unconfirmed bugs have led to in-depth discussions among developers. For instance, one bug led a compiler developer to submit a new issue to the C++ language standard, showing that we uncovered ambiguities in the language specification.

Commit messages are important for developers to understand the content and the reason for code changes. However, poor and even empty commit messages widely exist. To improve the quality of commit messages and development efficiency, many commit message generation methods have been proposed. Nevertheless, previous methods mainly focus on a brief generation problem, where both the input code change and the output commit messages are restricted to short. This may initiate a debate on the performance of these methods in practice. In this article, we attempt to remove the restrictions and move the needle forward to a holistic commit message generation problem. In particular, we conduct experiments to evaluate the performance of existing commit message generation methods in holistic commit message generation. In the experiments, we choose seven state-of-the-art commit generation methods and focus on two important scenarios in commit message generation (i.e., the within-project scenario and the cross-project scenario). To conduct our experiments, we publish a holistic commit message dataset HORDA with test data manually labeled. In our evaluations, we find that in generating holistic commit messages, the IR-based method has a better performance than non-pre-trained generation-based methods in the within-project scenario, contradicting previous research findings. Further, while the pre-trained generation-based methods are better than non-pre-trained generation-based methods, they are still constrained by the limitations of generation models.

Prevalent Fault Localization (FL) techniques rely on tests to localize buggy program elements. Tests could be treated as fuel to further boost FL by providing more debugging information. Therefore, it is highly valuable to measure the Fault Diagnosis Capability (FDC) of a test for diagnosing faults, so as to select or generate tests to better help FL (i.e., FL-oriented test selection or FL-oriented test generation). To this end, researchers have proposed many FDC metrics, which serve as the selection criterion in FL-oriented test selection or the fitness function in FL-oriented test generation. Existing FDC metrics can be classified into result-agnostic and result-aware metrics depending on whether they take test results (i.e., passing or failing) as input. Although result-aware metrics perform better in test selection, they have restricted applications due to the input of test results, e.g., they cannot be applied to guide test generation. Moreover, all the existing FDC metrics are designed based on some pre-defined heuristics and have achieved limited FL performance due to their inaccuracy. To address these issues, in this article, we reconsider result-agnostic metrics (i.e., metrics that do not take test results as input), and propose a novel result-agnostic metric RLFDC which predicts FDC values of tests through reinforcement learning. In particular, we treat FL results as reward signals, and train an FDC prediction model with the direct FL feedback to automatically learn a more accurate measurement rather than design one based on pre-defined heuristics. Finally, we evaluate the proposed RLFDC on Defects4J by applying the studied metrics to test selection and generation. According to the experimental results, the proposed RLFDC outperforms all the result-agnostic metrics in both test selection and generation, e.g., when applied to selecting human-written tests, RLFDC achieves 28.2% and 21.6% higher acc@1 and mAP values compared to the state-of-the-art result-agnostic metric TfD. Besides, RLFDC even achieves competitive performance compared to the state-of-the-art result-aware metric FDG in test selection.

Currently, smart contract vulnerabilities (SCVs) have emerged as a major factor threatening the transaction security of blockchain. Existing state-of-the-art methods rely on deep learning to mitigate this threat. They treat each input contract as an independent entity and feed it into a deep learning model to learn vulnerability patterns by fitting vulnerability labels. It is a pity that they disregard the correlation between contracts, failing to consider the commonalities between contracts of the same type and the differences among contracts of different types. As a result, the performance of these methods falls short of the desired level.

GUI test case migration is to migrate GUI test cases from a source app to a target app. The key of test case migration is widget matching. Recently, researchers have proposed various approaches by formulating widget matching as a matching task. However, since these matching approaches depend on static word embeddings without using contextual information to represent widgets and manually formulated matching functions, there are main limitations of these matching approaches when handling complex matching relations in apps. To address the limitations, we propose the first learning-based widget matching approach named TEMdroid (TEst Migration) for test case migration. Unlike the existing approaches, TEMdroid uses BERT to capture contextual information and learns a matching model to match widgets. Additionally, to balance the significant imbalance between positive and negative samples in apps, we design a two-stage training strategy where we first train a hard-negative sample miner to mine hard-negative samples, and further train a matching model using positive samples and mined hard-negative samples. Our evaluation on 34 apps shows that TEM-droid is effective in event matching (i.e., widget matching and target event synthesis) and test case migration. For event matching, TEM-droid's Top1 accuracy is 76%, improving over 17% compared to baselines. For test case migration, TEMdroid's F1 score is 89%, also 7% improvement compared to the baseline approach.

In Continuous Integration (CI), accurate build prediction is crucial for minimizing development costs and enhancing efficiency. However, existing build prediction methods, typically based on predefined rules or machine learning classifiers employing feature engineering, have been constrained by their limited ability to fully capture the intricate details of commit artifacts, such as code change and commit messages. These artifacts are critical for understanding the commit under a build but have been inadequately utilized in existing approaches. To address this problem, we propose GitSense, a Transformer-based model specifically designed to incorporate the rich and complex information contained within commit artifacts for the first. GitSense employs an advanced textual encoder with built-in sliding window text samplers for textual features and a statistical feature encoder for extracted statistical features. This innovative approach allows for a comprehensive analysis of lengthy and intricate commit artifacts, surpassing the capabilities of traditional methods. We conduct comprehensive experiments to compare GitSense with five state-of-the-art build prediction models, Longformer, and ChatGPT. The experimental results show that GitSense outperforms these models in predicting failed builds, evidenced by 32.7%-872.1.0% better on F1-score, 23.9%-437.5% better on Precision, and 40.2%-1396.0% better on Recall.

GUI test case migration is the process of migrating GUI test cases from a source app to a target app for a specific functionality. However, test cases obtained via existing migration approaches can hardly be directly used to test target functionalities and typically require additional manual modifications. This problem may significantly impact the effectiveness of testing target functionalities and the practical applicability of migration approaches. In this paper, we propose MigratePro, the first approach to enhancing GUI test case migration via synthesizing a new test case based on multiple test cases for the same functionality migrated from various source apps to the target app. The aim of MigratePro is to produce functional test cases with less human intervention. Specifically, given multiple migrated test cases for the same functionality in the target app, MigratePro first combines all the GUI states related to these migrated test cases into an overall state-sequence. Then, MigratePro organizes events and assertions from migrated test cases according to the overall state-sequence and endeavors to remove the should-be-removed events and assertions, while also incorporating some connection events in order to make the should-be-included events and assertions executable. Our evaluation on 30 apps, 34 functionalities, and 127 test cases shows that MigratePro improves the capability of three representative migration approaches (i.e., Craftdroid, AppFlow, ATM), successfully improving testing the target functionalities by 86%, 333%, and 300%, respectively. These results underscore the generalizability of MigratePro for effectively enhancing migration approaches.

Collaborative software development significantly enhances development productivity by enabling multiple contributors to work concurrently on different branches. Despite these advantages, such collaboration often increases the likelihood of causing conflicts. Resolving these conflicts brings huge challenges, primarily due to the necessity of comprehending the differences between conflicting versions. Researchers have explored various automatic conflict resolution techniques, including unstructured, structured, and learning-based approaches. However, these techniques are mostly heuristic-based or black-box in nature, which means they do not attempt to solve the root cause of the conflicts, i.e., the existence of different program behaviors exhibited by the conflicting versions.

Compilers are one of the most fundamental software systems. A large number of software systems rely on compilers for execution. Compiler bugs can significantly hinder software developers from diagnosing issues within their software. Therefore, it is essential to ensure the correctness of compilers and to isolate and fix compiler bugs. Isolating bugs within compilers is challenging due to compilers' complexity and large codebase. The prior studies on compiler bug isolation struggle to generate sufficient test cases for bug isolation and are not effective enough.

Deep code models (DCMs) have achieved impressive accomplishments and have been widely applied to various code-related tasks. However, existing studies show that some DCMs have poor robustness, and even small noise in the input data can lead to erroneous outputs. This phenomenon can seriously hinder the application of these DCMs in real-world scenarios. To address this limitation, we propose MARVEL, a mutual learning-based framework for enhancing the robustness of DCMs via adversarial training. Specifically, MARVEL initializes two identical DCMs, one of which receives Gaussian-distorted data and performs adversarial training, and the other receives the clean data. Then these two DCMs work together to not only fit the true labels but also fit each other's internal parameters. Our intuition is that the DCM can enhance robustness by training noisy data, while the DCM achieves accurate prediction performance by learn the clean data. Their mutual learning enables the DCM to balance both robustness and predictive performance.

Mutation testing is widely used to measure the test adequacy of a project. Despite its popularity, mutation testing is time-consuming and extremely expensive. To mitigate this problem, researchers propose Predictive Mutation Testing (PMT). Existing PMT approaches build classification models based on statistical program features or source code of programs to predict mutation testing results. Previous statistical feature-based PMT models need expensive overhead to collect dynamic features and neglect the rich information inherent in code text. Previous text-based PMT models extract essential code elements as input and outperform the feature-based models. However, they encode code text in a plain way. Therefore, they cannot sensitively capture subtle differences in mutants and they have difficulty in capturing the correlation between mutants and tests. To address these challenges, we propose a new model, SODA. SODA uses a new learning strategy, Mutational Semantic Learning, to make our model spot code mutation and its impact on test behavior. In particular, we employ a new sampling strategy to reinforce the corresponding relationship between mutants and tests by sampling same-mutant contrastive groups. Then we employ contrastive learning to make our model capture subtle differences in mutants. We conduct experiments to investigate the performance of SODA. The results demonstrate that both in the cross-project and cross-version scenarios, SODA achieves state-of-the-art classification performance (improves upon baselines by 5.32%-114.92% in kill-F1 score, 0.04%-25.54% in survive-F1 score, 4.25%-60.43% in accuracy) and has the lowest mutation score error.

In recent years, Automated Program Repair (APR) has been extensively studied in academia and even drawn wide attention from the industry. However, APR techniques can be extremely time consuming since (1) a large number of patches can be generated for a given bug, and (2) each patch needs to be executed on the original tests to ensure its correctness. In the literature, various techniques (e.g., based on learning, mining, and constraint solving) have been proposed/studied to reduce the number of patches. Intuitively, every patch can be treated as a software revision during regression testing; thus, traditional Regression Test Selection (RTS) techniques can be leveraged to only execute the tests affected by each patch (as the other tests would keep the same outcomes) to further reduce patch execution time. However, few APR systems actually adopt RTS and there is still a lack of systematic studies demonstrating the benefits of RTS and the impact of different RTS strategies on APR. To this end, this article presents the first extensive study of widely used RTS techniques at different levels (i.e., class/method/statement levels) for 12 state-of-the-art APR systems on over 2M patches. Our study reveals various practical guidelines for bridging the gap between APR and regression testing, including: (1) the number of patches widely used for measuring APR efficiency can incur skewed conclusions, and the use of inconsistent RTS configurations can further skew the conclusions; (2) all studied RTS techniques can substantially improve APR efficiency and should be considered in future APR work; (3) method- and statement-level RTS outperform class-level RTS substantially and should be preferred; (4) RTS techniques can substantially outperform state-of-the-art test prioritization techniques for APR, and combining them can further improve APR efficiency; and (5) traditional Regression Test Prioritization (RTP) widely studied in regression testing performs even better than APR-specific test prioritization when combined with most RTS techniques. Furthermore, we also present the detailed impact of different patch categories and patch validation strategies on our findings.

Machine translation is integral to international communication and extensively employed in diverse human-related applications. Despite remarkable progress, fairness issues persist within current machine translation systems. In this article, we propose FairMT, an automated fairness testing approach tailored for machine translation systems. FairMT operates on the assumption that translations of semantically similar sentences, containing protected attributes from distinct demographic groups, should maintain comparable meanings. It comprises three key steps: (1) test input generation, producing inputs covering various demographic groups; (2) test oracle generation, identifying potential unfair translations based on semantic similarity measurements; and (3) regression, discerning genuine fairness issues from those caused by low-quality translation. Leveraging FairMT, we conduct an empirical study on three leading machine translation systems–Google Translate, T5, and Transformer. Our investigation uncovers up to 832, 1,984, and 2,627 unfair translations across the three systems, respectively. Intriguingly, we observe that fair translations tend to exhibit superior translation performance, challenging the conventional wisdom of a fairness-performance tradeoff prevalent in the fairness literature.

Widely used compilers like GCC and LLVM usually have hundreds of optimizations controlled by optimization flags, which are enabled or disabled during compilation to improve the runtime performance (e.g., small execution time) of the compiler program. Due to the large number of optimization flags and their combination, it is difficult for compiler users to manually tune compiler optimization flags. In the literature, a number of autotuning techniques have been proposed, which tune optimization flags for a compiled program by comparing its actual runtime performance with different optimization flag combinations. Due to the huge search space and heavy actual runtime cost, these techniques suffer from the widely recognized efficiency problem. To reduce the heavy runtime cost, in this article we propose a lightweight learning approach that uses a small number of actual runtime performance data to predict the runtime performance of a compiled program with various optimization flag combinations. Furthermore, to reduce the search space, we design a novel particle swarm algorithm that tunes compiler optimization flags with the prediction model. To evaluate the performance of the proposed approach, CompTuner, we conduct an extensive experimental study on two popular C compilers, GCC and LLVM, with two widely used benchmarks, cBench and PolyBench. The experimental results show that CompTuner significantly outperforms the six compared techniques, including the state-of-the-art technique BOCA.

Commit messages summarize code changes and help developers understand the intention. To alleviate human efforts in writing commit messages, researchers have proposed various automated commit message generation techniques, among which learning-based techniques have achieved great success in recent years. However, existing evaluation on learning-based commit message generation relies on the automatic metrics (e.g., BLEU) widely used in natural language processing (NLP) tasks, which are aggregated scores calculated based on the similarity between generated commit messages and the ground truth. Therefore, it remains unclear what generated commit messages look like and what kind of commit messages could be precisely generated by existing learning-based techniques. To fill this knowledge gap, this work performs the first study to systematically investigate the detailed commit messages generated by learning-based techniques. In particular, we first investigate the frequent patterns of the commit messages generated by state-of-the-art learning-based techniques. Surprisingly, we find the majority (~90%) of their generated commit messages belong to simple patterns (i.e., addition/removal/fix/avoidance patterns). To further explore the reasons, we then study the impact of datasets, input representations, and model components. We surprisingly find that existing learning-based techniques have competitive performance even when the inputs are only represented by change marks (i.e., “+”/“-”/“ ”), It indicates that existing learning-based techniques poorly utilize syntax and semantics in the code while mostly focusing on change marks, which could be the major reason for generating so many pattern-matching commit messages. We also find that the pattern ratio in the training set might also positively affect the pattern ratio of generated commit messages; and model components might have different impact on the pattern ratio.

To alleviate the cost of regression testing in continuous integration (CI), a large number of machine learning-based (ML-based) test case prioritization techniques have been proposed. However, it is yet unknown how they perform under the same experimental setup, because they are evaluated on different datasets with different metrics. To bridge this gap, we conduct the first comprehensive study on these ML-based techniques in this paper. We investigate the performance of 11 representative ML-based prioritization techniques for CI on 11 open-source subjects and obtain a series of findings. For example, the performance of the techniques changes across CI cycles, mainly resulting from the changing amount of training data, instead of code evolution and test removal/addition. Based on the findings, we give some actionable suggestions on enhancing the effectiveness of ML-based techniques, e.g., pretraining a prioritization technique with cross-subject data to get it thoroughly trained and then finetuning it with within-subject data dramatically improves its performance. In particular, the pretrained MART achieves state-of-the-art performance, producing the optimal sequence on 80% subjects, while the existing best technique, the original MART, only produces the optimal sequence on 50% subjects.

Collaborative development is critical to improve the productivity. Multiple contributors work simultaneously on the same project and might make changes to the same code locations. This can cause conflicts and require manual intervention from developers to resolve them. To alleviate the human efforts of manual conflict resolution, researchers have proposed various automatic techniques. More recently, deep learning models have been adopted to solve this problem and achieved state-of-the-art performance. However, these techniques leverage classification to combine the existing elements of input. The classification- based models cannot generate new tokens or produce flexible combinations, and have a wrong hypothesis that fine-grained conflicts of one single coarse-grained conflict are independent. In this work, we propose to generate the resolutions of merge conflicts from a totally new perspective, that is, generation, and we present a conflict resolution technique, MergeGen. First, we design a structural and fine-grained conflict-aware representation for the merge conflicts. Then, we propose to leverage an encoder- decoder-based generative model to process the designed conflict representation and generate the resolutions auto-regressively. We further perform a comprehensive study to evaluate the effectiveness of MergeGen. The quantitative results show that MergeGen outperforms the state-of-the-art (SOTA) techniques from both precision and accuracy. Our evaluation on multiple programming languages verifies the good generalization ability of MergeGen. In addition, the ablation study shows that the major component of our technique makes a positive contribution to the performance of MergeGen, and the granularity analysis reveals the high tolerance of MergeGen to coarse-grained conflicts. Moreover, the analysis on generating new tokens further proves the advance of generative models.

Widely used compilers like GCC usually have hundreds of optimizations controlled by optimization flags, which can be enabled or disabled during compilation to improve the runtime performance of a compiled program. Due to the large number of optimization flags and their combination, it is difficult for compiler users to tune compiler optimization flags manually. In the literature, many auto-tuning techniques have been proposed, which find a desired setting on all optimization flags (i.e., an optimization sequence) by designing different search strategies in the entire optimization space. Due to the huge search space, these techniques suffer from the widely-recognized efficiency problem. To reduce the search space, in this paper, we propose a critical-flag selection based approach CFSCA which first finds flags potentially relevant to the target program by analyzing program structure and compiler documentation, and then identifies critical flags through statistical analysis on the program's predicted runtime performance with various optimization sequences. With the reduced search space, CFSCA selects a desired optimization sequence. To evaluate the performance of the proposed approach CFSCA, we conduct an extensive experimental study on the latest version of the compiler GCC with a widely used benchmark cBench. The experimental results show that CFSCA significantly outperforms the four compared techniques, including the state-of-art technique BOCA.

Commit messages summarize code changes of each commit in natural language, which help developers understand code changes without digging into detailed implementations and play an essential role in comprehending software evolution. To alleviate human efforts in writing commit messages, researchers have proposed various automated techniques to generate commit messages, including template-based, information retrieval-based, and learning-based techniques. Although promising, previous techniques have limited effectiveness due to their coarse-grained code change representations.

Unit testing could be used to validate the correctness of basic units of the software system under test. To reduce manual efforts in conducting unit testing, the research community has contributed with tools that automatically generate unit test cases, including test inputs and test oracles (e.g., assertions). Recently, ATLAS, a deep learning (DL) based approach, was proposed to generate assertions for a unit test based on other already written unit tests. Despite promising, the effectiveness of ATLAS is still limited. To improve the effectiveness, in this work, we make the first attempt to leverage Information Retrieval (IR) in assertion generation and propose an IR-based approach, including the technique of IR-based assertion retrieval and the technique of retrieved-assertion adaptation. In addition, we propose an integration approach to combine our IR-based approach with a DL-based approach (e.g., ATLAS) to further improve the effectiveness. Our experimental results show that our IR-based approach outperforms the state-of-the-art DL-based approach, and integrating our IR-based approach with the DL-based approach can further achieve higher accuracy. Our results convey an important message that information retrieval could be competitive and worthwhile to pursue for software engineering tasks such as assertion generation, and should be seriously considered by the research community given that in recent years deep learning solutions have been over-popularly adopted by the research community for software engineering tasks.

Fault localization is to identify faulty program elements. Among the large number of fault localization approaches in the literature, coverage-based fault localization, especially spectrum-based fault localization has been intensively studied due to its effectiveness and lightweightness. Despite the rich literature, almost all existing fault localization approaches and studies are conducted on imperative programming languages such as Java and C, leaving a gap in other programming paradigms. In this paper, we aim to study fault localization approaches for the functional programming paradigm, using Haskell language as a representation. We build up the first dataset on real Haskell projects including both real and seeded faults, which enables the research of fault localization for functional languages. With this dataset, we explore fault localization techniques for Haskell. In particular, as typically for SBFL approaches, we study methods for coverage collection as well as formulae for suspiciousness scores computation, and carefully adapt these two components to Haskell considering the language features and characteristics, resulting in a series of adaption approaches and a learning-based approach, which are evaluated on the dataset to demonstrate the promises of the direction.

Recently, Python has adopted optional type annotation to support type checking and program documentation. However, to enjoy the benefits, developers have to manually write type annotations, which is recognized to be a time-consuming task. To alleviate human efforts on manual type annotation, machine-learning-based approaches have been proposed to recommend types based on code features. However, they suffer from the correctness problem, i.e., the recommended types cannot pass type checking. To address the correctness problem of the machine-learning-based approaches, in this paper, we present a static type recommendation approach, named Stray. Stray can recommend types correctly. We evaluate Stray by comparing it against four state-of-art type recommendation approaches, and find that Stray outperforms these baselines by over 30% absolute improvement in both precision and recall.

Although regression testing is important to guarantee the software quality in software evolution, it suffers from the widely known cost problem. To address this problem, existing researchers made dedicated efforts on test prioritization, which optimizes the execution order of tests to detect faults earlier; while practitioners in industry leveraged more computing resources to save the time cost of regression testing. By combining these two orthogonal solutions, in this article, we define the problem of parallel test prioritization, which is to conduct test prioritization in the scenario of parallel test execution to reduce the cost of regression testing. Different from traditional sequential test prioritization, parallel test prioritization aims at generating a set of test sequences, each of which is allocated in an individual computing resource and executed in parallel. In particular, we propose eight parallel test prioritization techniques by adapting the existing four sequential test prioritization techniques, by including and excluding testing time in prioritization. To investigate the performance of the eight parallel test prioritization techniques, we conducted an extensive study on 54 open-source projects and a case study on 16 commercial projects from Baidu , a famous search service provider with 600M monthly active users. According to the two studies, parallel test prioritization does improve the efficiency of regression testing, and cost-aware additional parallel test prioritization technique significantly outperforms the other techniques, indicating that this technique is a good choice for practical parallel testing. Besides, we also investigated the influence of two external factors, the number of computing resources and time allowed for parallel testing, and find that more computing resources indeed improve the performance of parallel test prioritization. In addition, we investigated the influence of two more factors, test granularity and coverage criterion, and find that parallel test prioritization can still accelerate regression testing in parallel scenario. Moreover, we investigated the benefit of parallel test prioritization on the regression testing process of continuous integration, considering both the cumulative acceleration performance and the overhead of prioritization techniques, and the results demonstrate the superiority of parallel test prioritization.

In recent years, many test case prioritization (TCP) techniques have been proposed to speed up the process of fault detection. However, little work has taken the efficiency problem of these techniques into account. In this paper, we target the Greedy Additional (GA) algorithm, which has been widely recognized to be effective but less efficient, and try to improve its efficiency while preserving effectiveness. In our Accelerated GA (AGA) algorithm, we use some extra data structures to reduce redundant data accesses in the GA algorithm and thus the time complexity is reduced from O(m2n) to O(kmn) when n > m, where m is the number of test cases, n is the number of program elements, and k is the iteration number. Moreover, we observe the impact of iteration numbers on prioritization efficiency on our dataset and propose to use a specific iteration number in the AGA algorithm to further improve the efficiency. We conducted experiments on 55 open-source subjects. In particular, we implemented each TCP algorithm with two kinds of widely-used input formats, adjacency matrix and adjacency list. Since a TCP algorithm with adjacency matrix is less efficient than the algorithm with adjacency list, the result analysis is mainly conducted based on TCP algorithms with adjacency list. The results show that AGA achieves 5.95X speedup ratio over GA on average, while it achieves the same average effectiveness as GA in terms of Average Percentage of Fault Detected (APFD). Moreover, we conducted an industrial case study on 22 subjects, collected from Baidu, and find that the average speedup ratio of AGA over GA is 44.27X, which indicates the practical usage of AGA in real-world scenarios.

Coverage-based fault localization has been extensively studied in the literature due to its effectiveness and lightweightness for real-world systems. However, existing techniques often utilize coverage in an oversimplified way by abstracting detailed coverage into numbers of tests or boolean vectors, thus limiting their effectiveness in practice. In this work, we present a novel coverage-based fault localization technique, GRACE, which fully utilizes detailed coverage information with graph-based representation learning. Our intuition is that coverage can be regarded as connective relationships between tests and program entities, which can be inherently and integrally represented by a graph structure: with tests and program entities as nodes, while with coverage and code structures as edges. Therefore, we first propose a novel graph-based representation to reserve all detailed coverage information and fine-grained code structures into one graph. Then we leverage Gated Graph Neural Network to learn valuable features from the graph-based coverage representation and rank program entities in a listwise way. Our evaluation on the widely used benchmark Defects4J (V1.2.0) shows that GRACE significantly outperforms state-of-the-art coverage-based fault localization: GRACE localizes 195 bugs within Top-1 whereas the best compared technique can at most localize 166 bugs within Top-1. We further investigate the impact of each GRACE component and find that they all positively contribute to GRACE. In addition, our results also demonstrate that GRACE has learnt essential features from coverage, which are complementary to various information used in existing learning-based fault localization. Finally, we evaluate GRACE in the cross-project prediction scenario on extra 226 bugs from Defects4J (V2.0.0), and find that GRACE consistently outperforms state-of-the-art coverage-based techniques.

Compilers are one of the most fundamental software systems. Compiler testing is important for assuring the quality of compilers. Due to the crucial role of compilers, they have to be well tested. Therefore, automated compiler testing techniques (those based on randomly generated programs) tend to run a large number of test programs (which are test inputs of compilers). The cost for compilation and execution for these test programs is significant. These techniques can take a long period of testing time to detect a relatively small number of compiler bugs. That may cause many practical problems, e.g., bringing a lot of costs including time costs and financial costs, and delaying the development/release cycle. Recently, some approaches have been proposed to accelerate compiler testing by executing test programs that are more likely to trigger compiler bugs earlier according to some criteria. However, these approaches ignore an important aspect in compiler testing: different test programs may have similar test capabilities (i.e., testing similar functionalities of a compiler, even detecting the same compiler bug), which may largely discount their acceleration effectiveness if the test programs with similar test capabilities are executed all the time. Test coverage is a proper approximation to help distinguish them, but collecting coverage dynamically is infeasible in compiler testing since most test programs are generated on the fly by automatic test-generation tools like Csmith. In this paper, we propose the first method to predict test coverage statically for compilers, and then propose to prioritize test programs by clustering them according to the predicted coverage information. The novel approach to accelerating compiler testing through coverage prediction is called COP (short for COverage Prediction). Our evaluation on GCC and LLVM demonstrates that COP significantly accelerates compiler testing, achieving an average of 51.01 percent speedup in test execution time on an existing dataset including three old release versions of the compilers and achieving an average of 68.74 percent speedup on a new dataset including 12 latest release versions. Moreover, COP outperforms the state-of-the-art acceleration approach significantly by improving$17.16\%\sim 82.51\%$speedups in different settings on average.

Spectrum-based fault localization (SBFL) techniques are widely studied and have been evaluated to be effective in locating faults. Recent studies also showed that developers from industry value automated SBFL techniques. However, their effectiveness is still limited by two main reasons. First, the test coverage information leveraged to construct the spectrum does not reflect the root cause directly. Second, SBFL suffers from the tie issue so that the buggy code entities can not be well differentiated from non-buggy ones. To address these challenges, we propose to leverage the information of version histories in fault localization based on the following two intuitions. First, version histories record how bugs are introduced to software projects and this information reflects the root cause of bugs directly. Second, the evolution histories of code can help differentiate those suspicious code entities ranked in tie by SBFL. Our intuitions are also inspired by the observations on debugging practices from large open source projects and industry. Based on the intuitions, we propose a novel technique HSFL (historical spectrum based fault localization). Specifically, HSFL identifies bug-inducing commits from the version history in the first step. It then constructs historical spectrum (denoted as Histrum) based on bug-inducing commits, which is another dimension of spectrum orthogonal to the coverage based spectrum used in SBFL. HSFL finally ranks the suspicious code elements based on our proposed Histrum and the conventional spectrum. HSFL outperforms the state-of-the-art SBFL techniques significantly on the Defects4J benchmark. Specifically, it locates and ranks the buggy statement at Top-1 for 77.8 percent more bugs as compared with SBFL, and 33.9 percent more bugs at Top-5. Besides, for the metrics MAP and MRR, HSFL achieves an average improvement of 28.3 and 40.8 percent over all bugs, respectively. Moreover, HSFL can also outperform other six families of fault localization techniques, and our proposed Histrum model can be integrated with different families of techniques and boost their performance.

This paper presents a large-scale study that investigates the bug resolution characteristics among popular Github projects written in different programming languages. We explore correlations but, of course, we cannot infer causation. Specifically, we analyse bug resolution data from approximately 70 million Source Line of Code, drawn from 3 million commits to 600 GitHub projects, primarily written in 10 programming languages. We find notable variations in apparent bug resolution time and patch (fix) size. While interpretation of results from such large-scale empirical studies is inherently difficult, we believe that the differences in medians are sufficiently large to warrant further investigation, replication, re-analysis and follow up research. For example, in our corpus, the median apparent bug resolution time (elapsed time from raise to resolve) for Ruby was 4X that for Go and 2.5X for Java. We also found that patches tend to touch more files for the corpus of strongly typed and for statically typed programs. However, we also found evidence for alowerelapsed resolution time for bug resolution committed to projects constructed from statically typed languages. These findings, if replicated in subsequent follow on studies, may shed further empirical light on the debate about the importance of static typing.

With the increasing adoption of Deep Learning (DL) models in various applications, testing DL models is vitally important. However, testing DL models is costly and expensive, e.g., manual labelling is widely-recognized to be costly. To reduce testing cost, we propose to select only a subset of testing data, which is small but representative enough for a quick estimation of the performance of DL models. Our approach, DeepReduce, adopts a two-phase strategy. At first, our approach selects testing data for the purpose of satisfying testing adequacy. Then, it selects more testing data to approximate the distribution between the whole testing data and the selected data by leveraging relative entropy minimization. We evaluate DeepReduce on four widely-used datasets (with 15 models in total). We find that DeepReduce reduces the whole testing data to 7.5% on average and can reliably estimate the performance of DL models.

A large body of research efforts have been dedicated to automated software debugging, including both automated fault localization and program repair. However, existing fault localization techniques have limited effectiveness on real-world software systems while even the most advanced program repair techniques can only fix a small ratio of real-world bugs. Although fault localization and program repair are inherently connected, their only existing connection in the literature is that program repair techniques usually use off-the-shelf fault localization techniques (e.g., Ochiai) to determine the potential candidate statements/elements for patching. In this work, we propose the unified debugging approach to unify the two areas in the other direction for the first time, i.e., can program repair in turn help with fault localization? In this way, we not only open a new dimension for more powerful fault localization, but also extend the application scope of program repair to all possible bugs (not only the bugs that can be directly automatically fixed). We have designed ProFL to leverage patch-execution results (from program repair) as the feedback information for fault localization. The experimental results on the widely used Defects4J benchmark show that the basic ProFL can already at least localize 37.61% more bugs within Top-1 than state-of-the-art spectrum and mutation based fault localization. Furthermore, ProFL can boost state-of-the-art fault localization via both unsupervised and supervised learning. Meanwhile, we have demonstrated ProFL's effectiveness under different settings and through a case study within Alipay, a popular online payment system with over 1 billion global users.

Although tremendous efforts have been devoted to the quality assurance of online service systems, in reality, these systems still come across many incidents (i.e., unplanned interruptions and outages), which can decrease user satisfaction or cause economic loss. To better understand the characteristics of incidents and improve the incident management process, we perform the first large-scale empirical analysis of incidents collected from 18 real-world online service systems in Microsoft. Surprisingly, we find that although a large number of incidents could occur over a short period of time, many of them actually do not matter, i.e., engineers will not fix them with a high priority after manually identifying their root cause. We call these incidents incidental incidents. Our qualitative and quantitative analyses show that incidental incidents are significant in terms of both number and cost. Therefore, it is important to prioritize incidents by identifying incidental incidents in advance to optimize incident management efforts. In particular, we propose an approach, called DeepIP (Deep learning based Incident Prioritization), to prioritizing incidents based on a large amount of historical incident data. More specifically, we design an attention-based Convolutional Neural Network (CNN) to learn a prediction model to identify incidental incidents. We then prioritize all incidents by ranking the predicted probabilities of incidents being incidental. We evaluate the performance of DeepIP using real-world incident data. The experimental results show that DeepIP effectively prioritizes incidents by identifying incidental incidents and significantly outperforms all the compared approaches. For example, the AUC of DeepIP achieves 0.808, while that of the best compared approach is only 0.624 on average.

Regression testing is widely recognized as an important but time-consuming process. To alleviate this cost issue, test selection, reduction, and prioritization have been widely studied, and they share the commonality that they improve regression testing by optimizing the execution of the whole test suite. In this paper, we attempt to accelerate regression testing from a totally new perspective, i.e., skipping some execution of a new program by reusing program states of an old program. Following this intuition, we propose a state-reuse based acceleration approach SRRTA, consisting of two components: state storage and loading. With the former, SRRTA collects some program states during the execution of an old version through three heuristic-based storage strategies; with the latter, SRRTA loads the stored program states with efficiency optimization strategies. Through the preliminary study on commons-math, SRRTA reduces 82.7% of the regression testing time.

Build systems are essential for modern software maintenance and development, while build failures occur frequently across software systems, inducing non-negligible costs in development activities. Build failure resolution is a challenging problem and multiple studies have demonstrated that developers spend non-trivial time in resolving encountered build failures; to relieve manual efforts, automated resolution techniques are emerging recently, which are promising but still limitedly effective. Understanding how build failures are resolved in practice can provide guidelines for both developers and researchers on build issue resolution. Therefore, this work presents a comprehensive study of fix patterns in practical build failures. Specifically, we study 1,080 build issues of three popular build systems Maven, Ant, and Gradle from Stack Overflow, construct a fine-granularity taxonomy of 50 categories regarding to the failure symptoms, and summarize the fix patterns for different failure types. Our key findings reveal that build issues stretch over a wide spectrum of symptoms; 67.96% of the build issues are fixed by modifying the build script code related to plugins and dependencies; and there are 20 symptom categories, more than half of whose build issues can be fixed by specific patterns. Furthermore, we also address the challenges in applying non-intuitive or simplistic fix patterns for developers.

Metamorphic relations (MRs) describe the invariant relationships between program inputs and outputs. By checking for violations of MRs, faults in programs can be detected. Identifying MRs manually is a tedious and error-prone task. In this paper, we propose AutoMR, a novel method for systematically inferring and cleansing MRs. AutoMR can discover various types of equality and inequality MRs through a search method (particle swarm optimization). It also employs matrix singular-value decomposition and constraint solving techniques to remove the redundant MRs in the search results. Our experiments on 37 numerical programs from two popular open source packages show that AutoMR can effectively infer a set of accurate and succinct MRs and outperform the state-of-the-art method. Furthermore, we show that the discovered MRs have high fault detection ability in mutation testing and differential testing.

This artifact is related to our Research Track paper that is accepted at ICSME 2019 [1]. Metamorphic relations (MRs) describe the invariant relationships between program inputs and outputs. We propose AutoMR, a novel method for systematically inferring and cleansing MRs. AutoMR can discover various types of equality and inequality MRs through a search method (particle swarm optimization). It also employs matrix singular-value decomposition and constraint solving techniques to remove the redundant MRs in the search results. Our experiments on 37 numerical programs show that AutoMR can effectively infer accurate and succinct MRs and outperform the state-of-the-art method. Furthermore, we show that the discovered MRs have high fault detection ability in mutation testing and differential testing.

Code coverage is the most widely adopted criteria for measuring test effectiveness in software quality assurance. The performance of coverage criteria (in indicating test suites' effectiveness) has been widely studied in prior work. Most of the studies use randomly constructed pseudo test suites to facilitate data collection for correlation analysis, yet no previous work has systematically studied whether pseudo test suites would lead to inflated correlation results. This paper focuses on the potentially wide-spread threat with a study over 123 real-world Java projects. Following the typical experimental process of studying coverage criteria, we investigate the correlation between statement/assertion coverage and mutation score using both pseudo and original test suites. Except for direct correlation analysis, we control the number of assertions and the test suite size to conduct partial correlation analysis. The results reveal that 1) the correlation (between coverage criteria and mutation score) derived from pseudo test suites is much higher than from original test suites (from 0.21 to 0.39 higher in Kendall value); 2) contrary to previously reported, statement coverage has a stronger correlation with mutation score than assertion coverage.

Build systems are essential for modern software development and maintenance since they are widely used to transform source code artifacts into executable software. Previous work shows that build systems break frequently during software evolution. Therefore, automated build-fixing techniques are in huge demand. In this paper we target a mainstream build system, Gradle, which has become the most widely used build system for Java projects in the open-source community (e.g., GitHub). HireBuild, state-of-the-art build-fixing tool for Gradle, has been recently proposed to fix Gradle build failures via mining the history of prior fixes. Although HireBuild has been shown to be effective for fixing real-world Gradle build failures, it was evaluated on only a limited set of build failures, and largely depends on the quality/availability of historical fix information. To investigate the efficacy and limitations of the history-driven build fix, we first construct a new and large build failure dataset from Top-1000 GitHub projects. Then, we evaluate HireBuild on the extended dataset both quantitatively and qualitatively. Inspired by the findings of the study, we propose a simplistic new technique that generates potential patches via searching from the present project under test and external resources rather than the historical fix information. According to our experimental results, the simplistic approach based on present information successfully fixes 2X more reproducible build failures than the state-of-art HireBuild based on historical fix information. Furthermore, our results also reveal various findings/guidelines for future advanced build failure fixing.

In recent years, online service systems have become increasingly popular. Incidents of these systems could cause significant economic loss and customer dissatisfaction. Incident triage, which is the process of assigning a new incident to the responsible team, is vitally important for quick recovery of the affected service. Our industry experience shows that in practice, incident triage is not conducted only once in the beginning, but is a continuous process, in which engineers from different teams have to discuss intensively among themselves about an incident, and continuously refine the incident-triage result until the correct assignment is reached. In particular, our empirical study on 8 real online service systems shows that the percentage of incidents that were reassigned ranges from 5.43% to 68.26% and the number of discussion items before achieving the correct assignment is up to 11.32 on average. To improve the existing incident triage process, in this paper, we propose DeepCT, a Deep learning based approach to automated Continuous incident Triage. DeepCT incorporates a novel GRU-based (Gated Recurrent Unit) model with an attention-based mask strategy and a revised loss function, which can incrementally learn knowledge from discussions and update incident-triage results. Using DeepCT, the correct incident assignment can be achieved with fewer discussions. We conducted an extensive evaluation of DeepCT on 14 large-scale online service systems in Microsoft. The results show that DeepCT is able to achieve more accurate and efficient incident triage, e.g., the average accuracy identifying the responsible team precisely is 0.641~0.729 with the number of discussion items increasing from 1 to 5. Also, DeepCT statistically significantly outperforms the state-of-the-art bug triage approach.

Compilers, like other software systems, contain bugs, and compiler testing is the most widely-used way to assure compiler quality. A critical task of compiler testing is to generate test programs that could effectively and efficiently discover bugs. Though we can configure test generators such as Csmith to control the features of the generated programs, it is not clear what test configuration is effective. In particular, an effective test configuration needs to generate test programs that are bug-revealing, i.e., likely to trigger bugs, and diverse, i.e., able to discover different types of bugs. It is not easy to satisfy both properties. In this paper, we propose a novel test-program generation approach, called HiCOND, which utilizes historical data for configuration diversification to solve this challenge. HiCOND first infers the range for each option in a test configuration where bug-revealing test programs are more likely to be generated based on historical data. Then, it identifies a set of test configurations that can lead to diverse test programs through a search method (particle swarm optimization). Finally, based on the set of test configurations for compiler testing, HiCOND generates test programs, which are likely to be bug-revealing and diverse. We have conducted experiments on two popular compilers GCC and LLVM, and the results confirm the effectiveness of our approach. For example, HiCOND detects 75.00%, 133.33%, and 145.00% more bugs than the three existing approaches, respectively. Moreover, HiCOND has been successfully applied to actual compiler testing in a global IT company and detected 11 bugs during the practical evaluation.

Compiler bugs are extremely harmful, but are notoriously difficult to debug because compiler bugs usually produce few debugging information. Given a bug-triggering test program for a compiler, hundreds of compiler files are usually involved during compilation, and thus are suspect buggy files. Although there are lots of automated bug isolation techniques, they are not applicable to compilers due to the scalability or effectiveness problem. To solve this problem, in this paper, we transform the compiler bug isolation problem into a search problem, i.e., searching for a set of effective witness test programs that are able to eliminate innocent compiler files from suspects. Based on this intuition, we propose an automated compiler bug isolation technique, DiWi, which (1) proposes a heuristic-based search strategy to generate such a set of effective witness test programs via applying our designed witnessing mutation rules to the given failing test program, and (2) compares their coverage to isolate bugs following the practice of spectrum-based bug isolation. The experimental results on 90 real bugs from popular GCC and LLVM compilers show that DiWi effectively isolates 66.67%/78.89% bugs within Top-10/Top-20 compiler files, significantly outperforming state-of-the-art bug isolation techniques.

Test suites play a key role in ensuring software quality. A good test suite may detect more faults than a poor-quality one. Mutation testing is a powerful methodology for evaluating the fault-detection ability of test suites. In mutation testing, a large number of mutants may be generated and need to be executed against the test suite under evaluation to check how many mutants the test suite is able to detect, as well as the kind of mutants that the current test suite fails to detect. Consequently, although highly effective, mutation testing is widely recognized to be also computationally expensive, inhibiting wider uptake. To alleviate this efficiency concern, we propose Predictive Mutation Testing (PMT): the first approach to predicting mutation testing results without executing mutants. In particular, PMT constructs a classification model, based on a series of features related to mutants and tests, and uses the model to predict whether a mutant would be killed or remain alive without executing it. PMT has been evaluated on 163 real-world projects under two application scenarios (cross-version and cross-project). The experimental results demonstrate that PMT improves the efficiency of mutation testing by up to 151.4X while incurring only a small accuracy loss. It achieves above 0.80 AUC values for the majority of projects, indicating a good tradeoff between the efficiency and effectiveness of predictive mutation testing. Also, PMT is shown to perform well on different tools and tests, be robust in the presence of imbalanced data, and have high predictability (over 60 percent confidence) when predicting the execution results of the majority of mutants.

Symbolic execution is an effective but expensive technique for automated test generation. Over the years, a large number of refined symbolic execution techniques have been proposed to improve its efficiency. However, the symbolic execution efficiency problem remains, and largely limits the application of symbolic execution in practice. Orthogonal to refined symbolic execution, in this paper we propose to accelerate symbolic execution through semantic-preserving code transformation on the target programs. During the initial stage of this direction, we adopt a particular code transformation, compiler optimization, which is initially proposed to accelerate program concrete execution by transforming the source program into another semantic-preserving target program with increased efficiency (e.g., faster or smaller). However, compiler optimizations are mostly designed to accelerate program concrete execution rather than symbolic execution. Recent work also reported that unified settings on compiler optimizations that can accelerate symbolic execution for any program do not exist at all. Therefore, in this work we propose a machine-learning based approach to tuning compiler optimizations to accelerate symbolic execution, whose results may also aid further design of specific code transformations for symbolic execution. In particular, the proposed approach LEO separates source-code functions and libraries through our program-splitter, and predicts individual compiler optimization (i.e., whether a type of code transformation is chosen) separately through analyzing the performance of existing symbolic execution. Finally, LEO applies symbolic execution on the code transformed by compiler optimization (through our local-optimizer). We conduct an empirical study on GNU Coreutils programs using the KLEE symbolic execution engine. The results show that LEO significantly accelerates symbolic execution, outperforming the default KLEE configurations (i.e., turning on/off all compiler optimizations) in various settings, e.g., with the default training/testing time, LEO achieves the highest line coverage in 50/68 programs, and its average improvement rate on all programs is 46.48%/88.92% in terms of line coverage compared with turning on/off all compiler optimizations.

The effectiveness of search-based automated program repair is limited in the number of correct patches that can be successfully generated. There are two causes of such limitation. First, the search space does not contain the correct patch. Second, the search space is huge and therefore the correct patch cannot be generated (i.e., correct patches are either generated after incorrect plausible ones or not generated within the time budget).

Test prioritization aims to detect regression faults faster via reordering test executions, and a large number of test prioritization techniques have been proposed accordingly. However, test prioritization effectiveness is usually measured in terms of the average percentage of faults detected concerned with the number of test executions, rather than the actual regression testing time, making it unclear which technique is optimal in actual regression testing time. To answer this question, this paper first conducts an empirical study to investigate the actual regression testing time of various prioritization techniques. The results reveal a number of practical guidelines. In particular, no prioritization technique can always perform optimal in practice.

Spreadsheets are the most popular end-user programming software, where formulae act like programs and also have smells. One well recognized smell is the use of nested-IF expressions, which have low readability and high cognitive cost for users, and are error-prone during reuse or maintenance. End users usually lack essential programming language knowledge to tackle or even realize this problem, yet no automatic approaches are currently available. This paper proposes the first exploration of the nest-if usage status against two large-scale spreadsheet corpora containing over 80,000 industry-level spreadsheets. It turns out the use of nested-IF expressions are surprisingly common among end users. We then present an approach to tackling this problem through automatic formula refactoring. The general idea of the automatic approach is two-fold. First, we detect and remove logic redundancy based on the AST of a formula. Second, we identify higher-level semantics that have been represented with fragmented and scattered syntax, and reassemble the syntax using concise built-in functions. A comprehensive evaluation with over 28 million nested-IF formulae reveals that the approach is able to relieve the smell of over 90% of nested-IF formulae.

Compiler testing is a crucial way of guaranteeing the reliability of compilers (and software systems in general). Many techniques have been proposed to facilitate automated compiler testing. These techniques rely on a large number of test programs (which are test inputs of compilers) generated by some test-generation tools (e.g., CSmith). However, these compiler testing techniques have serious efficiency problems as they usually take a long period of time to find compiler bugs. To accelerate compiler testing, it is desirable to prioritize the generated test programs so that the test programs that are more likely to trigger compiler bugs are executed earlier. In this paper, we propose the idea of learning to test, which learns the characteristics of bug-revealing test programs from previous test programs that triggered bugs. Based on the idea of learning to test, we propose LET, an approach to prioritizing test programs for compiler testing acceleration. LET consists of a learning process and a scheduling process. In the learning process, LET identifies a set of features of test programs, trains a capability model to predict the probability of a new test program for triggering compiler bugs and a time model to predict the execution time of a test program. In the scheduling process, LET prioritizes new test programs according to their bug-revealing probabilities in unit time, which is calculated based on the two trained models. Our extensive experiments show that LET significantly accelerates compiler testing. In particular, LET reduces more than 50% of the testing time in 24.64% of the cases, and reduces between 25% and 50% of the testing time in 36.23% of the cases.

Code coverage is the dominant criterion in test-suite reduction. Typically, most test-suite reduction techniques repeatedly remove tests covering code that has been covered by other tests from the test suite. However, test-suite reduction based on code coverage alone may incur fault-detection capability loss, because a test detects faults if and only if its execution covers buggy code and its test oracle catches the buggy state. In other words, test oracles may also affect test-suite reduction, However, to our knowledge, their impacts have never been studied before. In this paper, we conduct the first empirical study on such impacts by using 10 real-world GitHub Java projects, and find that assertions (i.e., a typical type of test oracles) are significantly correlated with coverage-based test-suite reduction. Based on our preliminary study results, we also proposed an assertion-aware test-suite reduction technique which outperforms traditional test-suite reduction in terms of cost-effectiveness.

Mutation analysis has many applications, such as asserting the quality of test suites and localizing faults. One important bottleneck of mutation analysis is scalability. The latest work explores the possibility of reducing the redundant execution via split-stream execution. However, split-stream execution is only able to remove redundant execution before the first mutated statement.

Compilers, as one of the most important infrastructure of today's digital world, are expected to be trustworthy. Different testing techniques are developed for testing compilers automatically. However, it is unknown so far how these testing techniques compared to each other in terms of testing effectiveness: how many bugs a testing technique can find within a time limit.

In recent years, researchers have intensively investigated various topics in test prioritization, which aims to re-order tests to increase the rate of fault detection during regression testing. While the main research focus in test prioritization is on proposing novel prioritization techniques and evaluating on more and larger subject systems, little effort has been put on investigating the threats to validity in existing work on test prioritization. One main threat to validity is that existing work mainly evaluates prioritization techniques based on simple artificial changes on the source code and tests. For example, the changes in the source code usually include only seeded program faults, whereas the test suite is usually not augmented at all. On the contrary, in real-world software development, software systems usually undergo various changes on the source code and test suite augmentation. Therefore, it is not clear whether the conclusions drawn by existing work in test prioritization from the artificial changes are still valid for real-world software evolution. In this paper, we present the first empirical study to investigate this important threat to validity in test prioritization. We reimplemented 24 variant techniques of both the traditional and time-aware test prioritization, and investigated the impacts of software evolution on those techniques based on the version history of 8 real-world Java programs from GitHub. The results show that for both traditional and time-aware test prioritization, test suite augmentation significantly hampers their effectiveness, whereas source code changes alone do not influence their effectiveness much.

Selecting and prioritizing major device models are critical for mobile app developers to select testbeds and optimize resources such as marketing and quality-assurance resources. The heavily fragmented distribution of Android devices makes it challenging to select a few major device models out of thousands of models available on the market. Currently app developers usually rely on some reported or estimated general market share of device models. However, these estimates can be quite inaccurate, and more problematically, can be irrelevant to the particular app under consideration. To address this issue, we propose PRADA, the first approach to prioritizing Android device models for individual apps, based on mining large-scale usage data. PRADA adapts the concept of operational profiling (popularly used in software reliability engineering) for mobile apps -- the usage of an app on a specific device model reflects the importance of that device model for the app. PRADA includes a collaborative filtering technique to predict the usage of an app on different device models, even if the app is entirely new (without its actual usage in the market yet), based on the usage data of a large collection of apps. We empirically demonstrate the effectiveness of PRADA over two popular app categories, i.e., Game and Media, covering over 3.86 million users and 14,000 device models collected through a leading Android management app in China.

Test case prioritization aims to schedule the execution order of test cases so as to detect bugs as early as possible. For compiler testing, the demand for both effectiveness and efficiency imposes challenge to test case prioritization. In the literature, most existing approaches prioritize test cases by using some coverage information (e.g., statement coverage or branch coverage), which is collected with considerable extra effort. Although input-based test case prioritization relies only on test inputs, it can hardly be applied when test inputs are programs. In this paper we propose a novel text-vector based test case prioritization approach, which prioritizes test cases for C compilers without coverage information. Our approach first transforms each test case into a text-vector by extracting its tokens which reflect fault-relevant characteristics and then prioritizes test cases based on these text-vectors. In particular, in our approach we present three prioritization strategies: greedy strategy, adaptive random strategy, and search strategy. To investigate the efficiency and effectiveness of our approach, we conduct an experiment on two C compilers (i.e., GCC and LLVM), and find that our approach is much more efficient than the existing approaches and is effective in prioritizing test cases.

The correlation between test coverage and test effectiveness is important to justify the use of coverage in practice. Existing results on imperative programs mostly show that test coverage predicates effectiveness. However, since functional programs are usually structurally different from imperative ones, it is unclear whether the same result may be derived and coverage can be used as a prediction of effectiveness on functional programs. In this paper we report the first empirical study on the correlation between test coverage and test effectiveness on functional programs. We consider four types of coverage: as input coverages, statement/branch coverage and expression coverage, and as oracle coverages, count of assertions and checked coverage. We also consider two types of effectiveness: raw effectiveness and normalized effectiveness. Our results are twofold. (1) In general the findings on imperative programs still hold on functional programs, warranting the use of coverage in practice. (2) On specific coverage criteria, the results may be unexpected or different from the imperative ones, calling for further studies on functional programs.

Mutation testing is a powerful methodology for evaluating test suite quality. In mutation testing, a large number of mutants are generated and executed against the test suite to check the ratio of killed mutants. Therefore, mutation testing is widely believed to be a computationally expensive technique. To alleviate the efficiency concern of mutation testing, in this paper, we propose predictive mutation testing (PMT), the first approach to predicting mutation testing results without mutant execution. In particular, the proposed approach constructs a classification model based on a series of features related to mutants and tests, and uses the classification model to predict whether a mutant is killed or survived without executing it. PMT has been evaluated on 163 real-world projects under two application scenarios (i.e., cross-version and cross-project). The experimental results demonstrate that PMT improves the efficiency of mutation testing by up to 151.4X while incurring only a small accuracy loss when predicting mutant execution results, indicating a good tradeoff between efficiency and effectiveness of mutation testing.

In software testing, the program under test is usually executed with test inputs and checked against a test oracle, which is a mechanism to verify whether the program behaves as expected. Selecting the right oracle data to observe is crucial in test oracle construction. In the literature, researchers have proposed two dynamic approaches to oracle data selection by analyzing test execution information (e.g., variables' values or interaction information). However, collecting such information during program execution may incur extra cost. In this paper, we present the first static approach to oracle data selection, SODS (Static Oracle Data Selection). In particular, SODS first identifies the substitution relationships between candidate oracle data by constructing a probabilistic substitution graph based on the definition-use chains of the program under test, then estimates the fault-observing capability of each candidate oracle data, and finally selects a subset of oracle data with strong fault-observing capability. For programs with analyzable test code, we further extend SODS via pruning the probabilistic substitution graph based on 0-1-CFA call graph analysis. The experimental study on 11 subject systems written in C or Java demonstrates that our static approach is more effective and much more efficient than state-of-the-art dynamic approaches in most cases.

In software testing, it is very hard to achieve high coverage with the program under test, leaving many behaviors unexplored. To alleviate this problem, various automated test generation and augmentation approaches have been proposed, among which symbolic execution and search-based techniques are the most competitive, while each has key challenges to be solved. Different from prior work, we present a new methodology for regression testing --Isomorphic Regression Testing,which explores the behaviors of the program under test by creating its variants (i.e., modified programs) instead of generating tests. In this paper, we make the first implementation of isomorphic regression testing through an approach named ISON, which creates program variants by negating branch conditions. The results show that ISON is able to additionally execute 5.3% to 80.0% branches that are originally uncovered. Furthermore, ISON also detects a number of faults not detected by a popular automated test generation tool (i.e., EvoSuite) under the scenario of regression testing.

Software testing aims to assure the quality of software under test. To improve the efficiency of software testing, especially regression testing, test-case prioritization is proposed to schedule the execution order of test cases in software testing. Among various test-case prioritization techniques, the simple additional coverage-based technique, which is a greedy strategy, achieves surprisingly competitive empirical results. To investigate how much difference there is between the order produced by the additional technique and the optimal order in terms of coverage, we conduct a study on various empirical properties of optimal coverage-based test-case prioritization. To enable us to achieve the optimal order in acceptable time for our object programs, we formulate optimal coverage-based test-case prioritization as an integer linear programming (ILP) problem. Then we conduct an empirical study for comparing the optimal technique with the simple additional coverage-based technique. From this empirical study, the optimal technique can only slightly outperform the additional coverage-based technique with no statistically significant difference in terms of coverage, and the latter significantly outperforms the former in terms of either fault detection or execution time. As the optimal technique schedules the execution order of test cases based on their structural coverage rather than detected faults, we further implement the ideal optimal test-case prioritization technique, which schedules the execution order of test cases based on their detected faults. Taking this ideal technique as the upper bound of test-case prioritization, we conduct another empirical study for comparing the optimal technique and the simple additional technique with this ideal technique. From this empirical study, both the optimal technique and the additional technique significantly outperform the ideal technique in terms of coverage, but the latter significantly outperforms the former two techniques in terms of fault detection. Our findings indicate that researchers may need take cautions in pursuing the optimal techniques in test-case prioritization with intermediate goals.

This paper is a report on The 10th IEEE/ACMInternational Workshop on Automation of Software Test (AST2015) at the 37th International Conference on Software Engineering(ICSE 2015). It sets a special theme on testing oracles.Keynote speeches and charette discussions are organized aroundthis special theme. 16 full research papers and 2 keynotes willbe presented in the two-day workshop. The report will give thebackground of the workshop and the selection of the specialtheme, and report on the organization of the workshop. Theprovisional program will be presented with a list of the sessionsand papers to be presented at the workshop.

In the software development process, how to develop better software at lower cost has been a major issue of concern. One way that helps is to find more defects as early as possible, on which defect prediction can provide effective guidance. The most popular defect prediction technique is to build defect prediction models based on machine learning. To improve the performance of defect prediction model, selecting appropriate features is critical. On the other hand, static analysis is usually used in defect detection. As static defect analyzers detects defects by matching some well-defined "defect patterns", its result is useful for locating defects. However, defect prediction and static defect analysis are supposed to be two parallel areas due to the differences in research motivation, solution and granularity.

During software evolution, to assure the software quality, test cases for an early version tend to be reused by its latter versions. As a large number of test cases may aggregate during software evolution, it becomes necessary to schedule the execution order of test cases so that the faults in the latter version may be detected as early as possible, which is test-case prioritization in software evolution. In this paper, we proposed a novel test-case prioritization approach for software evolution, which first uses mutation faults on the difference between the early version and the latter version to simulate real faults occurred in software evolution, and then schedules the execution order of test cases based on their fault-detection capability, which is defined based on mutation faults. In particular, we present two models on calculating fault-detection capability, which are statistics-based model and probability-based model. Moreover, we conducted an experimental study and found that our approach with the statistics-based model outperforms our approach with the probability-based model and the total statement coverage-based approach, and slightly outperforms the additional statement-coverage based approach in many cases. Furthermore, compared with the total or additional statement coverage-based approach, our approach with either the statistics-based model or the probability-based model tends to be stably effective when the difference on the source code between the early version and the latter version is non-trivial.

Covering arrays (CAs) are often used as test suites for combinatorial interaction testing to discover interaction faults of real-world systems. Most real-world systems involve constraints, so improving algorithms for covering array generation (CAG) with constraints is beneficial. Two popular methods for constrained CAG are greedy construction and meta-heuristic search. Recently, a meta-heuristic framework called two-mode local search has shown great success in solving classic NPhard problems. We are interested whether this method is also powerful in solving the constrained CAG problem. This work proposes a two-mode meta-heuristic framework for constrained CAG efficiently and presents a new meta-heuristic algorithm called TCA. Experiments show that TCA significantly outperforms state-of-the-art solvers on 3-way constrained CAG. Further experiments demonstrate that TCA also performs much better than its competitors on 2-way constrained CAG.

In order to learn the impact of MOOCs, we conducted a SPOC experiment on the course of Data Structures and Algorithms in Peking University. In this paper, we analyze student online activities, test scores, and two surveys using statistical methods (t-test, analysis of variance, correlation analysis and OLS regression) to understand what factors will foster improvements in student learning. We find that the "SPOC + Flipped" is a helpful mode to teach algorithm, time spent on the course and students' confidence had a positive impact on learning effect, and SPOC resource should be made full use of.

Traditional test oracles are defined on the outputs of test executions, and cannot assert internal states of executions. Traditional assertions are common to all test execution, and are usually more difficult to construct than on oracle for one test input. In this paper we propose the concept of inner oracles, which are assertions on internal states that are specific to one test input. We first motivate the necessity of inner oracles, and then show that it can be implemented easily using the available programming mechanisms. Next, we report two initial empirical studies on inner oracles, showing that inner oracles have a significant impact on both the fault-detection capability of tests and the performance of test suite reduction. Finally, we highlight the implications of inner oracles on several research and practical problems.

To deal with post-release bugs, many software projects set up public bug repositories for users all over the world to report bugs that they have encountered. Recently, researchers have proposed various information retrieval based approaches to localizing faults based on bug reports. In these approaches, source files are processed as single units, where noise in large files may affect the accuracy of fault localization. Furthermore, bug reports often contain stack-trace information, but existing approaches often treat this information as plain text. In this paper, we propose to use segmentation and stack-trace analysis to improve the performance of bug localization. Specifically, given a bug report, we divide each source code file into a series of segments and use the segment most similar to the bug report to represent the file. We also analyze the bug report to identify possible faulty files in a stack trace and favor these files in our retrieval. According to our empirical results, our approach is able to significantly improve Bug Locator, a representative fault localization approach, on all the three software projects (i.e., Eclipse, AspectJ, and SWT) used in our empirical evaluation. Furthermore, segmentation and stack-trace analysis are complementary to each other for boosting the performance of bug-report-oriented fault localization.

Software testing plays an important role in ensuring software quality by running a program with test suites. Mutation testing is designed to evaluate whether a test suite is adequate in detecting faults. Due to the expensive cost of mutation testing, selective mutation testing was proposed to select a subset of mutants whose effectiveness is similar to the whole set of generated mutants. Although selective mutation testing has been widely investigated in recent years, many people still doubt whether it can suit well for large programs. To study the scalability of selective mutation testing, we systematically explore how the program size impacts selective mutation testing through four projects (including 12 versions all together). Based on the empirical study, for programs smaller than 16 KLOC, selective mutation testing has surprisingly good scalability. In particular, for a program whose number of lines of executable code is E, the number of mutants used in selective mutation testing is proportional to Ec, where c is a constant whose value is between 0.05 and 0.25.

Metamorphic testing (MT) is an effective methodology for testing those so-called ``non-testable'' programs (e.g., scientific programs), where it is sometimes very difficult for testers to know whether the outputs are correct. In metamorphic testing, metamorphic relations (MRs) (which specify how particular changes to the input of the program under test would change the output) play an essential role. However, testers may typically have to obtain MRs manually.

Test case prioritization techniques attempt to reorder test cases in a manner that increases the rate at which faults are detected during regression testing. Coverage-based test case prioritization techniques typically use one of two overall strategies: a total strategy or an additional strategy . These strategies prioritize test cases based on the total number of code (or code-related) elements covered per test case and the number of additional (not yet covered) code (or code-related) elements covered per test case, respectively. In this article, we present a unified test case prioritization approach that encompasses both the total and additional strategies. Our unified test case prioritization approach includes two models ( basic and extended ) by which a spectrum of test case prioritization techniques ranging from a purely total to a purely additional technique can be defined by specifying the value of a parameter referred to as the f p value. To evaluate our approach, we performed an empirical study on 28 Java objects and 40 C objects, considering the impact of three internal factors (model type, choice of f p value, and coverage type) and three external factors (coverage granularity, test case granularity, and programming/testing paradigm), all of which can be manipulated by our approach. Our results demonstrate that a wide range of techniques derived from our basic and extended models with uniform f p values can outperform purely total techniques and are competitive with purely additional techniques. Considering the influence of each internal and external factor studied, the results demonstrate that various values of each factor have nontrivial influence on test case prioritization techniques.

Test-case prioritization is to schedule the execution order of test cases so as to maximize some objective (e.g., revealing faults early). The existing test-case prioritization approaches separate the process of test-case prioritization and the process of test-case execution by presenting the execution order of all test cases before programmers start running test cases. As the execution information of the modified program is not available for the existing test-case prioritization approaches, these approaches mainly rely on only the execution information of the previous program before modification. To address this problem, we present an adaptive test-case prioritization approach, which determines the execution order of test cases simultaneously during the execution of test cases. In particular, the adaptive approach selects test cases based on their fault-detection capability, which is calculated based on the output of selected test cases. As soon as a test case is selected and runs, the fault-detection capability of each unselected test case is modified according to the output of the latest selected test case. To evaluate the effectiveness of the proposed adaptive approach, we conducted an experimental study on eight C programs and four Java programs. The experimental results show that the adaptive approach is usually significantly better than the total test-case prioritization approach and competitive to the additional test-case prioritization approach. Moreover, the adaptive approach is better than the additional approach on some subjects (e.g, replace and schedule).

In recent years, researchers have intensively investigated various topics in test-case prioritization, which aims to re-order test cases to increase the rate of fault detection during regression testing. The total and additional prioritization strategies, which prioritize based on total numbers of elements covered per test, and numbers of additional (not-yet-covered) elements covered per test, are two widely-adopted generic strategies used for such prioritization. This paper proposes a basic model and an extended model that unify the total strategy and the additional strategy. Our models yield a spectrum of generic strategies ranging between the total and additional strategies, depending on a parameter referred to as the p value. We also propose four heuristics to obtain differentiated p values for different methods under test. We performed an empirical study on 19 versions of four Java programs to explore our results. Our results demonstrate that wide ranges of strategies in our basic and extended models with uniform p values can significantly outperform both the total and additional strategies. In addition, our results also demonstrate that using differentiated p values for both the basic and extended models with method coverage can even outperform the additional strategy using statement coverage.

As test data widely spreading on the input domain may not thoroughly test the program's logic, in this paper, we propose an approach to generating test data widely spreading on a program's execution paths. In particular, we analyze execution paths of the program, distill constraints for executing the paths, calculate the path distance between test data according to their satisfaction for paths' constraints, and then generate test data far away from each other based on their path distance. The experimental results show that our approach significantly reduces the number of test data generated before the first fault is found.

Most test suite reduction techniques aim to select, from a given test suite, a minimal representative subset of test cases that retains the same code coverage as the suite. Empirical studies have shown, however, that test suites reduced in this manner may lose fault detection capability. Techniques have been proposed to retain certain redundant test cases in the reduced test suite so as to reduce the loss in fault-detection capability, but these still do concede some degree of loss. Thus, these techniques may be applicable only in cases where loose demands are placed on the upper limit of loss in fault-detection capability. In this work we present an on-demand test suite reduction approach, which attempts to select a representative subset satisfying the same test requirements as an initial test suite conceding at most l% loss in fault-detection capability for at least c% of the instances in which it is applied. Our technique collects statistics about loss in fault-detection capability at the level of individual statements and models the problem of test suite reduction as an integer linear programming problem. We have evaluated our approach in the contexts of three scenarios in which it might be used. Our results show that most test suites reduced by our approach satisfy given fault detection capability demands, and that the approach compares favorably with an existing test suite reduction approach.

Test case prioritization is used in regression testing to schedule the execution order of test cases so as to expose faults earlier in testing. Over the past few years, many test case prioritization techniques have been proposed in the literature. Most of these techniques require data on dynamic execution in the form of code coverage information for test cases. However, the collection of dynamic code coverage information on test cases has several associated drawbacks including cost increases and reduction in prioritization precision. In this paper, we propose an approach to prioritizing test cases in the absence of coverage information that operates on Java programs tested under the JUnit framework-an increasingly popular class of systems. Our approach, JUnit test case Prioritization Techniques operating in the Absence of coverage information (JUPTA), analyzes the static call graphs of JUnit test cases and the program under test to estimate the ability of each test case to achieve code coverage, and then schedules the order of these test cases based on those estimates. To evaluate the effectiveness of JUPTA, we conducted an empirical study on 19 versions of four Java programs ranging from 2K-80K lines of code, and compared several variants of JUPTA with three control techniques, and several other existing dynamic coverage-based test case prioritization techniques, assessing the abilities of the techniques to increase the rate of fault detection of test suites. Our results show that the test suites constructed by JUPTA are more effective than those in random and untreated test orders in terms of fault-detection effectiveness. Although the test suites constructed by dynamic coverage-based techniques retain fault-detection effectiveness advantages, the fault-detection effectiveness of the test suites constructed by JUPTA is close to that of the test suites constructed by those techniques, and the fault-detection effectiveness of the test suites constructed by some of JUPTA's variants is better than that of the test suites constructed by several of those techniques.

Software debugging is time-consuming and effort-consuming. Although software debugging, especially fault-localization, has been studied for long, few practical debugging tools have been developed and used by the industry. In this paper we present VIDA, a visual interactive debugging tool, which has been integrated with the Eclipse Integrated Development Environment to support a programmer's debugging process. During the programmer's conventional debugging process, VIDA continuously recommends break-points for the programmer based on the analysis of execution information and the gathered feedback from the programmer. Moreover, VIDA provides a program outline to help the programmer choose breakpoints and visualizes the static dependency relation to help the programmer make estimation at breakpoints.

Better orderings of test cases can detect faults in less time with fewer resources, and thus make the debugging process earlier and accelerate software delivery. As a result, test case prioritization has become a hot topic in the research of regression testing. With the popularity of using the JUnit testing framework for developing Java software, researchers also paid attention to techniques for prioritizing JUnit test cases in regression testing of Java software. Typically, most of them are based on coverage information of test cases. However, coverage information may need extra costs to acquire. In this paper, we propose an approach (named Jupta) for prioritizing JUnit test cases in absence of coverage information. Jupta statically analyzes call graphs of JUnit test cases and the software under test to estimate the test ability (TA) of each test case. Furthermore, Jupta provides two prioritization techniques: the total TA based technique (denoted as JuptaT) and the additional TA based technique (denoted as JuptaA). To evaluate Jupta, we performed an experimental study on two open source Java programs, containing 11 versions in total. The experimental results indicate that Jupta is more effective and stable than the untreated orderings and Jupta is approximately as effective and stable as prioritization techniques using coverage information at the method level.

Test case management may make the testing process more efficient and thus accelerate software delivery. With the popularity of using JUnit for testing Java software, researchers have paid attention to techniques to manage JUnit test cases in regression testing of Java software. Typically, most existing test case management tools are based on the coverage information. However, coverage information may need extra efforts to obtain. In this paper, we present an Eclipse IDE plug-in (named Jtop) for managing JUnit test cases in absence of coverage information. Jtop statically analyzes the program under test and its corresponding JUnit test cases to perform the following management tasks: regression test case selection, test suite reduction and test case prioritization. Furthermore, Jtop also enables the programmer to manually manipulate test cases through a graphical user interface.

Finding the location of a fault is a central task of debugging. Typically, a developer employs an interactive process for fault localization. To accelerate this task, several approaches have been proposed to automate fault localization. In practice, testing-based fault localization (TBFL), which uses test information to locate faults, has become a research focus. However, experimental results reported in the literature showed that current automation of fault localization can only serve as a means to confirming the search space and prioritizing search sequences, not a substitute of the interactive fault localization process. In this paper, we propose an approach based on test information to support the entire interactive fault localization process. During this process, the information gathered from previous interaction steps can be used to provide the ranking of suspicious statements for the current interaction step. As a feasibility study of our approach, we performed an experiment on applying our approach together with some other TBFL approaches on the Siemens programs, which have been used in the literature. Our experimental results show the effectiveness of our approach.

Article Share on Testing-based interactive fault localization Author: Dan Hao Peking University, Beijing, P.R. China Peking University, Beijing, P.R. ChinaView Profile Authors Info & Claims ICSE '06: Proceedings of the 28th international conference on Software engineeringMay 2006 Pages 957–960https://doi.org/10.1145/1134285.1134462Published:28 May 2006Publication History 6citation470DownloadsMetricsTotal Citations6Total Downloads470Last 12 Months2Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my Alerts New Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

In the process of software maintenance, it is usually a time-consuming task to track down bugs. To reduce the cost on debugging, several approaches have been proposed to localize the fault(s) to facilitate debugging. Intuitively, testing-based fault localization (TBFL), such as dicing and TRANTULA, is quite promising as it can take the advantage of a large set of execution traces at the same time. However, redundant test cases may bias the distribution of the test suite and harm this kind of approaches. Therefore, we suggest that the test suite, which is the input of TBFL, should be reduced before used in TBFL. To evaluate whether and to what extent TBFL can benefit from test suite reduction, we performed an experimental study on two source programs. The experimental results show that, for test suites containing unevenly distributed redundant test cases, performing test suite reduction before applying TBFL may be more advantageous.

Debugging is a time-consuming task in software development and maintenance. To accelerate this task, several approaches have been proposed to automate fault localization. In particular, testing based fault localization (TBFL), which utilizes the testing information to localize the faults, seem to be very promising. However, the similarity between test cases in the test suite has been ignored in the research on TBFL. In this paper, we investigate this similarity issue and propose a novel approach named similarity-aware fault localization (SAFL), which can calculate the suspicion probability of each statement with little impact by the similarity issue. To address and deal with the similarity between test cases, SAFL applies the theory of fuzzy sets to remove the uneven distribution of the test cases. We also performed an experimental study for two real-world programs at different size levels to evaluate SAFL together with another two approaches to TBFL. Experimental results show that SAFL is more effective than the other two approaches when the test suites contain injected redundancy, and SAFL can achieve a competitive result with normal test suites. SAFL can also be more effective than applying test suite reduction to current approaches to TBFL.

Considering the scalability of using formal concept analysis to locate features in source code, we present a set of alternative straightforward algorithms to achieve the same objectives. A preliminary experiment indicates that the alternative algorithms are more scalable to deal with the large numbers of data to some extent.

Understanding the behavior of a software system is an important problem in program comprehension. Use cases have been accepted as an effective means for describing behavioral requirements for a software system. We propose a novel approach for obtaining use cases from source code. The central idea of our approach is to use the branch-reserving call graph (BRCG) as the intermediate representation of a software program. We also provide strategies for pruning the BRCG to avoid generating too many fine-grained use cases. Use cases, which may just undergo some minor modifications from human experts, can be generated through traversing the pruned BRCG. The contributions of our approach are three-fold, i) This method represents a compromised approach, which differs from both the static and dynamic approaches for use case discovery, ii) This method takes into consideration the fact that it is the branch statements that separate one use case from another in source code. iii) This method can avoid intensive human involvement in determining the final set of use cases. We have also performed a case study for this method on a GNU system.