Automated program repair (APR) techniques have achieved conspicuous progress, and are now capable of producing genuinely correct fixes in scenarios that were well beyond their capabilities only a few years ago. Nevertheless, even when an APR technique can find a correct fix for a bug, it still runs the risk of ranking the fix lower than other patches that are plausible (they pass all available tests) but incorrect. This can seriously hurt the technique’s practical effectiveness, as the user will have to peruse a larger number of patches before finding the correct one. This paper presents PrevaRank , a technique that ranks plausible patches produced by any APR technique according to their feature similarity with historic programmer-written fixes for similar bugs. PrevaRank implements simple heuristics, which help make it scalable and applicable to any APR tool that produces plausible patches. In our experimental evaluation, after training PrevaRank on the fix history of 81 open-source Java projects, we used it to rank patches produced by 8 Java APR tools on 168 Defects4J bugs. PrevaRank consistently improved the ranking of correct fixes: for example, it ranked a correct fix within the top-3 positions in 27% more cases than the original tools did. Other experimental results indicate that PrevaRank works robustly with a variety of APR tools and bugs, with negligible overhead. • Ranks APR patches via category-conditioned syntactic frequency patterns. • Improves top-3 correct-patch placement by ＋27% over tools’ native order. • Evaluated on 168 Defects4J bugs, 8 tools, 23,032 plausible patches.

Reinforcement Learning (RL) has gained significant attention in recent years. As RL software becomes more complex and infiltrates critical application domains, ensuring its quality and correctness becomes increasingly important. An indispensable aspect of software quality/correctness insurance is testing. However, testing RL software faces unique challenges compared to testing traditional software, due to the difficulty on defining the outputs' correctness. This leads to the RL test oracle problem. Current approaches to testing RL software often rely on human oracles, i.e. convening human experts to judge the correctness of RL software outputs. This heavily depends on the availability and quality (including the experiences, subjective states, etc.) of the human experts, and cannot be fully automated. In this paper, we propose a novel approach to design test oracles for RL software by leveraging the Lyapunov stability control theory. By incorporating Lyapunov stability concepts to guide RL training, we hypothesize that a correctly implemented RL software shall output an agent that respects Lyapunov stability control theories. Based on this heuristics, we propose a Lyapunov stability control theory based oracle,$\operatorname{LPEA}(\vartheta, \theta)$, for testing RL software. We conduct extensive experiments over representative RL algorithms and RL software bugs to evaluate our proposed oracle. The results show that our proposed oracle can outperform the human oracle in most metrics. Particularly, LPEA$(\vartheta=100 \%, \theta=75 \%)$outperforms the human oracle by$53.6 \%, 50 \%, 18.4 \%, 34.8 \%, 18.4 \%, 127.8 \%, 60.5 \%, 38.9 \%$, and 31.7 % respectively on accuracy, precision, recall, F1 score, true positive rate, true negative rate, false positive rate, false negative rate, and ROC curve's AUC; and LPEA ($\vartheta=100 \%, \theta=50 \%$) outperforms the human oracle by$48.2 \%, 47.4 \%, 10.5 \%, 29.1 \%$,$10.5 \%, 127.8 \%, 60.5 \%, 22.2 \%$, and 26.0 % respectively on these metrics.

Large-language models (LLMs) have been leveraged to enhance the capability of automated program repair techniques in recent research. While existing LLM-based program repair techniques compared favorably to other techniques based on heuristics, constraint-solving, and learning in producing high-quality patches, they mainly target bugs that can be corrected by changing a single faulty method, which greatly limits the effectiveness of such techniques in repairing bugs that demand patches spanning across multiple methods. In this work, we propose the PReMM technique to effectively propose patches changing multiple methods. PReMM builds on three core component techniques: the faulty method clustering technique to partition the faulty methods into clusters based on the dependence relationship among them, enabling a divide-and-conquer strategy for the repairing task; the fault context extraction technique to gather extra information about the fault context which can be utilized to better guide the diagnosis of the fault and the generation of correct patches; the dual-agent-based patch generation technique that employs two LLM-based agents with different roles to analyze the fault more precisely and generate patches of higher-quality. We have implemented the PReMM technique into a tool with the same name and applied the tool to repair real-world bugs from datasets Defects4J V1.2 and V2.0. PReMM produced correct patches for 307 bugs in total. Compared with ThinkRepair, the state-of-the-art LLM-based program repair technique, PReMM correctly repaired 102 more bugs, achieving an improvement of 49.8%.

Abstract Given the scale and complexity of large online service systems and the diversity of environments in which the services are to be invoked, it is inevitable that those service systems contain bugs that affect the users. As a result, it is essential for service providers to discover issues in their systems based on information gathered from users. iFeedback is a state-of-the-art technique for user-feedback-based issue detection. While it has been deployed to help detect issues in real-world service systems, the accuracy of iFeedback’s detection results is relatively low due to limitations in its design. In this paper, we propose theSkyNettechnique and tool that analyzes both user feedback gathered via specific channels and public posts collected from social media platforms to more accurately detect issues in service systems. We have applied the tool to detect issues for three real-world, large-scale online service systems based on their historical data gathered over a ten-month period of time.SkyNetreported in total 2790 issues, among which 93.0% were confirmed by developers as reflecting real problems that deserve their close attention. It also detected 58 out of the 62 severe issues reported during the period, achieving a recall of 93.5% for severe issues. Such results suggestSkyNetis both effective and accurate in issue detection.

Graphical User Interface (GUI) testing is one of the primary approaches for testing mobile apps. Test scripts serve as the main carrier of GUI testing, yet they are prone to obsolescence when the GUIs change with the apps' evolution. Existing repair approaches based on GUI layouts or images prove effective when the GUI changes between the base and updated versions are minor, however, they may struggle with substantial changes. In this paper, a novel approach named COSER is introduced as a solution to repairing broken scripts, which is capable of addressing larger GUI changes compared to existing methods. COSER incorporates both external semantic information from the GUI elements and internal semantic information from the source code to provide a unique and comprehensive solution. The efficacy of COSER was demonstrated through experiments conducted on 20 Android apps, resulting in superior performance when compared to the state-of-the-art tools METER and GUIDER. In addition, a tool that implements the COSER approach is available for practical use and future research.

While the received wisdom says that testing at levels like classes and methods is necessary for detecting bugs in programs, the application of unit testing to Android development in practice is limited so far due to the lack of sufficient technical and tool support. This paper proposes the EvoDroid approach to the automated unit test suite generation for Android code. EvoDroid is inspired by Evoobj, a SOTA test generation technique for object-oriented Java programs based on Evo-SUITE. EvoObj generates unit test suites for Java methods and constructs object construction graphs to guide the synthesis of complex objects as test inputs. In contrast to that, EvoDroid generates test suites for Java classes, and its object synthesis is driven by input structure maps which are comparably effective but much less expensive to construct. EvoDroid also integrates the Robolectric framework to support running Android unit tests on regular Java virtual machines. Experimental evaluation results show that EvoDroid is both effective and efficient in generating unit test suites for Android.

Today, Android applications (apps) have become ubiquitous in various aspects of life, highlighting the importance of graphical user interface (GUI) testing in ensuring their quality. To address the high cost associated with manual GUI testing, automated GUI test scripts are commonly developed. However, as Android apps evolve rapidly to offer more services or enhance existing ones, the GUI of these applications undergoes changes between the base and updated versions, leading to the obsolescence of existing GUI test scripts and increased maintenance costs. While existing repair approaches effectively manage minor GUI changes by replacing the locators of relevant widgets, they face challenges in addressing complex updates that bring significant GUI changes, thereby altering the interaction logic. This paper proposes a novel GUI test script repair approach to tackle these challenges. The approach utilizes an exploration and backtracking method to address the substantial GUI changes introduced by complex updates. By controlling the exploration space through exploration priority and pruning techniques, the approach achieves high effectiveness in repairing obsolete GUI tests caused by complex updates. We implemented our approach into a tool and evaluated its effectiveness and efficiency on 22 open-source Android apps and a total of 122 obsolete GUI test scripts. The experimental results showed that our approach can successfully repair 90% of the obsolete GUI test scripts.

The problem of automated feedback generation for introductory programming assignments (IPAs) has attracted significant attention with the increasing demand for programming education. While existing approaches, like Refactory, that employ the ”block-by-block” repair strategy have produced promising results, they suffer from two limitations. First, Refactory randomly applies refactoring and mutation operations to correct and buggy programs, respectively, to align their control-flow structures (CFSs), which, however, has a relatively low success rate and often complicates the original repairing tasks. Second, Refactory generates repairs for each basic block of the buggy program when its semantics differs from the counterpart in the correct program, which, however, ignores the different roles that basic blocks play in the programs and often produces unnecessary repairs. To overcome these limitations, we propose the Brafar approach to feedback generation for IPAs. The core innovation of Brafar lies in its novel bidirectional refactoring algorithm and coarse-to-fine fault localization. The former aligns the CFSs of buggy and correct programs by applying semantics-preserving refactoring operations to both programs in a guided manner, while the latter identifies basic blocks that truly need repairs based on the semantics of their enclosing statements and themselves. In our experimental evaluation on 1783 real-life incorrect student submissions from a publicly available dataset, Brafar significantly outperformed Refactory and Clara, generating correct repairs for more incorrect programs with smaller patch sizes in a shorter time.

Static analysis frameworks (SAFs) such as Soot and WALA have been a fundamental support in today’s software analysis. They usually adopt various analysis techniques to transform programs into different representations which imply specific properties, e.g., call graph can demonstrate the calling relationships between methods in a program, and users rely on these program representations for further analysis like vulnerability detection and privacy leakage recognition. Hence, providing proper program representation is essential for SAFs. We conducted a systematic empirical study on program representation faults of static analysis frameworks. In our study, we first collect 141 issues from four popular SAFs and summarize their root causes, symptoms, and fix strategies, and reveal nine findings and some implications to avoid and detect program representation faults. Additionally, we implemented an automated testing framework named SAScope based on the metamorphic and differential testing motivated by findings and implications. Overall, SAScope can detect 19 program representation faults where 6 of them have been confirmed or fixed, demonstrating its effectiveness.

Code summarization aims to generate short functional descriptions for source code to facilitate code comprehension. While Information Retrieval (IR) approaches that leverage similar code snippets and corresponding summaries have led the early research, Deep Learning (DL) approaches that use neural models to capture statistical properties between code and summaries are now mainstream. Although some preliminary studies suggest that IR approaches are more effective in some cases, it is currently unclear how effective the existing approaches can be in general, where and why IR/DL approaches perform better, and whether the integration of IR and DL can achieve better performance. Consequently, there is an urgent need for a comprehensive study of the IR and DL code summarization approaches to provide guidance for future development in this area. This article presents the first large-scale empirical study of 18 IR, DL, and hybrid code summarization approaches on five benchmark datasets. We extensively compare different types of approaches using automatic metrics, we conduct quantitative and qualitative analyses of where and why IR and DL approaches perform better, respectively, and we also study hybrid approaches for assessing the effectiveness of integrating IR and DL. The study shows that the performance of IR approaches should not be underestimated, that while DL models perform better in predicting tokens from method signatures and capturing structural similarities in code, simple IR approaches tend to perform better in the presence of code with high similarity or long reference summaries, and that existing hybrid approaches do not perform as well as individual approaches in their respective areas of strength. Based on our findings, we discuss future research directions for better code summarization.

Control-CPSs are usually safety or mission critical, hence they demand thorough debugging. As nowadays control-CPSs reaching millions of lines of source code, traditional human-flesh debugging is no longer sufficient. We need automated software fault localization (SFL) to assist the debugging. In automated SFL, automatically generated test cases are fed to the control-CPS (or the simulator of the control-CPS), to generate thousands of cyber-subsystem code traces and physical-subsystem trajectories. Next, another automated program, aka oracle, is needed to label the correctness of these physical-subsystem trajectories (and hence cyber-subsystem code traces), even without knowing if there is a bug in the cyber-subsystem. Control-CPS oracle design is a known hard problem. To our best knowledge, AR-SI oracle (denoted as AO in the following) is the most widely adopted control-CPS oracle so far. On the other hand, recently, transformer emerges as a major game changer in the domain of time series prediction. As AO is also time series prediction based, people naturally wonder if transformers can also be used as control-CPS oracles; and if so, can it outperform AO. In this paper, we answer this question by comparing AO with an intuitive design of transformer control-CPS oracle (simplified as TO in the following). Our comparison results show that in terms of SFL accuracy and latency, the TO does not significantly outperform the AO; in terms of false positive rate, the AO performs significantly better; and in terms of false negative rate, the TO performs significantly better.

Static analyzers reason about the behaviors of programs without executing them and report issues when they violate pre-defined desirable properties. One of the key limitations of static analyzers is their tendency to produce inaccurate and incomplete analysis results, i.e., they often generate too many spurious warnings and miss important issues. To help enhance the reliability of a static analyzer, developers usually manually write tests involving input programs and the corresponding expected analysis results for the analyzers. Meanwhile, a static analyzer often includes example programs in its documentation to demonstrate the desirable properties and/or their violations. Our key insight is that we can reuse programs extracted either from the official test suite or documentation and apply semantic-preserving transformations to them to generate variants. We studied the quality of input programs from these two sources and found that most rules in static analyzers are covered by at least one input program, implying the potential of using these programs as the basis for test generation. We present Statfier, a heuristic-based automated testing approach for static analyzers that generates program variants via semantic-preserving transformations and detects inconsistencies between the original program and variants (indicate inaccurate analysis results in the static analyzer). To select variants that are more likely to reveal new bugs, Statfier uses two key heuristics: (1) analysis report guided location selection that uses program locations in the reports produced by static analyzers to perform transformations and (2) structure diversity driven variant selection that chooses variants with different program contexts and diverse types of transformations. Our experiments with five popular static analyzers show that Statfier can find 79 bugs in these analyzers, of which 46 have been confirmed.

Preferences, the setting options provided by Android, are an essential part of Android apps. Preferences allow users to change app features and behaviors dynamically, and therefore their impacts need to be considered when testing the apps. Unfortunately, few test cases explicitly specify the assignments of valid values to the preferences, or configurations , under which they should be executed, and few existing mobile testing tools take the impact of preferences into account or provide help to testers in identifying and setting up the configurations for running the tests. This article presents the Prefest approach to effective testing of Android apps with preferences. Given an Android app and a set of test cases for the app, Prefest amplifies the test cases with a small number of configurations to exercise more behaviors and detect more bugs that are related to preferences. In an experimental evaluation conducted on real-world Android apps, amplified test cases produced by Prefest from automatically generated test cases covered significantly more code of the apps and detected seven real bugs, and the tool’s test amplification time was at the same order of magnitude as the running time of the input test cases. Prefest ’s effectiveness and efficiency in amplifying programmer-written test cases was comparable with that in amplifying automatically generated test cases.

A key challenge in generate-and-validate automated program repair is directing the search for fixes so that it can efficiently find those that are more likely to be correct. To this end, several techniques use machine learning to capture the features of programmer-written fixes. In existing approaches, fitting the model typically takes placebeforefix generation and is independent of it: the fix generation process uses the learned model as one of its inputs. However, the intermediate outcomes of an ongoing fix generation process often provide valuable information about which candidate fixes were “better”; this information could profitably be used to retrain the model, so that each new iteration of the fixing process would also learn from the outcome of previous ones. In this paper, we propose theLianatechnique for automated program repair, which is based on this idea ofrepeatedlylearning the features of generated fixes. To this end,Lianauses a fine-grained model that combines information about fix characteristics, their relations to the fixing context, and the results of test execution. The model is initially trained offline, and then repeatedly updated online as the fix generation process unravels; at any step, the most up-to-date model is used to guide the search for fixes—prioritizing those that are more likely to include the right ingredients. In an experimental evaluation on 732 real-world Java bugs from 3 popular benchmarks,Lianabuilt correct fixes for 134 faults (83 ranked as first in its output)— improving over several other generate-and-validate program repair tools according to various measures.

Modern large-scale service systems are usually deployed with redundant components to ensure high dependability in distributed and volatile environments. Fault Injection Testing (FIT) is a popular technique for testing such systems, while the application of FIT to validating the correctness of redundant components remains a challenging task, especially when the system's structural information is unavailable when testing starts. In this study, we refer to a minimum set of faults that, when injected, will cut off all execution paths in a service system as afault-tolerance bottleneck, and we propose a novel Fault-tolerance Bottleneck driven Fault Injection (FBFI) approach to the exploration and validation of redundant components without prior knowledge of the system's business structure. The core idea of FBFI is to iteratively infer and inject bottlenecks of the business structure constructed so far. In this way, FBFI is able to discover and test redundant components by repeatedly triggering new system behaviors. The effectiveness and efficiency of FBFI is evaluated using two microservice benchmark systems with different deployment scales. The results reveal that FBFI is more practical and cost-effective than random and lineage-driven FIT approaches in testing service systems of high redundancy levels.

Although software libraries promote code reuse and facilitate software development, they increase the complexity of programme analysis tasks. To effectively analyse programmes built on top of software libraries, it is essential to have specifications for the library methods that can be easily processed by analysis tools. However, the availability of such specifications is seriously limited at the moment. Manually writing the specifications can be prohibitively expensive and error-prone, while existing automated approaches to inferring the specifications seldom produce results that are strong enough to be used in programme analysis. In this work, we propose the DOC2SMT approach to generating strong functional constraints in SMT for library methods based on their documentations. DOC2SMT first applies natural language processing (NLP) techniques and a set of rules to translate a method's natural language documentation into a large number of candidate constraint clauses in OCL. Then, it utilises a manually enhanced domain model to identify OCL candidate constraint clauses that comply with the problem domain in static validation, translates well-formed OCL constraints into the SMT-LIB format, and checks whether each 5MB-LIB constraint rightly abstracts the functionalities of the method under consideration via testing in dynamic validation. In the end, it reports the first functional constraint that survives both validations to the user as the result. We have implemented the approach into a supporting tool with the same name. In experiments conducted on 451 methods from the Java Collections Framework and the Java IO library, DOC2SMT generated correct constraints for 309 methods, with the average generation time for each correct constraint being merely 2.7 min. We have also applied the generated constraints to facilitate symbolic-execution-based test generation with the Symbolic Java PathFinder (SPF) tool. For 24 utility methods manipulating Java container and IO objects, SPF with access to the generated constraints produced 51.2 times more test cases than SPF without the access.

With the growth of Issue Tracking Systems, issue reports have become an important data to aid software maintenance and evaluation. Issue classification is one of the most important methods for such purpose, which aims to automatically distinguish issues related to bugs from other issues via machine learning algorithm. However, existing issue classification approaches are still inadequate due to either the incorrect usages of the textual fields of the issues or the ineffective feature representation methods. In this paper, we propose a novel issue classification approach named DeepLabel for achieving advanced issue classification. DeepLabel predicts the issue types by the ensemble of field-specific models that are applied on different textual fields, so as to make the maximum use of the information contained in the textual fields. In addition, DeepLabel adopts Word2Vec combined with attention-based Bi-directional Long Short-Term Memory (ABLSTM) as the feature extractor for the field-specific models in order to effectively extract the semantic information from the textual fields. We conduct an empirical study to evaluate the effectiveness of DeepLabel based on a widely used issue dataset. The results demonstrate that DeepLabel can significantly outperform the state-of-the-art approaches, in which DeepLabel correctly identifies more bug issues (160.1 vs. 140.1) and more non-bug issues (345.7 vs. 325.4) on average compared to the best one existing approach.

The lifecycle models of Android components such as Activities and Fragments predefine the possible orders in which the components' callback methods will be invoked during app executions. Correspondingly, resource utilization operations performed by Android components must comply with all possible lifecycles to ensure safe utilization of the resources in all circumstances, which, however, can be challenging to achieve. In response to the challenge, various techniques have been developed to detect resource utilization bugs that manifest themselves when components go through common lifecycles, but the fact that Android components may execute their callback methods in uncommon orders, leading to variant component lifecycles, has largely been overlooked by the existing techniques. In this paper, we first identify three variant lifecycles for Android Activities and Fragments and then develop a technique called VALA to automatically detect bugs in Android apps that are induced by the variant lifecycles and may cause resource utilization errors like resource leaks and data losses. In an experimental evaluation conducted on 35 Android apps, a supporting tool for the VALA technique automatically detected 8 resource utilization bugs. All the 8 bugs were manually confirmed to be real defects and 7 of them were reported for the first time.

With the rapid development of Deep Learning, deep predictive models have been widely applied to improve Software Engineering tasks, such as defect prediction and issue classification, and have achieved remarkable success. They are mostly trained in a supervised manner, which heavily relies on high-quality datasets. Unfortunately, due to the nature and source of software engineering data, the real-world datasets often suffer from the issues of sample mislabelling and class imbalance, thus undermining the effectiveness of deep predictive models in practice. This problem has become a major obstacle for deep learning-based Software Engineering.

Graphical User Interface (GUI) testing is widely used to test mobile apps. As mobile apps are frequently updated and need repeated testing, to reduce the test cost, their test cases are often coded as scripts to enable automated execution using test harnesses/tools. When those mobile apps evolve, many of the test scripts, however, may become broken due to changes made to the app GUIs. While it is desirable that the broken scripts get repaired, doing it manually can be preventively expensive if the number of tests need repairing is large. We propose in this paper a novel approach namedMeterto repairing broken GUI test scripts automatically when mobile apps evolve.Meterleverages computer vision techniques to infer GUI changes between two versions of a mobile app and uses the inferred changes to guide the repair of GUI test scripts. SinceMeteronly relies on screenshots to repair GUI tests, it is applicable to apps targeting open or closed source mobile platforms. In experiments conducted on 22 Android apps and 6 iOS apps, repairs produced byMeterhelped preserve 63.7 and 38.8 percent of all the test actions broken by the GUI changes, respectively.

Fault localization is a crucial step of automated program repair, because accurately identifying program locations that are most closely implicated with a fault greatly affects the effectiveness of the patching process. An ideal fault localization technique would provide precise information while requiring moderate computational resources—to best support an efficient search for correct fixes. In contrast, most automated program repair tools use standard fault localization techniques—which are not tightly integrated with the overall program repair process, and hence deliver only subpar efficiency. In this paper, we presentretrospective fault localization: a novel fault localization technique geared to the requirements of automated program repair. A key idea of retrospective fault localization is to reuse the outcome of failed patch validation to support mutation-based dynamic analysis—providing accurate fault localization information without incurring onerous computational costs. We implemented retrospective fault localization in a tool calledRestore—based on theJaidJava program repair system. Experiments involving faults from theDefects4Jstandard benchmark indicate that retrospective fault localization can boost automated program repair:Restoreefficiently explores a large fix space, delivering state-of-the-art effectiveness (41Defects4Jbugs correctly fixed, 8 of which no other automated repair tool for Java can fix) while simultaneously boosting performance (speedup over 3 compared toJaid). Retrospective fault localization is applicable to any automated program repair techniques that rely on fault localization and dynamic validation of patches.

GUI testing is an essential part of regression testing for Android apps. For regression GUI testing to remain effective, it is important that obsolete GUI test scripts get repaired after the app has evolved. In this paper, we propose a novel approach named GUIDER to automated repair of GUI test scripts for Android apps. The key novelty of the approach lies in the utilization of both structural and visual information of widgets on app GUIs to better understand what widgets of the base version app become in the updated version. A supporting tool has been implemented for the approach. Experiments conducted on the popular messaging and social media app WeChat show that GUIDER is both effective and efficient. Repairs produced by GUIDER enabled 88.8% and 54.9% more test actions to run correctly than those produced by existing approaches to GUI test repair that rely solely on visual or structural information of app GUIs.

Summary Although software libraries promote code reuse and facilitate software development, they increase the complexity of programme analysis tasks. To effectively analyse programmes built on top of software libraries, it is essential to have specifications for the library methods that can be easily processed by analysis tools. However, the availability of such specifications is seriously limited at the moment. Manually writing the specifications can be prohibitively expensive and error‐prone, while existing automated approaches to inferring the specifications seldom produce results that are strong enough to be used in programme analysis. In this work, we propose the Doc2smt approach to generating strong functional constraints in SMT for library methods based on their documentations. Doc2smt first applies natural language processing (NLP) techniques and a set of rules to translate a method's natural language documentation into a large number of candidate constraint clauses in OCL. Then, it utilizes a manually enhanced domain model to identify OCL candidate constraint clauses that comply with the problem domain in static validation, translates well‐formed OCL constraints into the SMT‐LIB format, and checks whether each SMB‐LIB constraint rightly abstracts the functionalities of the method under consideration via testing in dynamic validation. In the end, it reports the first functional constraint that survives both validations to the user as the result. We have implemented the approach into a supporting tool with the same name. In experiments conducted on 451 methods from the Java Collections Framework and the Java IO library, Doc2smt generated correct constraints for 309 methods, with the average generation time for each correct constraint being merely 2.7 min. We have also applied the generated constraints to facilitate symbolic‐execution‐based test generation with the Symbolic Java PathFinder (SPF) tool. For 24 utility methods manipulating Java container and IO objects, SPF with access to the generated constraints produced 51.2 times more test cases than SPF without the access.

Most techniques for automated program repair (APR) use tests to drive the repair process; this makes them prone to generating spurious repairs that overfit the available tests unless additional information about expected program behavior is available. Our previous work onJaid, an APR technique for Java programs, showed that constructing detailed state abstractions—similar to those employed by techniques for programs with contracts—from plain Java code without any special annotations provides valuable additional information, and hence helps mitigate the overfitting problem. This paper extends the work onJaidwith a comprehensive experimental evaluation involving 693 bugs in three different benchmark suites. The evaluation shows, among other things, that: 1)Jaidis effective: it produced correct fixes for over 15 percent of all bugs, with a precision of nearly 60 percent; 2)Jaidis reasonably efficient: on average, it took less than 30 minutes to output a correct fix; 3)Jaidis competitive with the state of the art, as it fixed more bugs than any other technique, and 11 bugs that no other tool can fix; 4)Jaidis robust: its heuristics are complementary and their effectiveness does not depend on the fine-tuning of parameters. The experimental results also indicate the main trade-offs involved in designing an APR technique based on tests, as well as possible directions for further progress in this line of work.

Developers often neglect to handle exceptions, which leads to exception handling defects that affect the robustness of applications or even cause crashes. To improve the robustness of android applications while reducing the development burden of developers, we present Fixeh and Automatic Detection Tool, as an approach that can automatically detect exception handling defects related to external resources. By implanting exception control codes into the input application, Fixeh helps applications throw exceptions at the specified call position while running the UI test. During running the UI test, Automatic Detection Tool generates a limited number of exception trigger patterns by using suspicious call filtering algorithm and traversal algorithm. After collecting and analyzing the running results under these patterns, the exception handling defects will be detected. We evaluate our approach by applying it to detect anomalies in 6 different types of applications with stable operation. We conducted 1422 rounds of experiments under different exception triggering patterns, and we observed abnormalities in 517 rounds. A comparison with other related work shows that our approach can detect defects more effectively. Through the analysis of our experiments, we confirmed 39 exception handling defects related to external resources. Finally, we summarized three common types of defects from them.

Mobile apps often need to collect and/or access sensitive user information to fulfill their purposes, but they may also leak such information either intentionally or accidentally, causing financial and/or emotional damages to users. In the past few years, researchers have developed various techniques to detect privacy leakage in mobile apps, however, such detection remains a challenging task when privacy leakage is implemented via dynamic class loading (DCL). In this work, we propose the DL2technique that enhances static analysis with dynamic app execution to effectively detect privacy leakage implemented via DCL in Android apps. To evaluate DL2, we construct a benchmark of 88 subject apps with 2578 injected privacy leaks and apply DL2to the apps. DL2was able to detect 1073, or 42%, of the leaks, significantly outperforming existing state-of-the-art privacy leakage detection tools.

Control-CPS software fault localization (SFL, aka bug localization) is of critical importance as bugs may cause major failures, even injuries/deaths. To locate the bugs in control-CPSs, SFL tools often demand many labeled ("correct"/"incorrect") source code execution traces as inputs. To label the correctness of these traces, we must judge the corresponding control-CPS physical trajectories' correctness. However, unlike discrete outputs, the boundaries between correct and incorrect physical trajectories are often vague. The mechanism (aka oracle) to judge the physical trajectories' correctness thus becomes a major challenge. So far, the ad hoc practice of ``human oracles'' is still widely used, whose qualities heavily depend on the human experts' expertise and availability. This paper proposes an oracle based on the well adopted autoregressive system identification (AR-SI). With proven success for controlling black-box physical systems, AR-SI is adapted by us to identify the buggy control-CPS as a black-box. We use this identification result as an oracle to judge the control-CPS's behaviors, and propose a methodology to prepare traces for control-CPS debugging. Comprehensive evaluations on classic control-CPSs with injected real-life and artificial bugs show that our proposed approach significantly outperforms the human oracle approach in SFL accuracy (recall) and latency, and in oracle false positive/negative rates. Our approach also helps discover a new real-life bug in a consumer-grade control-CPS.

The widespread real-time and embedded systems are mostly interrupt-driven because their heavy interaction with the environment is often initiated by interrupts. With the interrupt arrival being unpredictable and the interrupt handling being preemptive, a large number of possible system behaviours are generated, which makes the correctness assurance of such systems difficult and costly. Model checking is considered to be one of the effective methods for exhausting behavioural state space for correctness. However, existing modelling approaches for interrupt-driven systems are based on either calculus or automata theory, and have a steep learning curve. To address this problem, we propose a new modelling language called interrupt sequence diagram (ISD). By extending the popular UML sequence diagram notations, the ISD supports the modelling of interrupts' essential features visually and concisely. We also propose an automata-based semantics for ISD, based on which ISD can be transformed to a subset of hybrid automata so as to leverage the abundant off-the-shelf checkers. Experiments on examples from both real-world and existing literature were conducted, and the results demonstrate our approach's usability and effectiveness.

In regression GUI testing for Android apps, test scripts often fail due to changes to, rather than faults in, those apps. To avoid such false positives while still retaining the value of the old test scripts as much as possible, programmers need an automatic way to maintain the tests after the corresponding GUI has evolved. In this paper, we propose the CHATEM approach to automate GUI test script maintenance for Android apps. Taking as input the models for the GUIs of the base and updated version app and the original test scripts, CHATEM automatically extracts the changes between the two GUIs and generates maintenance actions for each change, which are then combined to form the maintenance actions for affected test scripts. In an experimental evaluation on 16 Android apps, CHATEM was able to automatically maintain the test scripts so that overall more than 95% of the remaining behaviors tested before are still tested, and almost 80% of the reusable test actions are retained in the result tests.

As the quality and quantity of open source code increase, semantics-based code search has become an emerging need for software developers to retrieve and reuse existing source code. We present an approach of semantics-based code search using input/output examples for the Java language. Our approach encodes Java methods in code repositories into path constraints via symbolic analysis and leverages SMT solvers to find the methods whose path constraints can satisfy the given input/output examples. Our approach extends the applicability of the semantics-based search technology to more general Java code compared with existing methods. To evaluate our approach, we encoded 1228 methods from GitHub and applied semantics-based code search on 35 queries extracted from Stack Overflow. Correct method code for 29 queries was obtained during the search and the average search time was just about 48 seconds.

The importance of regression testing in assuring the integrity of a program after changes is well recognized. One major obstacle in practicing regression testing is in maintaining tests that become obsolete due to evolved program behavior or specification. For mobile apps, the problem of maintaining obsolete GUI test scripts for regression testing is even more pressing. Mobile apps rely heavily on the correct functioning of their GUIs to compete on the market and provide good user experiences. But on the one hand, GUI tests break easily when changes happen to the GUI, On the other hand, mobile app developers often need to fight for a tight feedback loop and are left with limited time for test maintenance. In this paper, we propose a novel approach, called ATOM, to automatically maintain GUI test scripts of mobile apps for regression testing. ATOM uses an event sequence model to abstract possible event sequences on a GUI and a delta ESM to abstract the changes made to the GUI. Given both models as input, ATOM automatically updates the test scripts written for a base version app to reflect the changes. In an experiment with 22 versions from 11 production Android apps, ATOM updated all the test scripts affected by the version change, the updated scripts achieve over 80% of the coverage by the original scripts on the base version app, all except one set of updated scripts preserve over 60% of the actions in the original test scripts.

Automated program repair (APR) is a promising approach to automatically fixing software bugs. Most APR techniques use tests to drive the repair process; this makes them readily applicable to realistic code bases, but also brings the risk of generating spurious repairs that overfit the available tests. Some techniques addressed the overfitting problem by targeting code using contracts (such as pre- and postconditions), which provide additional information helpful to characterize the states of correct and faulty computations; unfortunately, mainstream programming languages do not normally include contract annotations, which severely limits the applicability of such contract-based techniques. This paper presents JAID, a novel APR technique for Java programs, which is capable of constructing detailed state abstractions-similar to those employed by contract-based techniques-that are derived from regular Java code without any special annotations. Grounding the repair generation and validation processes on rich state abstractions mitigates the overfitting problem, and helps extend APR's applicability: in experiments with the DEFECTS4J benchmark, a prototype implementation of JAID produced genuinely correct repairs, equivalent to those written by programmers, for 25 bugs-improving over the state of the art of comparable Java APR techniques in the number and kinds of correct fixes.

We present the integration of the AutoFix automated program repair technique into the EiffelStudio Development Environment. AutoFix presents itself like a recommendation system capable of automatically finding bugs and suggesting fixes in the form of source-code patches. Its performance suggests usage scenarios where it runs in the background or during work interruptions, displaying fix suggestions as they become available. This is a contribution towards the vision of semantic Integrated Development Environments, which offer powerful automated functionality within interfaces familiar to developers. A screencast highlighting the main features of AutoFix can be found at: http://youtu.be/Ff2ULiyL-80.

This paper describes AutoFix, an automatic debugging technique that can fix faults in general-purpose software. To provide high-quality fix suggestions and to enable automation of the whole debugging process, AutoFix relies on the presence of simple specification elements in the form of contracts (such as pre- and postconditions). Using contracts enhances the precision of dynamic analysis techniques for fault detection and localization, and for validating fixes. The only required user input to the AutoFix supporting tool is then a faulty program annotated with contracts; the tool produces a collection of validated fixes for the fault ranked according to an estimate of their suitability. In an extensive experimental evaluation, we applied AutoFix to over 200 faults in four code bases of different maturity and quality (of implementation and of contracts). AutoFix successfully fixed 42 percent of the faults, producing, in the majority of cases, corrections of quality comparable to those competent programmers would write; the used computational resources were modest, with an average time per fix below 20 minutes on commodity hardware. These figures compare favorably to the state of the art in automated program fixing, and demonstrate that the AutoFix approach is successfully applicable to reduce the debugging burden in real-world scenarios.

Experience with lightweight formal methods suggests that programmers are willing to write specification if it brings tangible benefits to their usual development activities. This paper considers stronger specifications and studies whether they can be deployed as an incremental practice that brings additional benefits without being unacceptably expensive. We introduce a methodology that extends Design by Contract to write strong specifications of functional properties in the form of preconditions, postconditions, and invariants. The methodology aims at being palatable to developers who are not fluent in formal techniques but are comfortable with writing simple specifications. We evaluate the cost and the benefits of using strong specifications by applying the methodology to testing data structure implementations written in Eiffel and C#. In our extensive experiments, testing against strong specifications detects twice as many bugs as standard contracts, with a reasonable overhead in terms of annotation burden and run-time performance while testing. In the wide spectrum of formal techniques for software quality, testing against strong specifications lies in a “sweet spot” with a favorable benefit to effort ratio.

Initial research in automated program fixing has generally limited itself to specific areas, such as data structure classes with carefully designed interfaces, and relied on simple approaches. To provide high-quality fix suggestions in a broad area of applicability, the present work relies on the presence of contracts in the code, and on the availability of static and dynamic analyses to gather evidence on the values taken by expressions derived from the code. The ideas have been built into the AutoFix-E2 automatic fix generator. Applications of AutoFix-E2 to general-purpose software, such as a library to manipulate documents, show that the approach provides an improvement over previous techniques, in particular purely model-based approaches.

Automated random testing has shown to be an effective approach to finding faults but still faces a major unsolved issue: how to generate test inputs diverse enough to find many faults and find them quickly. Stateful testing, the automated testing technique introduced in this article, generates new test cases that improve an existing test suite. The generated test cases are designed to violate the dynamically inferred contracts (invariants) characterizing the existing test suite. As a consequence, they are in a good position to detect new faults, and also to improve the accuracy of the inferred contracts by discovering those that are unsound. Experiments on 13 data structure classes totalling over 28,000 lines of code demonstrate the effectiveness of stateful testing in improving over the results of long sessions of random testing: stateful testing found 68.4% new faults and improved the accuracy of automatically inferred contracts to over 99%, with just a 7% time overhead.

In program debugging, finding a failing run is only the first step; what about correcting the fault? Can we automate the second task as well as the first? The AutoFix-E tool automatically generates and validates fixes for software faults. The key insights behind AutoFix-E are to rely on contracts present in the software to ensure that the proposed fixes are semantically sound, and on state diagrams using an abstract notion of state based on the boolean queries of a class. Out of 42 faults found by an automatic testing tool in two widely used Eiffel libraries, AutoFix-E proposes successful fixes for 16 faults. Submitting some of these faults to experts shows that several of the proposed fixes are identical or close to fixes proposed by humans.