Garbage Collection (GC) in the Java Virtual Machine (JVM) serves as an automatic memory management mechanism, efficiently reclaiming unused memory space in different production scenarios. To optimize JVM performance, developers typically fine-tune the garbage collector by identifying an optimal set of GC configurations for specific scenarios. Despite the sophisticated design of garbage collectors, they still have the potential for bugs in different settings, and these bugs can result in more severe consequences. Hence, comprehensive testing of these garbage collectors is imperative before their release. Code coverage criteria are typically employed to assess the comprehensiveness of a test suite. However, traditional code coverage metrics, such as branch coverage, are hardly applicable for GC testing due to their inherent concurrency. Additionally, existing JVM testing techniques do not adequately consider the characteristics of garbage collectors, making it difficult to test these garbage collectors sufficiently. In this article, we make the first effort to design coverage criteria against garbage collectors based on the events of GC called Event-Coverage . Its key insight is to measure the diversity of GC executions for testing purposes by assessing the range of GC events these executions cover. Furthermore, we design a new testing method for maximizing Event-Coverage called GCFuzz . GCFuzz conducts an exhaustive investigation of the memory state space of GC and thoroughly explores the memory state under various GC configurations. To enhance GCFuzz’s efficiency in achieving higher Event-Coverage, we have further designed a coverage-driven strategy for preserving candidate seed programs and selecting GC configurations. Extensive evaluations demonstrate a positive correlation between Event-Coverage and the bug-revealing efficiency. Moreover, GCFuzz outperforms state-of-the-art techniques in detecting unique GC-related inconsistencies and achieving higher Event-Coverage. Remarkably, GCFuzz has identified 20 previously undetected GC bugs, with 15 of them already confirmed or fixed by developers.

JVM fuzzing techniques serve as a cornerstone for guaranteeing the quality of implementations. In typical fuzzing workflows, initial seeds are crucial as they form the basis of the process. Literature in traditional program fuzzing has confirmed that effectiveness is largely impacted by redundancy among initial seeds, thereby proposing a series of seed selection methods. JVM fuzzing, compared to traditional ones, presents unique characteristics, including large-scale and intricate code, and programs with both syntactic and semantic features. However, it remains unclear whether the existing initial seed selection methods are suitable for JVM fuzzing and whether utilizing program features can enhance effectiveness. To address this, we devise a total of 10 initial seed selection methods, comprising coverage-based, prefuzz-based, and program-feature-based methods. We then conduct an empirical study on three JVM implementations to extensively evaluate the performance of the initial seed selection methods within two state-of-the-art fuzzing techniques (JavaTailor and VECT). Specifically, we examine performance from three aspects: (i) effectiveness and efficiency using widely studied initial seeds, (ii) effectiveness using the programs in the wild, and (iii) the ability to detect new bugs. Evaluation results first show that the program-feature-based method that utilizes the control flow graph not only has a significantly lower time overhead (i.e., 30s), but also outperforms other methods, achieving 142% to 269% improvement compared to the full set of initial seeds. Second, results reveal that the initial seed selection greatly improves the quality of wild programs and exhibits complementary effectiveness by detecting new behaviors. Third, results demonstrate that given the same testing period, initial seed selection improves the JVM fuzzing techniques by detecting more unknown bugs. Particularly, 21 out of the 25 detected bugs have been confirmed or fixed by developers. This work takes the first look at initial seed selection in JVM fuzzing, confirming its importance in fuzzing effectiveness and efficiency.

Java Virtual Machine (JVM) holds a crucial position in executing various Java programs, thereby necessitating rigorous testing to ensure software reliability and security. Regarding existing JVM testing techniques, synthesis-based techniques have proven to be state-of-the-art, which construct a test program by synthesizing various program ingredients extracted from historical bug-revealing test programs into a seed program. However, existing synthesis-based techniques directly use the program ingredients specific to historical bugs, which limits the test scope without the ability of covering more JVM features and negatively affects the diversity of synthesized test programs.

MLIR (Multi-Level Intermediate Representation) compiler infrastructure has gained widespread popularity in recent years. It introduces dialects to accommodate various levels of abstraction within the representation. Due to its fundamental role in compiler construction, it is critical to ensure its correctness. Recently, a grammar-based fuzzing technique (i.e., MLIRSmith) has been proposed for it and achieves notable effectiveness. However, MLIRSmith generates test programs in a random manner, which restricts the exploration of the input space, thereby limiting the overall fuzzing effectiveness. In this work, we propose a novel fuzzing technique, called MLIR. As complicated or uncommon data/control dependencies among various operations are often helpful to trigger MLIR bugs, it constructs the operation dependency graph for an MLIR program and defines the associated operation dependency coverage to guide the fuzzing process. To drive the fuzzing process towards increasing operation dependency coverage, MLIR then designs a set of dependency-targeted mutation rules. By applying MLIR to the latest revisions of the MLIR compiler infrastructure, it detected 63 previously unknown bugs, among which 38/48 bugs have been fixed/confirmed by developers.

Abstract Smart contracts are distributed self‐enforcing programs which execute on top of blockchain networks. They have the potential to revolutionize many industries and have already been adopted for applications such as distributed finance and crowdfunding. Because smart contracts are immutable once they are deployed, it is important to identify and eliminate code vulnerabilities in smart contracts systematically. In this work, we propose sFuzz2.0, a storage‐access‐pattern guided adaptive fuzzer based on sFuzz. sFuzz2.0 is motivated by the fact that certain vulnerabilities only manifest in the presence of certain function call sequences (as well as particular arguments). Given that there are exponentially many function call sequences, sFuzz randomly generates sequences without guidance. As a result, the probability of discovering those vulnerabilities is negligible. sFuzz2.0 tackles the problem with two approaches, that is, by generating function call sequences that trigger different storage‐access patterns passively (i.e., by prioritizing seeds which cover new patterns) or actively (i.e., by actively seeking out different patterns). The experiment results suggest that the passive strategy outperforms sFuzz by achieving better code coverage (i.e., 37.53%) and discovering more vulnerabilities (i.e., 20.49%).

JVM testing is one of the most widely-used methodologies for guaranteeing the quality of JVMs. Among various JVM testing techniques, synthesis-based JVM testing, which constructs a test program by synthesizing various code snippets (also called program ingredients), has been demonstrated state-of-the-art. The existing synthesis-based JVM testing work puts more efforts in ensuring the validity of synthesized test programs, but ignores the influence of huge ingredient space, which largely limits the ingredient exploration efficiency as well as JVM testing performance. In this work, we propose Vectorized JVM Testing (called VECT) to further promote the performance of synthesis-based JVM testing. Its key insight is to reduce the huge ingredient space by clustering semantically similar ingredients via vectorizing ingredients using state-of-the-art code representation. To make VECT complete and more effective, based on vectorized ingredients, VECT further designs a feedback-driven ingredient selection strategy and an enhanced test oracle. We conducted an extensive study to evaluate VECT on three popular JVMs (i.e., HotSpot, OpenJ9, and Bisheng JDK) involving five OpenJDK versions. The results demonstrate VECT detects 115.03% ~ 776.92% more unique inconsistencies than the state-of-the-art JVM testing technique during the same testing time. In particular, VECT detects 26 previously unknown bugs for them, 15 of which have already been confirmed/fixed by developers.

Due to the critical role of compilers, many compiler testing techniques have been proposed, two most notable categories among which are grammar-based and metamorphic-based techniques. All of them have been extensively studied for testing mature compilers. However, it is typical to develop a new compiler for a new-born programming language in practice. In this scenario, the existing techniques are hardly applicable due to some major reasons: (1) no reference compilers to support differential testing, (2) lack of program analysis tools to support most of metamorphic-based compiler testing, (3) substantial implementation effort incurred by different programming language features. Hence, it is unknown how the existing techniques perform in this new scenario.

MLIR (Multi-Level Intermediate Representation) compiler infrastructure has gained popularity in recent years to support the construction of many compilers. Instead of designing a new IR with a single abstraction for each domain, MLIR compiler infrastructure provides systematic passes to support a wide range of functionalities for benefiting multiple domains together and introduces dialects to support different levels of abstraction in MLIR. Due to its fundamental role in compiler community, ensuring its quality is very critical. In this work, we propose MLIRSmith, the first fuzzing technique for MLIR compiler infrastructure. MLIRSmith employs a two-phase strategy to generate valid and diverse MLIR programs, which first constructs diverse program templates guided by extended MLIR syntax rules and then generates valid MLIR programs through template instantiation guided by our designed context-sensitive grammar. After applying MLIRSmith to the latest revision of MLIR compiler infrastructure, we detected 53 previously unknown bugs, among which 49/38 have been confirmed/fixed by developers. We also transform the high-level programs generated by NNSmith (a high-level program generator for deep learning compilers) to MLIR programs for indirectly fuzzing MLIR compiler infrastructure. During the same testing time, MLIRSmith largely outperforms such an indirect technique by detecting 328.57% more bugs and covering 194.67%/225.87% more lines/branches in MLIR compiler infrastructure.

While the Java Virtual Machine (JVM) plays a vital role in ensuring correct executions of Java applications, testing JVMs via generating and running class files on them can be rather challenging. The existing techniques, e.g., ClassFuzz and Classming, attempt to leverage the power of fuzzing and differential testing to cope with JVM intricacies by exposing discrepant execution results among different JVMs, i.e., inter-JVM discrepancies, for testing analytics. However, their adopted fuzzers are insufficiently guided since they include no well-designed seed and mutator scheduling mechanisms, leading to inefficient differential testing. To address such issues, in this paper, we propose SJFuzz, the first JVM fuzzing framework with seed and mutator scheduling mechanisms for automated JVM differential testing. Overall, SJFuzz aims to mutate class files via control flow mutators to facilitate the exposure of inter-JVM discrepancies. To this end, SJFuzz schedules seeds (class files) for mutations based on the discrepancy and diversity guidance. SJFuzz also schedules mutators for diversifying class file generation. To evaluate SJFuzz, we conduct an extensive study on multiple representative real-world JVMs, and the experimental results show that SJFuzz significantly outperforms the state-of-the-art mutation-based and generation-based JVM fuzzers in terms of the inter-JVM discrepancy exposure and the class file diversity. Moreover, SJFuzz successfully reported 46 potential JVM issues, and 20 of them have been confirmed as bugs and 16 have been fixed by the JVM developers.

Testing multi-threaded programs is challenging due to the enormous space of thread interleavings. Recently, a code coverage criterion for multi-threaded programs called MAP-coverage has been proposed and shown to be effective for testing concurrent programs. Existing approaches for achieving high MAP-coverage are based on random testing with simple heuristics, which is ineffective in systematically triggering rare thread interleavings. In this study, we propose a novel approach called pattern constraint reduction (PCR), which employs optimized constraint solving to generate thread interleavings for high MAP-coverage. The idea is to iteratively encode and solve path conditions to generate thread interleavings which are guaranteed to improve MAP-coverage. Furthermore, we effectively apply interpolation techniques to reduce the efforts of constraint solving by avoiding solving infeasible constraints. The experiment results on 20 benchmark programs show that our approach complements existing random testing based approaches when there are rare failure-inducing interleaving in the whole search space. Specifically, PCR finds concurrency bugs faster in 18 out of 20 programs, with an average speedup of 4.2x and a maximum speedup of 11.4x.

Java Virtual Machine (JVM) provides the runtime environment for Java programs, which allows Java to be "write once, run anywhere". JVM plays a decisive role in the correctness of all Java programs running on it. Therefore, ensuring the correctness and robustness of JVM implementations is essential for Java programs. To date, various techniques have been proposed to expose JVM bugs via generating potential bug-revealing test programs. However, the diversity and effectiveness of test programs generated by existing research are far from enough since they mainly focus on minor syntactic/semantic mutations. In this paper, we propose JavaTailor, the first history-driven test program synthesis technique, which synthesizes diverse test programs by weaving the ingredients extracted from JVM historical bug-revealing test programs into seed programs for covering more JVM behaviors/paths. More specifically, JavaTailor first extracts five types of code ingredients from the historical bug-revealing test programs. Then, to synthesize diverse test programs, it iteratively inserts the extracted ingredients into the seed programs and strengthens their interactions via introducing extra data dependencies between them. Finally, JavaTailor employs these synthesized test programs to differentially test JVMs. Our experimental results on popular JVM implementations (i.e., HotSpot and OpenJ9) show that JavaTailor outperforms the state-of-the-art technique in generating more diverse and effective test programs, e.g., test programs generated by JavaTailor can achieve higher JVM code coverage and detect many more unique inconsistencies than the state-of-the-art technique. Furthermore, JavaTailor has detected 10 previously unknown bugs, 6 of which have been confirmed/fixed by developers.

Security vulnerability prediction (SVP) can identify potential vulnerable modules in advance and then help developers to allocate most of the test resources to these modules. To evaluate the performance of different SVP methods, we should take the security audit and code inspection into account and then consider effort-aware performance measures (such as ACC and Popt). However, to the best of our knowledge, the effectiveness of different SVP methods has not been thoroughly investigated in terms of effort-aware performance measures. In this article, we consider 48 different SVP methods, of which 36 are supervised methods and 12 are unsupervised methods. For the supervised methods, we consider 34 software-metric-based methods and two text-mining-based methods. For the software-metric-based methods, in addition to a large number of classification methods, we also consider four state-of-the-art methods (i.e., EALR, OneWay, CBS, and MULTI) proposed in recent effort-aware just-in-time defect prediction studies. For text-mining-based methods, we consider the Bag-of-Word model and the term-frequency-inverse-document-frequency model. For the unsupervised methods, all the modules are ranked in the ascendent order based on a specific metric. Since 12 software metrics are considered when measuring extracted modules, there are 12 different unsupervised methods. To the best of our knowledge, over 40 SVP methods have not been considered in previous SVP studies. In our large-scale empirical studies, we use three real open-source web applications written in PHP as benchmark. These three web applications include 3466 modules and 223 vulnerabilities in total. We evaluate these SVP methods both in the within-project SVP scenario and the cross-project SVP scenario. Empirical results show that two unsupervised methods [i.e., lines of code (LOC) and Halstead's volume (HV)] and four recently proposed state-of-the-art supervised methods (i.e., MULTI, OneWay, CBS, and EALR) can achieve better performance than the other methods in terms of effort-aware performance measures. Then, we analyze the reasons why these six methods can achieve better performance. For example, when using 20% of the entire efforts, we find that these six methods always require more modules to be inspected, especially for unsupervised methods LOC and HV. Finally, from the view of practical vulnerability localization, we find that all the unsupervised methods and the OneWay method have high false alarms before finding the first vulnerable module. This may have an impact on developers' confidence and tolerance, and supervised methods (especially MULTI and text-mining-based methods) are preferred.

Multi-threaded programs often exhibit erroneous behaviours due to unintended interactions among threads. Those bugs are often difficult to find because they typically manifest under very specific thread schedules. The traditional randomized algorithms increase the probability of exploring infrequent interleavings using randomized scheduling and improve the chances of detecting concurrency defects. However, they may generate many redundant trials, especially for those hard-to-detect defects, and thus their performance is often not stable. In this work, we propose an adaptive randomized scheduling algorithm~(ARS), which adaptively explores the search space and detects concurrency bugs more efficiently with less efforts. We compare ARS with random searching and the state-of-the-art maximal causality reduction method on 27 concurrent Java programs. The evaluation results show that ARS shows a more stable performance in terms of effectiveness in detecting multi-threaded bugs. Particularly, ARS shows a good potential in detecting hard-to-expose bugs.

Concurrent programs must be thoroughly tested, as concurrency bugs are notoriously hard to detect. Code coverage criteria can be used to quantify the richness of a test suite (e.g., whether a program has been tested sufficiently) or provide practical guidelines on test case generation (e.g., as objective functions used in program fuzzing engines). Traditional code coverage criteria are, however, designed for sequential programs and thus ineffective for concurrent programs. In this work, we introduce a novel code coverage criterion for testing thread-safe classes called MAP-coverage (short for memory-access patterns). The motivation is that concurrency bugs are often correlated with certain memory-access patterns, and thus it is desirable to comprehensively cover all memory-access patterns. Furthermore, we propose a testing method for maximizing MAP-coverage. Our method has been implemented as a self-contained toolkit, and the experimental results on 20 benchmark programs show that our toolkit outperforms existing testing methods. Lastly, we show empirically that there exists positive correlation between MAP-coverage and the effectiveness of a set of test executions.

Abstract Within‐project defect prediction assumes that we have sufficient labeled data from the same project, while cross‐project defect prediction assumes that we have plenty of labeled data from source projects. However, in practice, we might only have limited labeled data from both the source and target projects in some scenarios. In this paper, we want to apply multitask learning to investigate such a new scenario. To our best knowledge, this problem (ie, both the source project and the target project have limited labeled data) has not been thoroughly investigated, and we are the first to propose a novel multitask defect prediction approach mask. mask consists of a differential evolution optimization phase and a multitask learning phase. The former phase aims to find optimal weights for shared and nonshared information in related projects (ie, the target project and its related source projects), while the latter phase builds prediction models for each project simultaneously. To verify the effectiveness of mask, we perform experimental studies on 18 real‐world software projects and compare our approach with four state‐of‐the‐art baseline approaches: single‐task learning (STL), simple combined learning (SCL), Peters filter, and Burak filter. Experimental results show that mask can achieve F1 of 0.397 and AUC of 0.608 on average with a few labeled data (ie, 10% of data). Across the 18 projects, mask can outperform baseline methods significantly in terms of F1 and AUC. Therefore, by utilizing the relatedness among multiple projects, mask can perform significantly better than the state‐of‐the‐art methods. The results confirm that mask is promising for software defect prediction when the source and target projects both have limited training data.