History-driven testing techniques have been proven to be an effective method for detecting compiler bugs. It employs fuzzing history (e.g., historical test cases or historical execution information) to guide to generate valid test cases. However, prior methods either have an inefficient capability in synthesizing bug-triggering test cases or suffer from a plateau of code coverage, causing a low bug-exposing ability. This paper presents ASMFUZZ, another history-driven compiler testing framework by applying a multi-metric hybrid scheduling strategy. Specifically, ASMFUZZ first extracts the bug-triggering code segments from the historical test cases that triggered bugs. The extracted bug-triggering code segments are then used to assemble new test cases. To ensure the correctness of the newly synthesized test cases, ASMFUZZ always selects the segments with code context dependencies for assembly. Duration assembly, the ingredients to be assembled are determined based on multiple feedback metrics (e.g., anomalous behaviors and code coverage). To do so, we proposed a multi-metric hybrid scheduling scheme to select optimal code segments in each testing iteration. This contributes to continuously covering deep code branches of compiler duration whole testing process, avoiding getting stuck in the plateau of code coverage. We evaluated ASMFUZZ on three mainstream JVMs including OpenJ9, HotSpot, and GraalVM involving six JDK versions. Within a 72-hour concurrent test run, ASMFUZZ exposed 16 previously unknown unique bugs, of which 11 have been confirmed by the developers. We also compared ASMFUZZ to four prior state-of-the-art fuzzers. ASMFUZZ uncovers 1.6~2.2× more bugs than comparative baselines.

High-level programming models like Q# significantly simplify the complexity of programming for quantum computing. These models are supported by a set of foundation libraries for code development. However, errors can occur in the library implementation, and one common root cause is the lack of or incomplete checks on properties like values, length, and quantum states of inputs passed to user-facing subroutines. This paper presents Upbeat, a fuzzing tool to generate random test cases for bugs related to input checking in Q# libraries. Upbeat develops an automated process to extract constraints from the API documentation and the developer implemented input-checking statements. It leverages open-source Q# code samples to synthesize test programs. It frames the test case generation as a constraint satisfaction problem for classical computing and a quantum state model for quantum computing to produce carefully generated subroutine inputs to test if the input-checking mechanism is appropriately implemented. Under 100 hours of automated test runs, Upbeat has successfully identified 16 bugs in API implementations and 4 documentation errors. Of these, 14 have been confirmed, and 12 have been fixed by the library developers.

Random test case generation, or fuzzing, is a viable means for uncovering compiler bugs. Unfortunately, compiler fuzzing can be time-consuming and inefficient with purely randomly generated test cases due to the complexity of modern compilers. We present COMFUZZ, a focused compiler fuzzing framework. COMFUZZ aims to improve compiler fuzzing efficiency by focusing on testing components and language features that are likely to trigger compiler bugs. Our key insight is human developers tend to make common and repeat errors across compiler implementations; hence, we can leverage the previously reported buggy-exposing test cases of a programming language to test a new compiler implementation. To this end, COMFUZZ employs deep learning to learn a test program generator from open-source projects hosted on GitHub. With the machine-generated test programs in place, COMFUZZ then leverages a set of carefully designed mutation rules to improve the coverage and bug-exposing capabilities of the test cases. We evaluate COMFUZZ on 11 compilers for JS and Java programming languages. Within 260 hours of automated testing runs, we discovered 33 unique bugs across nine compilers, of which 29 have been confirmed and 22, including an API documentation defect, have already been fixed by the developers. We also compared COMFUZZ to eight prior fuzzers on four evaluation metrics. In a 24-hour comparative test, COMFUZZ uncovers at least 1.5× more bugs than the state-of-the-art baselines.