History-driven testing techniques have been proven to be an effective method for detecting compiler bugs. It employs fuzzing history (e.g., historical test cases or historical execution information) to guide to generate valid test cases. However, prior methods either have an inefficient capability in synthesizing bug-triggering test cases or suffer from a plateau of code coverage, causing a low bug-exposing ability. This paper presents ASMFUZZ, another history-driven compiler testing framework by applying a multi-metric hybrid scheduling strategy. Specifically, ASMFUZZ first extracts the bug-triggering code segments from the historical test cases that triggered bugs. The extracted bug-triggering code segments are then used to assemble new test cases. To ensure the correctness of the newly synthesized test cases, ASMFUZZ always selects the segments with code context dependencies for assembly. Duration assembly, the ingredients to be assembled are determined based on multiple feedback metrics (e.g., anomalous behaviors and code coverage). To do so, we proposed a multi-metric hybrid scheduling scheme to select optimal code segments in each testing iteration. This contributes to continuously covering deep code branches of compiler duration whole testing process, avoiding getting stuck in the plateau of code coverage. We evaluated ASMFUZZ on three mainstream JVMs including OpenJ9, HotSpot, and GraalVM involving six JDK versions. Within a 72-hour concurrent test run, ASMFUZZ exposed 16 previously unknown unique bugs, of which 11 have been confirmed by the developers. We also compared ASMFUZZ to four prior state-of-the-art fuzzers. ASMFUZZ uncovers 1.6~2.2× more bugs than comparative baselines.

Random test case generation, or fuzzing, is a viable means for uncovering compiler bugs. Unfortunately, compiler fuzzing can be time-consuming and inefficient with purely randomly generated test cases due to the complexity of modern compilers. We present COMFUZZ, a focused compiler fuzzing framework. COMFUZZ aims to improve compiler fuzzing efficiency by focusing on testing components and language features that are likely to trigger compiler bugs. Our key insight is human developers tend to make common and repeat errors across compiler implementations; hence, we can leverage the previously reported buggy-exposing test cases of a programming language to test a new compiler implementation. To this end, COMFUZZ employs deep learning to learn a test program generator from open-source projects hosted on GitHub. With the machine-generated test programs in place, COMFUZZ then leverages a set of carefully designed mutation rules to improve the coverage and bug-exposing capabilities of the test cases. We evaluate COMFUZZ on 11 compilers for JS and Java programming languages. Within 260 hours of automated testing runs, we discovered 33 unique bugs across nine compilers, of which 29 have been confirmed and 22, including an API documentation defect, have already been fixed by the developers. We also compared COMFUZZ to eight prior fuzzers on four evaluation metrics. In a 24-hour comparative test, COMFUZZ uncovers at least 1.5× more bugs than the state-of-the-art baselines.