We report on our experience at Google deploying a variety of coverage-guided and black-box fuzzers to find bugs in compilers for two graphics shading languages: the WebGPU Shading Language (WGSL) and Standard Portable Intermediate Representation (SPIR-V). We discuss the deployment of coverage-guided fuzzing on ClusterFuzz and OSS-Fuzz using libFuzzer’s built-in mutators, and a number of custom mutators that exploit knowledge of the syntax and semantics of WGSL and SPIR-V. We also discuss the deployment of several black box fuzzers on ClusterFuzz, including two that are based on new randomised program generators for WGSL, which we have also used in a targeted fashion for end-to-end testing of WebGPU implementations. We discuss a series of insights arising from our experience that we hope will be valuable to practitioners and researchers interested in applying fuzzing to industrial problems.

Recent transformation-based approaches to compiler testing look for mismatches between the results of pairs of equivalent programs, where one program is derived from the other by randomly applying semantics-preserving transformations. We present a formulation of transformation-based compiler testing that provides effective test-case reduction almost for free: if transformations are designed to be as small and independent as possible, standard delta debugging can be used to shrink a bug-inducing transformation sequence to a smaller subsequence that still triggers the bug. The bug can then be reported as a delta between an original and minimally-transformed program. Minimized transformation sequences can also be used to heuristically deduplicate a set of bug-inducing tests, recommending manual investigation of those that involve disparate types of transformations and thus may have different root causes. We demonstrate the effectiveness of our approach via a new tool, spirv-fuzz, the first compiler-testing tool for the SPIR-V intermediate representation that underpins the Vulkan GPU programming model.