Grammar-based fuzzing is an effective method for testing programs that consume structured inputs, particularly input parsers. However, if the available grammar does not accurately represent the input format, or if the system under test (SUT) does not conform strictly to the grammar, there may be an impedance mismatch between inputs generated via grammars and inputs accepted by the SUT. Even if the SUT has been designed to strictly conform to the grammar, the SUT parser may exhibit vulnerabilities that would only be triggered by slightly invalid inputs. Grammar-based generation, by construction, will not yield such edge case inputs. To overcome these limitations, we present two mutational-based approaches: Gmutator and G+M . Both approaches are built upon Grammarinator , a grammar-based generator. Gmutator applies mutations to the grammar input of Grammarinator , while G+M directly applies byte-level mutations to Grammarinator -generated inputs. To evaluate the effectiveness of these techniques ( Grammarinator , Gmutator , G+M ) in testing programs that parse various input formats, we conducted an experimental evaluation over four different input formats and twelve SUTs (three per input format). Our findings suggest that both Gmutator and G+M excel in generating edge case inputs, facilitating the detection of disparities between input specifications and parser implementations.

This document presents the artefact that was used to run experiments and produce results reported in the article ‘Grammar Mutation for Testing Input Parsers’. The artefact includes a docker image and a dockerfile. The image can be reconstructed by executing the provided dockerfile. The dockerfile includes all instructions needed to reconstruct the image. Files stored in the image include scripts, systems under test and grammar files. We also list the steps and instructions required to reproduce the results of the experiment.

Fuzz testing, an automated technique that introduces random data inputs to systems, has demonstrated remarkable effectiveness in identifying vulnerabilities. Its scalability and automation have made it a focal point of interest in both academic and industrial settings. However, traditional fuzzing techniques often struggle to generate diverse, rare inputs that conform to a program's input specifications, thereby limiting their full potential. To address these challenges, I propose AFLRepair, a novel approach that applies random mutations to program inputs and subsequently repairs the syntax of any resulting invalid inputs. AFLRepair leverages bytelevel mutations to create a wide array of test cases while ensuring their validity, facilitating the exploration of diverse execution paths within critical program regions. This significantly increases the likelihood of uncovering hidden bugs. Preliminary experiments have revealed a crash on the Lua interpreter. The plan is to continue to validate AFLRepair through comprehensive fuzzing experiments on several open-source software, reporting any vulnerabilities discovered.

When producing test inputs for a program, test generators ("fuzzers") can greatly profit from grammars that formally describe the language of expected inputs. In recent years, researchers thus have studied means to recover input grammars from programs and their executions. The GLADE algorithm by Bastani et al., published at PLDI 2017, was the first black-box approach to claim context-free approximation of input specification for non-trivial languages such as XML, Lisp, URLs, and more.