Fuzzing has proven to be a highly effective approach to uncover software bugs over the past decade. After AFL popularized the groundbreaking concept of lightweight coverage feedback, the field of fuzzing has seen a vast amount of scientific work proposing new techniques, improving methodological aspects of existing strategies, or porting existing methods to new domains. All such work must demonstrate its merit by showing its applicability to a problem, measuring its performance, and often showing its superiority over existing works in a thorough, empirical evaluation. Yet, fuzzing is highly sensitive to its target, environment, and circumstances, e. g., randomness in the testing process. After all, relying on randomness is one of the core principles of fuzzing, governing many aspects of a fuzzer’s behavior. Combined with the often highly difficult to control environment, the reproducibility of experiments is a crucial concern and requires a prudent evaluation setup. To address these threats to validity, several works, most notably Evaluating Fuzz Testing by Klees et al., have outlined how a carefully designed evaluation setup should be implemented, but it remains unknown to what extent their recommendations have been adopted in practice.In this work, we systematically analyze the evaluation of 150 fuzzing papers published at the top venues between 2018 and 2023. We study how existing guidelines are implemented and observe potential shortcomings and pitfalls. We find a surprising disregard of the existing guidelines regarding statistical tests and systematic errors in fuzzing evaluations. For example, when investigating reported bugs, we find that the search for vulnerabilities in real-world software leads to authors requesting and receiving CVEs of questionable quality. Extending our literature analysis to the practical domain, we attempt to reproduce claims of eight fuzzing papers. These case studies allow us to assess the practical reproducibility of fuzzing research and identify archetypal pitfalls in the evaluation design. Unfortunately, our reproduced results reveal several deficiencies in the studied papers, and we are unable to fully support and reproduce the respective claims. To help the field of fuzzing move toward a scientifically reproducible evaluation strategy, we propose updated guidelines for conducting a fuzzing evaluation that future work should follow.

Hardware reverse engineering is a powerful and universal tool for both security engineers and adversaries. From a defensive perspective, it allows for detection of intellectual property infringements and hardware Trojans, while it simultaneously can be used for product piracy and malicious circuit manipulations. From a designer's perspective, it is crucial to have an estimate of the costs associated with reverse engineering, yet little is known about this, especially when dealing with obfuscated hardware. The contribution at hand provides new insights into this problem, based on algorithms with sound mathematical underpinnings. Our contributions are threefold: First, we present the graph similarity problem for automating hardware reverse engineering. To this end, we improve several state-of-the-art graph similarity heuristics with optimizations tailored to the hardware context. Second, we propose a novel algorithm based on multiresolutional spectral analysis of adjacency matrices. Third, in three extensively evaluated case studies, namely (1) gate-level netlist reverse engineering, (2) hardware Trojan detection, and (3) assessment of hardware obfuscation, we demonstrate the practical nature of graph similarity algorithms.

The risk of code reverse-engineering is particularly acute for embedded processors which often have limited available resources to protect program information. Previous efforts involving code obfuscation provide some additional security against reverse- engineering of programs, but the security benefits are typically limited and not quantifiable. Hence, new approaches to code protection and creation of associated metrics are highly desirable. This paper has two main contributions. We propose the first hybrid diversification approach for protecting embedded software and we provide statistical metrics to evaluate the protection. Diversification is achieved by combining hardware obfuscation at the microarchitecture level and the use of software-level obfuscation techniques tailored to embedded systems. Both measures are based on a compiler which generates obfuscated programs, and an embedded processor implemented in an FPGA with a randomized Instruction Set Architecture (ISA) encoding to execute the hybrid obfuscated program. We employ a fine-grained, hardware-enforced access control mechanism for information exchange with the processor and hardware-assisted booby traps to actively counteract manipulation attacks. It is shown that our approach is effective against a wide variety of possible information disclosure attacks in case of a physically present adversary. Moreover, we propose a novel statistical evaluation methodology that provides a security metric for hybrid-obfuscated programs.

Statistical tests are introduced for checking whether an image functionf(x,y) defined on the unit discD={(x,y):x2+y2les 1} is invariant under certain symmetry transformations ofD, given that discrete and noisy data are observed. Invariance under reflections or under rotations by rational angles is considered, as well as rotational invariance. These symmetry relations can be naturally expressed as restrictions for the Zernike moments off(x,y). Therefore, the test statistics are based on theL2distance between Zernike series estimates of the image function itself and its version obtained after applying the symmetry transformation. The asymptotic distribution of the test statistics under both the hypothesis of symmetry as well as under fixed alternatives is derived. Furthermore, the quality of the asymptotic approximations via simulation studies is investigated. The usefulness of our theory is verified by examining an important problem in confocal microscopy, i.e., possible imprecise alignments in the optical path of the microscope are investigated. For optical systems with rotational symmetry, the theoretical point-spread function (PSF) is reflection symmetric with respect to two orthogonal axes, and rotationally invariant if the detector plane matches the optical plane of the microscope. The tests are used to investigate whether the required symmetries can indeed be detected in the empirical PSF.

This paper considers the problem of testing parametric assumptions on signals f from which only noisy observations y/sub k/=f(/spl tau/k)+/spl epsi//sub k/ are available, and where the signal is assumed to be either band-limited or time-limited. To this end, the signal is reconstructed by an estimator based on the Whittaker-Shannon (WS) sampling theorem with oversampling. As test statistic, the minimal L/sub 2/ distance between the estimated signal and the parametric model is used. To construct appropriate tests, the asymptotic distribution of the test statistic is derived both under the hypothesis of the validity of the parametric model and under fixed local alternatives. As a byproduct, the asymptotic distribution of the integrated square error of the estimator is computed, which is of interest by itself, e.g., for the analysis of a cross-validated bandwidth selector.