Skeletal program enumeration (SPE) can generate a great number of test programs for validating the correctness of compilers or interpreters. The classic SPE generates programs by exhaustively enumerating all possible variable usage patterns into a given syntactic structure. Even though it is capable of producing many test programs, the exhaustive enumeration strategy generates a large number of invalid programs, which may waste plenty of testing time and resources. To address the problem, this article proposes a tree-based SPE technique. Compared to the state-of-the-art, the key merit of the tree-based approach is that it allows us to take the dependency information into consideration when producing test programs and, thus, make it possible to (1) directly generate non-equivalent programs and (2) apply dominance relations to eliminate invalid test programs that have undefined variables. Hence, our approach significantly saves the cost of the naïve SPE approach. We have implemented our approach into an automated testing tool, IFuzzer , and applied it to test eight different implementations of Python interpreters, including CPython, PyPy, IronPython, Jython, RustPython, GPython, Pyston, and Codon. In three months of fuzzing, IFuzzer detected 142 bugs, of which 87 have been confirmed to be previously unknown bugs, of which 34 have been fixed. Compared to the state-of-the-art SPE techniques, IFuzzer takes only 61.0% of the time cost given the same number of testing seeds and improves 5.3% source code function coverage in the same time budget of testing.

Skeletal Program Enumeration (SPE) is one of the state-of-the-art techniques for generating programs and validating the correctness of compilers/interpreters. However, existing SPE approaches neglect the enumeration of function calls, thereby overlooking a crucial feature of modern programming languages. To address this gap, this paper proposes an approach that integrates function calls into SPE techniques. Our approach first defines function skeletons and then employs the staged enumeration of variables and arguments. Consequently, our approach can eliminate equivalent programs on-the-fly. To address the test oracle problem in SPE, it takes a differential strategy to identify interpreter bugs, i.e., differentially comparing program execution results across various interpreters. Hence it can effectively detect crash bugs, behavior bugs, and miscompilation bugs. To evaluate the effectiveness, we have implemented the proposed approach into an open-source tool, i.e., FcFuzzer, for testing Python interpreters. The evaluation is conducted on seven Python interpreters: CPython, RustPython, IronPython, Pyston, PyPy, Codon, and GPython. Using the same seeds, FcFuzzer can detect, on average, 1.6 times more bugs than the state-of-the-art SPE technique. In a one-month fuzzing experiment, FcFuzzer successfully identified 35 Python interpreter bugs, of which 16 were confirmed to be unknown, and six have been fixed.

Programming libraries are indispensable for programming languages. Programmers can access the pre-written codes in these libraries via the application programmable interfaces (API), optimizing and accelerating their programming tasks. However, defects in these libraries may cause unexpected software behaviors, threatening their robustness and safety. Thus, it is crucial to ensure the quality of the libraries. This paper explores an alternative approach, namely Fuzzing Skeleton API (FSA), for detecting library bugs in Python. For the given API, FSA aims to generate massive inputs, i.e., different argument combinations, and pass them to the API to verify its correctness and reliability. To realize this, FSA first abstracts the API into a skeleton by modeling its usage of parameters as placeholders. Then, it can generate the seed API calls by filling these placeholders with pre-defined arguments. Finally, the approach incorporates four mutation strategies, i.e., bit mutation, literal mutation, element mutation, and attribute mutation, to mutate different arguments and hence generate massive API calls. We have implemented the proposed approach into an automated tool, namely DyFuzz, for testing Python libraries. In less than one month of the fuzzing experiment, DyFuzz detected 14 library bugs, of which nine have been confirmed as unknown bugs.

Rust compilers play a foundational role in the Rust language. Like any complex system, they are susceptible to bugs, which can impact the correctness and reliability of the compiled Rust programs. To gain a deeper understanding of these bugs, this paper presents the first comprehensive analysis of historical bugs in two widely used Rust compilers: Rustc and Rust-GCC. The analysis delves into the bugs’ characteristics, bug-proneness locations, bug root causes, and bug-fixing efforts. The findings reveal that the majority of bugs in Rustc are associated with the compiler’s kernel, while Rust-GCC experiences most bugs related to the cleanup process. Among all modules, the ‘src/librustc’ module exhibits the highest bug-proneness in the Rustc compiler, whereas the ‘gcc/rust’ modules demonstrate the highest bug-proneness in the Rust-GCC compiler. Furthermore, the study reveals that the bug-fixing process is accelerated when test cases utilize Rust’s concurrency features.

Python is a popular typical dynamic programming language. In Python, dynamic typing is one of the most critical dynamic features. The lack of type information is likely to hinder the maintenance of Python projects. However, existing work has seldom focused on studying the impact of Python dynamic typing on project maintenance. This paper focuses on the two most common practices of Python dynamic typing, i.e. inconsistent-type assignments (ITA) and inconsistent variable types (IVT). Two approaches are proposed to identify ITA and IVT, i.e. identifying ITA by analyzing Abstract Syntax Trees and comparing identifiers types and identifying IVT by constructing a type dependency graph. In empirical experiments, we first locate the usage of ITA and IVT in 10 open-source Python projects. Then, we investigate the relations between the occurrence of ITA and IVT and the results of maintenance tasks. The study results show that projects are more prone to change as the number of dynamic typing identifiers increases. There is a weak connection between change-proneness and variable dynamic typing. There is a high probability that maintenance time and the acceptance of commits decrease as dynamic typing identifiers increase in projects. These results implicate that dynamic and static variables should be divided while developing new programming languages. Dynamic typing identifiers may not be the direct root causes for most software bugs. The categories of these bugs are worth exploring.