Fuzz testing embedded OSs is difficult because their implementations vary widely and rely on specialized hardware. These factors render many existing methods ineffective, since they prevent adapting established fuzzing routines, disrupt communication with the target OS, and impede observation of runtime behavior. This paper introduces EOF, a feedback-guided fuzzer designed to test embedded OSs running on actual hardware. Through the debug port, EOF communicates with the target embedded OS, executes test cases, and collect feedback data, with no dependence on OS services. Then, EOF deploys a cross-platform agent and executes API-aware input across diverse hardware. Last, EOF collects runtime coverage and critical execution events to identify interesting seeds and find potential bugs during fuzzing. We implemented EOF and evaluated its performance on four different embedded OSs, where EOF discovered 19 bugs and achieved a 50.84% coverage improvement on average compared with other comparable fuzzing methods.

Embedded Android Devices have proliferated in many security-critical embedded scenarios, requiring sufficient testing to root out vulnerabilities. Due to Android’s architecture, which uses a Hardware ion Layer (HAL) for vendor-specific driver implementations, traditional kernel testing techniques cannot detect such bugs within the actual driver logic, which are commonly proprietary and vendorspecific. In this paper, we propose DroidFuzz, an embedded Android system fuzzer that targets such vendor-specific driver implementations to find such bugs. Through leveraging pre-testing HAL driver probing, kernel-user relational payload generation, and cross-boundary execution state feedback, we effectively test the proprietary drivers in both the kernel and the HAL layer. We implemented DroidFuzz and evaluated its effectiveness on 7 embedded Android devices, and found 12 security-critical previously unknown bugs, all of which have been confirmed by the respective vendors.

The Linux distribution kernel differs significantly from the mainline kernel, incorporating additional features and vendor-specific extensions. Among these additions, many runtime parameter interfaces are unique to distribution kernels, which expands the attack surface and increases the risk of potential vulnerabilities. Fuzzing has been used to assess Linux distributions, but existing tools cannot systematically test these distribution-specific interfaces due to two main challenges: (1) generating test cases for these runtime parameter interfaces, and (2) concentrating test resources on the distribution-specific interface code. To address these challenges, we propose RPG, a distribution-specific runtime parameter-guided kernel fuzzer. RPG operates in three phases: First, RPG extracts distribution-specific runtime parameter interfaces. Then, RPG uses LLM and tuning software databases to model each parameter range to generate meaningful interface test cases. Third, RPG utilizes the distribution kernel’s function control flow graph to guide the fuzzer to generate generic test cases that are more closely related to the distribution-specific interface code. We evaluated RPG on four Linux distribution kernels: Ubuntu 22.04, Fedora 42, OpenAnolis 8.8, and OpenAnolis 23.1. RPG detected 22 previously unknown bugs (13 distribution-specific), of which 15 were confirmed and 10 fixed by kernel maintainers. RPG also achieved 20.4% and 21.2% higher branch coverage than Syzkaller and Healer, respectively.

An open source Linux distribution often undergoes version upgrades and migrations, which is prone to incompatibility issues especially when it comes to large-scale software changes. Although differential testing has been widely used in software testing, it is still challenging to apply it for detecting such incompatibilities in the context of industrial settings. In this paper, we report our experience in leveraging LLMs to address the challenges faced by the Linux distribution community. Specifically, we develop an LLM-based differential testing method called Versify to assist maintainers of Linux distributions in locating incompatibilities during version upgrades and migrations. Its trial operation period within the Linux distribution community shows that it uncovered 8,489 instances of differing behavior, of which 644 were prioritized for attention by developers. After deduplication and filtering, 39 unique compatibility reports were identified. Feedback from Linux distributions developers indicates that our reports have provided valuable recommendations for package selection in future OS releases.

The Linux kernel network stack is a critical component of modern operating systems, widely deployed across platforms and often exposed to untrusted inputs. Its complex and stateful nature makes it a frequent target of security vulnerabilities, particularly those triggered by subtle protocol interactions. While existing fuzzers like syzkaller have demonstrated strong capabilities in discovering kernel bugs, they face challenges in exercising deep protocol logic due to the lack of coordinated inputs and protocol awareness. In this paper, we present Tron, a tool designed for fuzzing the Linux kernel network stack. By synthesizing syscall–packet input sequences based on protocol structure and incorporating runtime feedback, Tron enables the exploration of protocol-dependent state transitions and deep execution paths. Our approach addresses the fundamental challenges in dual-input fuzzing by integrating protocol knowledge with execution feedback. We evaluate Tron on four recent Linux kernel versions and compare it against syzkaller and kernelGPT. The results show that Tron improves branch coverage by 22.9% and 12.1% over syzkaller and kernelGPT, respectively, and discovers 25 previously unknown bugs, 7 of which have been fixed. These results demonstrate the effectiveness of protocol–system call input synthesis in enhancing network stack fuzzing and uncovering hard-to-reach bugs in kernel protocol implementations.

Kata Containers is a secure container runtime with lightweight virtual machines and a customized Linux kernel optimized for cloud-native workloads, which is important for cloud-native systems. Fuzzing is a widely-used technique for detecting kernel vulnerability. However, current kernel fuzzers can't be simply applied to kernels in cloud-native environments because of the discrepancies between test and actual deployment scenarios. This paper introduces DragonRadar, a kernel fuzzing tool adapted for Kata Containers, which aligns the testing environment with cloud deployment realities. We extend to support kernel fuzzing in cloud-native environments by integrating Syzkaller's capability with a lightweight virtual machine manager called Dragonball. The evaluation shows that DragonRadar effectively identifies 25 kernel vulnerabilities in the mainline Linux kernel used in the Kata Containers environment, while maintaining code coverage similar to vanilla Syzkaller. DragonRadar is available at https://github.com/TOBESTONG//DragonRadar.

Modern file systems have become increasingly feature-rich and highly complex, making crash consistency increasingly difficult to perform correctly. Thoroughly testing file systems for crash consistency bugs, however, is difficult to achieve good results due to insufficient state exploration, a lack of guidance for test case generation, and missing support for modern file system features. In this article, we present a new approach towards testing file system consistency: systematic file system persistent state exploration. In contrast to previous efforts, our design addresses these shortcomings through testing the crash consistency property of file systems systematically using the following procedures. Initially, we use system call generation and execution feedback from fuzzers to generate workloads that stress the file system code. During this process, we systematically explore all possible persistent states of the underlying file system for the given workload and subsequently use them as file system image inputs for the crash recovery routines to produce a corresponding file system state. After the file system finishes processing an image input, we deploy an efficient file system checker to compare the contents of the image to that of a correct image and determine whether the image is inconsistent, consequently determining whether we have triggered a crash consistency bug in the underlying file system. We implemented a prototype tool SnapCC and deployed it for testing multiple mainstream file systems on Linux. We compared its effectiveness along with other relevant tools Hydra and B3, where our results show that SnapCC achieves 16% to 44% better coverage over Hydra, and finds 15 new consistency bugs, whereas B3 and Hydra find 2 and 6, over a period of 2 weeks, further demonstrating SnapCC’s effectiveness in discovering file system consistency bugs. To demonstrate our approach’s adaptability, we also tested SnapCC on 5 other file systems, upon which 7 additional bugs were found.

State-of-the-art kernel fuzzers use edge-based code coverage metrics for novel behavior detection. However, code coverage is not sufficient for operating system kernels, for they contain many untracked but interesting features, such as comparison operands, kernel state identifiers, flags, and executable code, within its data segments, that reflects different execution patterns, and can profoundly increase the granularity and scope of the coverage metrics.

Embedded operating systems, considering their widespread use in security-critical applications, are not effectively tested with sanitizers to effectively root out bugs. Sanitizers provide a means to detect bugs that are not visible directly through exceptional or erroneous behaviors, thus uncovering more potent bugs during testing.

eBPF is an inspiring technique in Linux that allows user space processes to extend the kernel by dynamically injecting programs. However, it poses security issues, since the untrusted user code is now executed in the kernel space. eBPF utilizes a verifier to validate the safety of the provided programs, thus its correctness is of paramount importance as attackers may exploit vulnerabilities within it to inject malicious programs. Bug-finding tools like kernel fuzzers currently can detect memory bugs in eBPF system calls, but they experience difficulties in finding correctness bugs in the verifier, e.g., incorrect validations that allow the loading of unsafe programs. Because, unlike detecting memory bugs, where sanitizers can capture such errors once observed, automatically uncovering correctness bugs is very difficult, without an effective test oracle that determines if the verifier behaves correctly for given programs.

The Robot Operating System 2 (ROS) is the de-facto standard for robotic software development, with a wide application in diverse safety-critical domains. There are many efforts in testing that seek to deliver a more secure ROS codebase. However, existing testing methods are often inadequate to capture the complex and stateful behaviors inherent to ROS deployments, resulting in limited test- ing effectiveness. In this paper, we propose R2D2, a ROS system fuzzer that leverages ROS’s runtime states as guidance to increase fuzzing effectiveness and efficiency. Unlike traditional fuzzers, R2D2 employs a systematic instrumentation strategy that captures the system’s runtime behaviors and profiles the current system state in real-time. This approach provides a more in-depth understanding of system behaviors, thereby facilitating a more insightful explo- ration of ROS’s extensive state space. For evaluation, we applied it to four well-known ROS applications. Our evaluation shows that R2D2 achieves an improvement of 3.91× and 2.56× in code coverage compared to state-of-the-art ROS fuzzers, including Ros2Fuzz and RoboFuzz, while also uncovering 39 previously unknown vulnera- bilities, with 6 fixed in both ROS runtime and ROS applications. For its runtime overhead, R2D2 maintains an average execution and memory usage overhead with 10.4% and 1.0% in respect, making R2D2 effective in ROS testing.

Directed grey-box fuzzing is a widely used automatic testing technique that has helped developers test specific code space in the target program. Although many directed fuzzers are designed to test the Linux kernel, challenges still remain due to the complexity of industrial requirements and deployment environments. In this paper, we collaborate with developers from Alibaba and the OpenAnolis community to conduct an industry practice of directed kernel fuzzing for open-source Linux distribution. We highlight typical challenges in deploying directed kernel fuzzing, including target-related kernel configuration options being disabled, unrelated initial seeds limiting fuzzing startup performance, no support for kernel feature interface fuzzing, independent fuzzer execution limiting fuzzing effectiveness, much manual work to triage and analyze crashes, and hard to integrate into the existing fuzzing framework. We provide solutions to these challenges, which allowed us to discover 11 previously unknown kernel bugs related to cloud-native features, io_uring, and other components in the OpenAnolis Linux distribution.

The Universal Serial Bus (USB) is an essential component in modern operating systems, allowing for a wide assortment of peripherals to connect conveniently to a computer. The USB stack in an operating system usually consists of the following two components: the host-side driver and the device-side gadget driver, both of which are security-critical. If any vulnerabilities in these privileged-mode drivers are exploited, a malicious or malformed device could crash the whole system. Fuzzing, a popular automated vulnerability detection technology, has been applied to testing kernel components such as drivers with varying degrees of success. However, existing works mainly focus on one side and test drivers through emulating malicious input from userspace or peripherals while neglecting intricate internal states triggered only through interaction between the two boundaries, leaving a multitude of bugs exposed.In this paper, we propose Saturn, a host-gadget synergistic USB driver fuzzing approach, aiming to cover the entire handling chain throughout the USB communication. To achieve this, Saturn first leverages extracted driver information to attach gadgets systematically and trigger more driver types, facilitating the transition to interactive logic. Then, Saturn performs a persistent synergistic fuzzing process through canonical operation injection on both sides to play their own important roles, significantly expanding the states explored and exposing bugs in such logic. Compared to the state-of-the-art USB fuzzers, such as Syzkaller, USBFuzz and FUZZUSB, Saturn improves the branch coverage statistics on the corresponding stack by 1.53×, 3.69× and 2.3×, respectively. In addition, Saturn found 26 previously unknown bugs, among which are 4 CVEs, including drivers on each side.

Stable patch identification is crucial in merging patches into stable versions, which helps ensure the stability of the Linux kernel. Although many tools have been proposed to mitigate the manual effort of stable patch identification, challenges still arise because they neglect continuous stable patch tracking and advanced Natural Language Processing (NLP) pre-training techniques. In this paper, in collaboration with developers from the openAnolis Linux operating system distribution community, we present a stable patch identification model called PatchBERT. It utilizes BERT and CodeBERT to capture the semantic patch representation from the commit message and code changes in a patch. We then perform patch classification and output the probability that the patch should be merged into the stable versions. We perform experiments on the dataset used by the previous methods. The experimental results show the superior performance of PatchBERT over state-of-the-art baselines. Additionally, it is common practice to train the model using the latest Linux patches and implement it in a real-world industrial setting. In this exercise, we randomly select 10,000 patches for identification, accurately identifying 8,617 patches and incorrectly identifying 1,383 patches. This practical outcome further confirms the effectiveness and utility of PatchBERT in real-world scenarios.

Embedded operating systems (Embedded OSs) power much of our critical infrastructure but are, in general, much less tested for bugs than general-purpose operating systems. Fuzzing Embedded OSs encounter significant roadblocks due to much less documented specifications, an inherent ineffectiveness in generating high-quality payloads. In this article, we propose ECG, an Embedded OS fuzzer empowered by large language models (LLMs) to sufficiently mitigate the aforementioned issues. ECG approaches fuzzing Embedded OS by automatically generating input specifications based on readily available source code and documentation, instrumenting and intercepting execution behavior for directional guidance information, and generating inputs with payloads according to the pregenerated input specifications and directional hints provided from previous runs. These methods are empowered by using an interactive refinement method to extract the most from LLMs while using established parsing checkers to validate the outputs. Our evaluation results demonstrate that ECG uncovered 32 new vulnerabilities across three popular open-source Embedded OS (RT-Linux, RaspiOS, and OpenWrt) and detected ten bugs in a commercial Embedded OS running on an actual device. Moreover, compared to Syzkaller, Moonshine, KernelGPT, Rtkaller, and DRLF, ECG has achieved additional kernel code coverage improvements of 23.20%, 19.46%, 10.96%, 15.47%, and 11.05%, respectively, with an overall average improvement of 16.02%. These results underscore ECG’s enhanced capability in uncovering vulnerabilities, thus contributing to the overall robustness and security of the Embedded OS.

Kernel fuzzing is an effective technique in operating system vulnerability detection. Fuzzers such as Syzkaller and Moonshine frequently pass highly structured data between fuzzer processes in guest virtual machines and manager processes in the host operating system to synchronize fuzzing-relevant data and information. Since the guest virtual machines’ and the host operating system’s memory spaces are mutually isolated, fuzzers conduct synchronization operations using mechanisms such as Remote Procedure Calls over TCP/IP networks, incurring significant overheads that negatively impact the fuzzer’s efficiency and effectiveness in increasing code coverage and finding vulnerabilities. In this paper, we propose Horus , a kernel fuzzing data transfer mechanism that mitigates the aforementioned data transfer overheads. Horus removes host-VM memory isolation and performs data transfers through copying to and from target memory locations in the guest virtual machine. Horus facilitates such efficient transfers through using fixed stub structures in the guest’s memory space, whose addresses, along with the guest’s RAM contents, are exposed to the host during the fuzzer’s initialization process. When conducting transfers, Horus passes highly-structured non-trivial data between the host and guest instances through copying the data directly to and from the stub structures, reducing the overall overhead significantly compared to that of using a network-based approach. We implemented Horus upon state-of-the-art kernel fuzzers Syzkaller , Moonshine and kAFL and evaluated its effectiveness. For Syzkaller and Moonshine , Horus increased their transfer speeds by 84.5% and 85.8% for non-trivial workloads on average and improved their fuzzing throughputs by 31.07% and 30.62%, respectively. Syzkaller and Moonshine both achieved a coverage speedup of 1.6× through using Horus . For kAFL, Horus improved specifically its Redqueen component’s execution speeds by 19.4%.

Rt-Linux contains critical modifications that are much less tested than the vanilla kernel, thus placing many systems at risk. In this paper, we present DRLF, a directed fuzzer targeted towards fuzzing any code area in Rt- Linux, thus allowing for more efficient tests on Rt-Linux's unique code sections. DRLF performs directed fuzzing through a kernel-level weighted callgraph construction technique, and prioritizing input sequences that exhibit less distance to the target code. Evaluations show that DRLF delivers better cover speed while achieving a 24.70% coverage increase for the targeting code areas. DRLF also found 11 previously unknown bugs within Rt-Linux, and has been integrated into Alibaba's CI/CD pipeline.

Auto-tuning attracts increasing attention in industry practice to optimize the performance of a system with many configurable parameters. It is particularly useful for cloud applications and services since they have complex system hierarchies and intricate knob correlations. However, existing tools and algorithms rarely consider practical problems such as workload pressure control, the support for distributed deployment, and expensive time costs, etc., which are utterly important for enterprise cloud applications and services. In this work, we significantly extend an open source tuning tool – KeenTune to optimize several typical enterprise cloud applications and services. Our practice is in collaboration with enterprise users and tuning tool developers to address the aforementioned problems. Specifically, we highlight five key challenges from our experiences and provide a set of solutions accordingly. Through applying the improved tuning tool to different application scenarios, we achieve 2%-14% improvements for the performance of MySQL, OceanBase, nginx, ingress-nginx, and 5%-70% improvements for the performance of ACK cloud container service.

Embedded operating systems (Embedded OSs) are extensively deployed in many mission-critical industrial scenarios. Any defects within these systems may result in unacceptable losses. Therefore, it is imperative to develop tools to detect bugs within Embedded OSs, thus minimizing potential impacts on industrial infrastructures. Coverage-guided fuzzing is a vulnerability detection technique that has found numerous real-world vulnerabilities within both application programs as well as kernels. However, state-of-the-art kernel fuzzers, e.g., Syzkaller, mainly target general purpose-operating systems, such as Linux, macOS, and Windows, whereas Embedded OSs support is mostly lacking. In this article, we propose Tardis, the first Embedded OSs fuzzer capable of testing a wide selection of Embedded OSs while leveraging coverage feedback. Tardis conducts OS-agnostic code coverage collection and analysis, allowing developers and testers to test a wide range of Embedded OSs without significant manual efforts. We implemented and evaluated Tardis on several well-known Embedded OSs, such as UC/OS and FreeRTOS. Tardis can successfully perform fuzz testing on these kernels without significant manual effort for adaptation. By leveraging coverage feedback, Tardis can cover 51.32% more branches than black-box fuzzing on average on the respective Embedded OSs over 24 h. Tardis also found 17 previously unknown bugs among the target Embedded OSs.

Modern operating system kernels are too complex to be free of bugs. Fuzzing is a promising approach for vulnerability detection and has been applied to kernel testing. However, existing work does not consider the influence relations between system calls when generating and mutating inputs, resulting in difficulties when trying to reach into the kernel's deeper logic effectively.

A real-time operating system (RTOS) is an operating system designed to meet certain real-time requirements. It is widely used in embedded applications, and its correctness is safety-critical. However, the validation of RTOS is challenging due to its complex real-time features and large code base. In this paper, we propose Rtkaller , a state-aware kernel fuzzer for the vulnerability detection in RTOS. First, Rtkaller implements an automatic task initialization to transform the syscall sequences into initial tasks with more real-time information. Then, a coverage-guided task mutation is designed to generate those tasks that explore more in-depth real-time related code for parallel execution. Moreover, Rtkaller realizes a task modification to correct those tasks that may hang during fuzzing. We evaluated it on recent versions of rt-Linux, which is one of the most widely used RTOS. Compared to the state-of-the-art kernel fuzzers Syzkaller and Moonshine, Rtkaller achieves the same code coverage at the speed of 1.7X and 1.6X, gains an increase of 26.1% and 22.0% branch coverage within 24 hours respectively. More importantly, Rtkaller has confirmed 28 previously unknown vulnerabilities that are missed by other fuzzers.

Industrial Control System (ICS) protocols play an essential role in building communications among system components. Recently, many severe vulnerabilities, such as Stuxnet and DragonFly, exposed in ICS protocols have affected a wide distribution of devices. Therefore, it is of vital importance to ensure their correctness. However, the vulnerability detection efficiency of traditional techniques such as fuzzing is challenged by the complexity and diversity of the protocols.In this paper, we propose to equip the traditional protocol fuzzing with coverage-guided packet crack and generation. We collect the coverage information during the testing procedure, save those valuable packets that trigger new path coverage and crack them into pieces, based on which, we can construct higher-quality new packets for further testing. For evaluation, we build Peach*on top of Peach, which is one of the most widely used protocol fuzzers, and conduct experiments on several ICS protocols such as Modbus and DNP3. Results show that, compared with the original Peach, Peach*achieves the same code coverage and bug detection numbers at the speed of 1.2X-25X. It also gains final increase with 8.35%-36.84% more paths within 24 hours and has exposed 9 previously unknown vulnerabilities.