LLMs offer a promising avenue to overcome the limitations of traditional taint analysis techniques, with a growing number of studies leveraging LLMs for taint analysis and its downstream applications. However, these studies lack a systematic understanding of LLMs’ taint analysis capabilities, limiting their transferability and reliability. To bridge this gap and better apply LLMs to static taint analysis, we aim to comprehensively measure and understand LLMs’ taint analysis capabilities.Using existing benchmarks is a straightforward approach, but they are unsuitable due to issues such as training data leakage, not accounting for LLMs’ features, and improper assessment criteria. Manually constructing new benchmarks is not only labor-intensive but also struggles to remain effective as LLMs evolve. To address these, we propose LLMCapLens, a dynamic benchmark generation framework to systematically measure and enhance LLMs’ capabilities. LLMCapLens models influencing factors of LLMs’ taint analysis capabilities, employing a Basic Unit-Based generation method and a lightweight dynamic taint analysis-based verification method to implement the automated generation of targeted benchmarks, ensuring both diversity and correctness. Furthermore, LLMCapLens proposes a measurement-driven, training-free, model-specific enhancement approach.We apply LLMCapLens to 10 mainstream LLMs, revealing how they perform under various influencing factors and identifying unique characteristics, such as the underlying error causes for each model. Notably, our enhancement approach significantly improves LLM performance—GPT-4 Turbo, for instance, achieved improvements across 16 out of 19 factors, with an average True Negative Rate increase of 21.29%. Finally, we validate the real-world impact of our method by applying enhanced LLMs to vulnerability detection, demonstrating a substantial improvement over prior approaches.

Java Object Injection (JOI) is a severe type of vulnerability affecting Java deserialization, which allows adversaries to inject a well-crafted, serialized object, thus triggering a series of chained internal methods (called gadgets) and then achieving attack consequences such as Remote Code Execution (RCE). Prior works studied the problem of detecting and chaining gadgets for JOI vulnerability using static search for possible gadget chains and dynamic construction of payload via fuzzing. However, prior works face two following challenges: (i) path explosion in static gadget search and (ii) a lack of fine-grained object relations connected via object fields in dynamic payload construction.In this paper, we design and implement a novel Java deserialization gadget detection framework, called JDD. On one hand, JDD solves the static path explosion problem by a bottom-up approach, which first looks for gadget fragments and then chains gadget fragments from sinks to sources. The approach reduces maximum static search time from exponential to polynomial, i.e., from O(eMn) to O(M2n3+ enM), where n is the number of dynamic function calls in a gadget chain, M is the average number of dynamic function call candidates, and e is the number of entry points. On the other hand, JDD constructs a so-called Injection Object Construction Diagram (IOCD), which models the dataflow dependencies between injection objects’ fields to facilitate dynamic fuzzing. Our evaluation of JDD upon six real-world Java applications reveals 127 zero-day, exploitable gadget chains with six Common Vulnerabilities and Exposures (CVE) identifiers assigned. We also responsibly reported these vulnerabilities to application developers and obtained their acknowledgments and confirmations.

The Android system services usually play a critical role in running multiple important tasks, and delivering seamless user experiences, e.g., conveniently storing user data. In this paper, we conduct the first systematic security study on the data storing process in Android system services, and consequently discover a novel class of design flaws (named Straw), which can lead to serious DoS (Denial-of-Service) attacks, e.g., permanently crashing the whole victim Android device.Then we propose a novel directed fuzzing based approach, called StrawFuzzer, to automatically vet all system services against the straw vulnerabilities. StrawFuzzer balances the tradeoff between path exploration and vulnerability exploitation. By applying StrawFuzzer on three Android systems with the latest security updates, we identified 35 unique straw vulnerabilities affecting 474 interfaces across 77 system services and successfully generated corresponding exploits, which can be used to conduct various permanent/temporary DoS attacks. We have reported our findings with suggestions for repairing the vulnerabilities to corresponding vendors. Up to now, Google has rated our vulnerability as high severity.

Dynamic analysis of Android apps is often used together with an exerciser to increase its code coverage. One big obstacle in designing such Android app exercisers comes from the existence of text-based inputs, which are often constrained by the nature of the input field, such as the length and character restrictions.In this paper, we propose TextExerciser, an iterative, feedback-driven text input exerciser, which generates text inputs for Android apps. Our key insight is that Android apps often provide feedback, called hints, for malformed inputs so that our system can utilize such hints to improve the input generation.We implemented a prototype of TextExerciser and evaluated it by comparing TextExerciser with state-of-the-art exercisers, such as The Monkey and DroidBot. Our evaluation shows that TextExerciser can achieve significantly higher code coverage and trigger more sensitive behaviors than these tools. We also combine TextExerciser with dynamic analysis tools and show they are able to detect more privacy leaks and vulnerabilities with TextExerciser than with existing exercisers. Particularly, existing tools, under the help of TextExerciser, find several new vulnerabilities, such as one user credential leak in a popular social app with more than 10,000,000 downloads.