More apps are published every day and the functionality of each app increases steadily as well. Consequently app analyses are often overwhelmed when confronted with up-to-date, real-world apps. One of the biggest issues originates from the scalability of analyses with respect to libraries. Analyses, more precisely the tools implementing them, cannot distinguish the app’s code from the code of a library. Always analyzing the whole code base is the result. However, this is usually not necessary, for example, when a security property is checked, trusted libraries must not be analyzed.

Debugging, that is, identifying and fixing bugs in software, is a central part of software development. Developers are therefore often confronted with the task of deciding whether a given code snippet contains a bug, and if yes, where. Recently, data-driven methods have been employed to learn this task of bug detection, resulting (amongst others) in so called neural bug detectors. Neural bug detectors are trained on millions of buggy and correct code snippets.

Benchmarking is the most often used technique to empirically evaluate software. To do so, benchmarks are often manually created when they are needed. Mainly two kinds of benchmarks are frequently employed: micro and real-world benchmarks. While micro benchmarks are most of the time handcrafted from scratch, real-world benchmarks are typically created by collecting available software from repositories or markets. Both types have their deficits. On the one hand, a handcrafted micro benchmark can only be of limited complexity, but the creator knows its ground-truth which is needed for precise evaluations. On the other hand, in case of a complex real-world benchmark, a ground-truth is unavailable in most cases. To bring together the best of both worlds we propose the concept of benchmark fuzzing, a three step procedure that allows for an automatic generation, execution and evaluation of benchmarks of configurable size and versatility. We implemented benchmark fuzzing in our novel Android taint analysis benchmark generation tool GenBenchDroid. Our evaluation performed on GenBenchDroidshows the benefits of benchmark fuzzing. We show that over-adaptation of benchmarks can broadly be decreased, scalability issues of analysis tools can be detected and combinations of analysis challenges that negatively impact analysis' accuracy can be identified. In addition, benchmark fuzzing allows to regenerate up-to-date versions of state-of-the-art micro and real-world benchmarks. Furthermore, our evaluation shows that the cost of benchmark fuzzing can be estimated and appears to be reasonable in regards of the advantages.

Abstract Due to the lack of established real-world benchmark suites for static taint analyses of Android applications, evaluations of these analyses are often restricted and hard to compare. Even in evaluations that do use real-world apps, details about the ground truth in those apps are rarely documented, which makes it difficult to compare and reproduce the results. To push Android taint analysis research forward, this paper thus recommends criteria for constructing real-world benchmark suites for this specific domain, and presents TaintBench, the first real-world malware benchmark suite with documented taint flows. TaintBench benchmark apps include taint flows with complex structures, and addresses static challenges that are commonly agreed on by the community. Together with the TaintBench suite, we introduce the TaintBench framework, whose goal is to simplify real-world benchmarking of Android taint analyses. First, a usability test shows that the framework improves experts’ performance and perceived usability when documenting and inspecting taint flows. Second, experiments using TaintBench reveal new insights for the taint analysis tools Amandroid and FlowDroid: (i) They are less effective on real-world malware apps than on synthetic benchmark apps. (ii) Predefined lists of sources and sinks heavily impact the tools’ accuracy. (iii) Surprisingly, up-to-date versions of both tools are less accurate than their predecessors.

Slicing is an established technique for program inspection employed in use cases such as debugging, analysis, understanding and restructuring. Slicing techniques compute program parts which affect (or are affected by) certain slicing criteria. Slicing tools are most often specialized to a language and an application use case.In this paper, we present the tool Jicer, the only functional and available static slicer for Android apps. Jicer is a multi-purpose app slicer, configurable to different use cases by its ability to generate debuggable as well as analyzable and executable output. In its core, Jicer is a slicer for Java bytecode, tailored towards Android app specifics like the lack of a main method, extensive use of callbacks and inter-component communication.Jicer in particular supports security (data leak) analysis of Android apps through an interface allowing Jicer to work as one tool in a cooperative analysis. The role of Jicer in cooperative analyses is twofold: Jicer acts as an aid for other tools (via the reduction of the app size) and Jicer benefits from other tools (via the usage of analysis information in Jicer’s app dependence graph). The evaluation shows that Jicer is able to slice real-world apps thereby reducing app size about most ~55 to And importantly in a cooperative ~96%. analysis Jicer can increase the overall precision by significantly reducing the large number of false positives.

Recent years have seen the development of numerous tools for the analysis of taint flows in Android apps. Taint analyses aim at detecting data leaks, accidentally or by purpose programmed into apps. Often, such tools specialize in the treatment of specific features impeding precise taint analysis (like reflection or inter-app communication). This multitude of tools, their specific applicability and their various combination options complicate the selection of a tool (or multiple tools) when faced with an analysis instance, even for knowledgeable users, and hence hinders the successful adoption of taint analyses.

In recent years, researchers have developed a number of tools to conduct taint analysis of Android applications. While all the respective papers aim at providing a thorough empirical evaluation, comparability is hindered by varying or unclear evaluation targets. Sometimes, the apps used for evaluation are not precisely described. In other cases, authors use an established benchmark but cover it only partially. In yet other cases, the evaluations differ in terms of the data leaks searched for, or lack a ground truth to compare against. All those limitations make it impossible to truly compare the tools based on those published evaluations.