Today, there are millions of third-party Android applications. Some of them are buggy or even malicious. To identify such applications, novel frameworks for automated black-box testing and dynamic analysis are being developed by the Android community. Code coverage is one of the most common metrics for evaluating effectiveness of these frameworks. Furthermore, code coverage is used as a fitness function for guiding evolutionary and fuzzy testing techniques. However, there are no reliable tools for measuring fine-grained code coverage in black-box Android app testing. We present the Android Code coVerage Tool, ACVTool for short, that instruments Android apps and measures code coverage in the black-box setting at class, method and instruction granularity. ACVTool has successfully instrumented 96.9% of apps in our experiments. It introduces a negligible instrumentation time overhead, and its runtime overhead is acceptable for automated testing tools. We demonstrate practical value of ACVTool in a large-scale experiment with Sapienz, a state-of-the-art automated testing tool. Using ACVTool on the same cohort of apps, we have compared different coverage granularities applied by Sapienz in terms of the found amount of crashes. Our results show that none of the applied coverage granularities clearly outperforms others in this aspect.

The deluge of Android apps from third-party developers calls for sophisticated security testing and analysis techniques to inspect suspicious apps without accessing their source code. Code coverage is an important metric used in these techniques to evaluate their effectiveness, and even as a fitness function to help achieving better results in evolutionary and fuzzy approaches. Yet, so far there are no reliable tools for measuring fine-grained bytecode coverage of Android apps. In this work we present ACVTool that instruments Android apps and measures the smali code coverage at the level of classes, methods, and instructions. Tool repository: https://github.com/pilgun/acvtool