Code coverage is the primary metric used to assess the quality of test suites, and it is the foundation of many automated techniques ranging from fault localization to search-based optimization approaches. Code coverage is measured by inserting probes into programs which keep track of executed code when running tests. While this can be easily done in many testing domains, it remains a challenging task for Android apps, mainly due to the nature of the Dalvik bytecode used for Android apps: First, the internal handling of registers inhibits common types of probes. To circumvent this problem, existing tools often rely on conversion of Dalvik bytecode to standard Java bytecode or source code, but during the conversion back to Dalvik bytecode errors and inconsistencies may occur. Furthermore, a strict limit of the number of methods and classes contained in a single archive of Dalvik bytecode (DEX file) requires spliting apps into multiple such DEX files (multidex approach), which is rarely supported by existing coverage instrumentation frameworks. This is not only a problem when trying to instrument regular multidex apps, but the coverage instrumentation itself increases the number of methods, potentially requiring a multidex solution even for apps that would otherwise fit in a single DEX file. In this paper we present WallMauer, a new code coverage tool that overcomes these limitations: It supports multidex, and avoids inconsistencies by rigorously instrumenting Dalvik byte-code directly. WallMauer solely requires an APK file as input and as such it can be easily integrated into any existing testing environment. Using a set of 1000 open source apps from the F-Droid repository we demonstrate that WallMauer is extremely robust, successfully instrumenting more than 99% of apps, more than any other state-of-the-art instrumentation framework.

Android apps are known to be fragile: Users as well as automated test generators frequently encounter app crashes. An important prerequisite for fixing the underlying faults is to provide developers with automated tests to reliably reproduce such crashes. Unfortunately, often the only information available is the stack trace of the crash. While search-based test generation has been successfully used for finding tests that reproduce crashes from stack traces in other domains, such approaches are fundamentally limited in their applicability on Android apps. For example, even the basic search operator of crossover used in evolutionary algorithms is challenged since applicable inputs depend on the state of the app, such that sequences of inputs cannot be arbitrarily concatenated. To overcome this problem we use an estimation of distribution search algorithm, which guides the reproduction using a probabilistic model of relevant actions, requiring no complicated search operators. The probabilistic model is bootstrapped using established Android testing heuristics and crash-related information extracted from the stack trace and byte code, and is updated throughout the search using a fitness function based on stack traces. Evaluation on 30 real-world app crashes, of which 24 are successfully reproduced, demonstrates that the approach is effective, reliable and fast.

The ESPRESSO testing framework for ANDROID has gained popularity among developers as it allows to write concise and reliable VI tests. State-of-the-art tools for automatically testing ANDROID apps, however, tend to produce crash reports rather than human-readable tests, and even if they produce tests these (1) rarely use the ESPRESSO format; (2) are often unreliable due to the volatile nature of widget identifiers; and (3) usually contain no test assertions to serve as regression oracles. While the lack of ESPRESSO support of test generation tools has been addressed by reverse engineering ESPRESSO tests, the other problems remain even with this workaround. In this paper, we therefore introduce a novel ESPREsso-based representation that allows test generators to generate ESPRESSO test cases directly that (1) can reliably identify widgets using clear and concise ESPRESSO selectors, and (2) can check test executions using ESPRESSO assertions. Experiments on 1,035 ANDROID apps demonstrate that the proposed approach generates ESPRESSO tests that are significantly more reliable than reverse engineered tests, and the ESPRESSO assertions of the generated tests are effective at detecting faults in ANDROID apps.

Android apps can be effectively tested by randomly generating inputs and triggering corresponding events. Most test generators focus on user-triggered events, such as button clicks. However, the state of an app is not only determined by the interactions with a user, but also inputs from the system and other apps, which are called intents in Android. Intent fuzzing, that is, the automated generation of randomized intents as test inputs, has been demonstrated to be an effective means for identifying crashes in apps. However, the behavior of intent handlers is influenced by the state of the app, which may depend on the user’s interactions with the app that triggers corresponding events. Recent test generators have therefore started integrating some of both types of events, leaving open questions about the best way to combine and balance UI inputs and intents. In this paper, we describe a general framework for integrating user events and intents for testing Android apps. We study empirically how to best combine these two types of events, and evaluate the effectiveness of the combination. Our experiments suggest that combining UI inputs and intents reveals substantially higher code coverage as well as more unique crashes (844 on 500 F-Droid apps) than sending only user-events (762) or only intents (511): The combined approach achieves a magnitude higher activity coverage (78.07%) than using only user inputs (70.45%) and sending only intents (58.23%). Furthermore, 121 unique crashes were found only through the combination of UI inputs and intents. Although intent crashes and UI crashes result from similar exception types, they are distinct, which is relevant when comparing test generators.