How can we know whether new mechanistic interpretability methods achieve real improvements? In pursuit of lasting evaluation standards, we propose MIB, a Mechanistic Interpretability Benchmark, with two tracks spanning four tasks and five models. MIB favors methods that precisely and concisely recover relevant causal pathways or causal variables in neural language models. The circuit localization track compares methods that locate the model components---and connections between them---most important for performing a task (e.g., attribution patching or information flow routes). The causal variable track compares methods that featurize a hidden vector, e.g., sparse autoencoders (SAE) or distributed alignment search (DAS), and align those features to a task-relevant causal variable. Using MIB, we find that attribution and mask optimization methods perform best on circuit localization. For causal variable localization, we find that the supervised DAS method performs best, while SAEs features are not better than neurons, i.e., non-featurized hidden vectors. These findings illustrate that MIB enables meaningful comparisons, and increases our confidence that there has been real progress in the field.

Code coverage is the primary metric used to assess the quality of test suites, and it is the foundation of many automated techniques ranging from fault localization to search-based optimization approaches. Code coverage is measured by inserting probes into programs which keep track of executed code when running tests. While this can be easily done in many testing domains, it remains a challenging task for Android apps, mainly due to the nature of the Dalvik bytecode used for Android apps: First, the internal handling of registers inhibits common types of probes. To circumvent this problem, existing tools often rely on conversion of Dalvik bytecode to standard Java bytecode or source code, but during the conversion back to Dalvik bytecode errors and inconsistencies may occur. Furthermore, a strict limit of the number of methods and classes contained in a single archive of Dalvik bytecode (DEX file) requires spliting apps into multiple such DEX files (multidex approach), which is rarely supported by existing coverage instrumentation frameworks. This is not only a problem when trying to instrument regular multidex apps, but the coverage instrumentation itself increases the number of methods, potentially requiring a multidex solution even for apps that would otherwise fit in a single DEX file. In this paper we present WallMauer, a new code coverage tool that overcomes these limitations: It supports multidex, and avoids inconsistencies by rigorously instrumenting Dalvik byte-code directly. WallMauer solely requires an APK file as input and as such it can be easily integrated into any existing testing environment. Using a set of 1000 open source apps from the F-Droid repository we demonstrate that WallMauer is extremely robust, successfully instrumenting more than 99% of apps, more than any other state-of-the-art instrumentation framework.

The ESPRESSO testing framework for ANDROID has gained popularity among developers as it allows to write concise and reliable VI tests. State-of-the-art tools for automatically testing ANDROID apps, however, tend to produce crash reports rather than human-readable tests, and even if they produce tests these (1) rarely use the ESPRESSO format; (2) are often unreliable due to the volatile nature of widget identifiers; and (3) usually contain no test assertions to serve as regression oracles. While the lack of ESPRESSO support of test generation tools has been addressed by reverse engineering ESPRESSO tests, the other problems remain even with this workaround. In this paper, we therefore introduce a novel ESPREsso-based representation that allows test generators to generate ESPRESSO test cases directly that (1) can reliably identify widgets using clear and concise ESPRESSO selectors, and (2) can check test executions using ESPRESSO assertions. Experiments on 1,035 ANDROID apps demonstrate that the proposed approach generates ESPRESSO tests that are significantly more reliable than reverse engineered tests, and the ESPRESSO assertions of the generated tests are effective at detecting faults in ANDROID apps.

Mechanistic interpretability methods aim to identify the algorithm a neural network implements, but it is difficult to validate such methods when the true algorithm is unknown. This work presents InterpBench, a collection of semi-synthetic yet realistic transformers with known circuits for evaluating these techniques. We train simple neural networks using a stricter version of Interchange Intervention Training (IIT) which we call Strict IIT (SIIT). Like the original, SIIT trains neural networks by aligning their internal computation with a desired high-level causal model, but it also prevents non-circuit nodes from affecting the model's output. We evaluate SIIT on sparse transformers produced by the Tracr tool and find that SIIT models maintain Tracr's original circuit while being more realistic. SIIT can also train transformers with larger circuits, like Indirect Object Identification (IOI). Finally, we use our benchmark to evaluate existing circuit discovery techniques.

Abstract Several tools for automatically testing Android applications have been proposed. In particular, Sapienz is a search‐based tool that has been recently deployed in an industrial setting. Although it has been shown that Sapienz outperforms several state‐of‐the‐art tools, it is still to be seen what features of SAPIENZ impact the most on its effectiveness. We conducted an extensive empirical study where we compare the impact of the search algorithm and the usage of motif genes, a more compact representation of individuals. Our empirical study shows that the usage of motif genes improves coverage both for Evolutionary Algorithms and random approaches. In particular, it also shows that NSGA‐II, the multi‐objective evolutionary algorithm used by Sapienz, does not have a clear improvement over other algorithms. In terms of number of crashes detected, our study shows that both NSGA‐II and Random Search perform similarly. While the usage of motif genes improves the crash detection of algorithms, it is not enough to make it statistically significant. These facts cast doubts about the use of Evolutionary Algorithms in the context of Android test generation and suggest that motif genes can have a great impact on the overall effectiveness.

Several tools have been proposed to automatically test Android applications, achieving outstanding results in terms of both code coverage and crash discovery. While useful for crash reproduction and bug-fixing, these tools usually do not present the generated interactions in a format that motivates developers to read and modify such tests later on. This hinders the ability of developers to add those tests to their existing test suites, or adapt them to new scenarios - common practices in modern software development where tests are maintained and evolve alongside production code.