Inline tests validate individual program statements and expressions, and they detect many seeded faults (i.e., mutants) that unit tests miss in these target statements. ExLi is the only automated inline-test generation technique today; it carves inline tests from method-level unit tests that are written for methods that enclose target statements. Thus, ExLi cannot work for target statements that method-level unit tests do not cover. Also, the quality of ExLi-generated inline tests depends on the quality of method-level unit tests. We propose Smack to generate inline tests without relying on method-level unit tests. Smack is inspired by how inline tests are run: each is extracted with the target statement and run independently. Smack exploits this independence. Smack first extracts each target statement into a new method. Next, Smack applies a unit-test generator to the extracted method, which tends to have simpler control flow than the target statement’s enclosing method and carves inline tests from them. Finally, Smack transplants the resulting inline tests to be right after the target statement in the original enclosing method. We evaluate Smack on the same 957 target statements in 31 open-source projects that ExLi was evaluated on. Smack generates inline tests for 277 of 312 (88.8%) statements that ExLi cannot handle. These inline tests kill 96.7% of 1,815 mutants in these 312 target statements. Therefore, Smack improves on ExLi by extending the reach and fault-detection ability of inline-test generation. Smack also generates inline tests for 609 of 645 (94.4%) target statements that ExLi can handle. Smack-generated inline tests kill 83.1% of 2,844 mutants in these 645 target statements. 147 of these killed mutants survive ExLi-generated inline tests. So, Smack is also complementary to ExLi on target statements that ExLi can handle.

Inline tests validate single program statements and were shown to find single-statement bugs or kill mutants that unit tests miss. Inline tests complement unit tests by enabling testing at a finer program granularity level than methods. So, inline tests can more easily find faults in target statements that unit tests do not reach, or where errors do not propagate to unit tests’ oracles. But, the limitation to single statements means inline tests cannot validate data or control flow across code fragments—sequences of multiple statements in a method. We motivate the need for testing arbitrary fragments and propose block tests, which generalize inline tests and validate code fragments. To motivate, we discuss six software testing needs (e.g., due to increasing usage of lambdas in imperative code) for which unit tests are too coarse grained and inline tests are too fine grained. To bridge this gap, we propose syntax and semantics for specifying inputs, expected outputs, and scope of block tests. We also implement a block-test development kit (BDK) for writing and running block tests in Java. We evaluate block tests and BDK in two ways. First, we write 1,012 block tests for 346 fragments in 146 open-source projects. Developer written unit tests do not cover 58.7% of these fragments, and automated unit-test generation does not reach 46% of them even after 30.8 CPU days. But, each block test takes us 2.2 minutes to write and 0.9 seconds to run on average. Second, we use mutation testing to evaluate the fault-finding effectiveness of block tests in fragments that unit tests cover. Block tests kill 4,418 of 9,554 mutants that survived unit tests. These results provide initial but strong evidence on block tests’ feasibility and utility. We outline an agenda for future research on block testing.