In an object-oriented program, a unit test often consists of a sequence of method calls that create and mutate objects, then use them as arguments to a method under test. It is challenging to automatically generate sequences that are legal and behaviorally-diverse, that is, reaching as many different program states as possible.

No abstract available

Continuous testing uses excess cycles on a developer's workstation to continuously run regression tests in the background, providing rapid feedback about test failures as code is edited. It reduces the time and energy required to keep code well-tested, and it prevents regression errors from persisting uncaught for long periods of time.

Test factoring creates fast, focused unit tests from slow system-widetests; each new unit test exercises only a subset of the functionalityexercised by the system test. Augmenting a test suite with factoredunit tests should catch errors earlier in a test run.One way to factor a test is to introduce 'mock' objects. If a testexercises a component T, which interacts with another component E (the'environment'), the implementation of E can be replaced by a mock.The mock checks that T's calls to E are as expected, and it simulatesE's behavior in response. We introduce an automatic technique fortest factoring. Given a system test for T and E, and a record of T'sand E's behavior when the system test is run, test factoring generatesunit tests for T in which E is mocked. The factored tests can isolatebugs in T from bugs in E and, if E is slow or expensive, improve testperformance or cost.We have built an implementation of automatic dynamic test factoring for theJava language. Our experimental data indicates that it can reduce therunning time of a system test suite by up to an order of magnitude.

Continuous testing uses excess cycles on a developer's workstation to continuously run regression tests in the background, providing rapid feedback about test failures as source code is edited. It is intended to reduce the time and energy required to keep code well-tested and prevent regression errors from persisting uncaught for long periods of time. This paper reports on a controlled human experiment to evaluate whether students using continuous testing are more successful in completing programming assignments. We also summarize users' subjective impressions and discuss why the results may generalize.The experiment indicates that the tool has a statistically significant effect on success in completing a programming task, but no such effect on time worked. Participants using continuous testing were three times more likely to complete the task before the deadline than those without. Participants using continuous compilation were twice as likely to complete the task, providing empirical support to a common feature in modern development environments. Most participants found continuous testing to be useful and believed that it helped them write better code faster, and 90% would recommend the tool to others. The participants did not find the tool distracting, and intuitively developed ways of incorporating the feedback into their workflow.

Test factoring creates fast, focused unit tests from slow system-wide tests; each new unit test exercises only a subset of the functionality exercised by the system tests. Augmenting a test suite with factored unit tests, and prioritizing the tests, should catch errors earlier in a test run.One way to factor a test is to introduce mock objects. If a test exercises a component A, which is designed to issue queries against or mutate another component B, the implementation of B can be replaced by a mock. The mock has two purposes: it checks that A's calls to B are as expected, and it simulates B's behavior in response. Given a system test for A and B, and a record of A's and B's behavior when the system test is run, we would like to automatically generate unit tests for A in which B is mocked. The factored tests can isolate bugs in A from bugs in B and, if B is slow or expensive, improve test performance or cost.This paper motivates test factoring with an illustrative example, proposes a simple procedure for automatically generating mock objects for factored tests, and gives examples of how the procedure can be extended to produce more robust factored tests.

Testing is often performed frequently during development to ensure software reliability by catching regression errors quickly. However, stopping frequently to test also wastes time by holding up development progress. User studies on real development projects indicate that these two sources of wasted time account for 10-15% of development time. These measurements use a novel technique for computing the wasted extra development time incurred by a delay in discovering a regression error. We present a model of developer behavior that infers developer beliefs from developer behavior, and that predicts developer behavior in new environments - in particular, when changing testing methodologies or tools to reduce wasted time. Changing test ordering or reporting reduces wasted time by 4-41% in our case study. Changing the frequency with which tests are run can reduce wasted time by 31-82% (but developers cannot know the ideal frequency except after the fact). We introduce and evaluate a new technique, continuous testing, that uses spare CPU resources to continuously run tests in the background, providing rapid feedback about test failures as as source code is edited. Continuous testing reduced wasted time by 92-98%, a substantial improvement over the other approaches. We have integrated continuous testing into two development environments, and are beginning user studies to evaluate its efficacy. We believe it has the potential to reduce the cost and improve the efficacy of testing and, as a result, to improve the reliability of delivered systems.