Work on automated test generation has produced several tools capable of generating test data which achieves high structural coverage over a program. In the absence of a specification, developers are expected to manually construct or verify the test oracle for each test input. Nevertheless, it is assumed that these generated tests ease the task of testing for the developer, as testing is reduced to checking the results of tests. While this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the limited adoption in industry indicates this assumption may not be correct, and calls into question the practical value of test generation tools. To investigate this issue, we performed two controlled experiments comparing a total of 97 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, E vo S uite . We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.

We describe an automated approach for detecting concurrency defects from design diagrams of a software, in particular, sequence diagrams. From a given sequence diagram, we automatically infer a formal, parallel specification that generalizes the communication behavior that is designed informally and incompletely in the diagram. We model-check the parallel specification against generic concurrency defect patterns. No additional specification of the software is needed. We present several case-studies to evaluate our approach. The results show that our approach is technically feasible, and effective in detecting nasty concurrency defects at the design level.

Automated test generation techniques can efficiently produce test data that systematically cover structural aspects of a program. In the absence of a specification, a common assumption is that these tests relieve a developer of most of the work, as the act of testing is reduced to checking the results of the tests. Although this assumption has persisted for decades, there has been no conclusive evidence to date confirming it. However, the fact that the approach has only seen a limited uptake in industry suggests the contrary, and calls into question its practical usefulness. To investigate this issue, we performed a controlled experiment comparing a total of 49 subjects split between writing tests manually and writing tests with the aid of an automated unit test generation tool, EvoSuite. We found that, on one hand, tool support leads to clear improvements in commonly applied quality metrics such as code coverage (up to 300% increase). However, on the other hand, there was no measurable improvement in the number of bugs actually found by developers. Our results not only cast some doubt on how the research community evaluates test generation tools, but also point to improvements and future work necessary before automated test generation tools will be widely adopted by practitioners.

We present our ALCHEMY platform that supports the automatic parallelization of R programs during execution. Parallelization occurs fully transparent to the user. Different parallelization techniques can be implemented as modules, linked into the platform, and combined with each other. The parallelization analysis modules and code transformation modules use a new intermediate representation for sequential and parallelized R code. Successfully parallelized parts of the R program are executed on a multicore processor, the results and the remaining sequential parts are fed back into the standard R interpreter and evaluated to completion. This way, an R user can benefit from multiprocessor performance without writing a single line of parallel code. At this stage of the research project, the main goal is to enable ample experimentation with different approaches to the automatic parallelization of scripting languages such as R.

We compute optimal scheduling policies for software development projects. We use reinforcement learning as the optimization technique. Our approach is based on a formal, stochastic scheduling model that explicitly captures the strong feedback between the tasks in software development ("ripple effects"). For sample projects, we compute the optimal policy, simulate the project, and analyze the task assignments that are made by the optimal policy. We find that optimal policies typically assign tasks according to the past performance of the developers and the characteristics of the software design. In particular, we address the problem of when to schedule large or strongly coupled components. We also sketch approaches to the optimization of large projects.

In this paper, we highlight the application potential of process simulation techniques for software cybernetics research. Software engineering has seen many fruitful applications of simulation when modeling, understanding, and improving the software development process. In particular, process simulation has proven to be a valuable and efficient tool in our own software cybernetics research, having helped us to understand how scheduling policies actually behave in our discrete-time Markov decision process model for software projects. We outline how to advance the use of process simulation in our model to a much higher level: when computing optimal scheduling policies, simulation can be applied in the optimization step of the dynamic programming algorithms in order to save computation time. This approach resembles optimization techniques from the field of reinforcement learning, providing further evidence of the potential of simulation in software cybernetics.

The time needed to complete a software development project is hard to estimate in practice. In addition to the uncertainty about the duration of the individual development steps, software projects often suffer from unexpected rework and delays. As a result, project scheduling remains a difficult task for the managers of software projects. In this paper, we compute optimal scheduling strategies for a set of sample software projects. The computations are based on a stochastic Markov decision model for software projects which focuses on capturing the feedback between concurrent development activities. The optimal strategies are computed using a variant of the value iteration algorithm from stochastic dynamic programming. Since the underlying process model is stochastic, the strategies are stochastically optimal, that is, they minimize the expected project duration. The ultimate goal of this research is to develop guidelines for managers how to schedule their software projects under uncertainty in the best possible way. The sample projects are similar, but differ in certain characteristics of the project or product, such as the strength of the coupling between the components or the degree of specialization of the teams on the tasks. By using a set of related projects, we can study how the project characteristics influence the optimal scheduling decisions in a project. In addition to computing the optimal scheduling policies for the sample projects, we use extensive simulations to compare the performance of the optimal policy against the so-called list policies for each given setting. List policies are a simple, but commonly used class of scheduling policies. For our sample projects, we find that the best list policy in general is not optimal. The performance gap is the larger the higher the degree of specialization of the teams is. On the other hand, the stronger the coupling between the components, the smaller is the improvement which the optimal policy achieves over the best list policy.

Pair programmers need a "warmup phase" before the pair can work at full speed. The length of the learning interval varies, depending on how experienced the developers are with pair programming and how familiar they are with each other. We study how large the impact of the lower pair productivity during warmup is on the business value of the pair programming project. To this end, we extend our net present value model for pair programming to explicitly include a learning interval for pairs. We then carry out a simulation study where we vary the shape of the learning curve, the length of the learning interval, the final productivity level of the pairs, the market pressure, and the size of the workforce. Our simulations show that the cost of the warmup phase is reasonably small compared to the project value, but nonetheless must be taken into account when estimating the project. Our results also suggest that the learning overhead is not an obstacle to introducing and using pair programming in a real environment.

We view the problem of estimating the defect content of a document after an inspection as a machine learning problem: The goal is to learn from empirical data the relationship between certain observable features of an inspection (such as the total number of different defects detected) and the number of defects actually contained in the document. We show that some features can carry significant nonlinear information about the defect content. Therefore, we use a nonlinear regression technique, neural networks, to solve the learning problem. To select the best among all neural networks trained on a given data set, one usually reserves part of the data set for later cross-validation; in contrast, we use a technique which leaves the full data set for training. This is an advantage when the data set is small. We validate our approach on a known empirical inspection data set. For that benchmark, our novel approach clearly outperforms both linear regression and the current standard methods in software engineering for estimating the defect content, such as capture-recapture. The validation also shows that our machine learning approach can be successful even when the empirical inspection data set is small.

To cut development cost and meet tight deadlines in short staffed software projects, managers must optimize the project schedule. Scheduling a software project is extremely difficult, though, because the time needed to complete a software development activity is hard to estimate. Often, the completion of a task is delayed because of unanticipated rework caused by feedback between activities in the process.

From a project economics point of view, the most important practices of Extreme Programming (XP) are Pair Programming and Test-Driven Development. Pair Programming leads to a large increase in the personnel cost, and Test-Driven Development adds to the development effort. On the other hand, pp can speed the project up, both pp and Tdd can reduce the defect density of the code. Can the increased cost of XP be balanced by its shorter time to market and higher code quality? To answer this question, we construct a new model for the business value of software projects. We then analyze the cost and benefit of XP by applying our model to a realistic sample project. We systematically vary important model parameters to provide a sensitivity analysis. Our analysis shows that the economic value of, XP strongly depends on how large the XP speed and defect advantage really are. We also find that the market pressure is an important factor when assessing the business value of XP., Our study provides clear guidelines for managers when to consider using XP -- or better not.

We present a discrete simulation model for software projects which explicitly takes a scheduling strategy as input. The model represents varying staff skill levels, component coupling, rework caused by design changes, and changing task assignments. The simulation model is implemented in the ModL language of the general-purpose graphical simulation tool EXTEND. The simulations provide quick feedback about the impact which the scheduling strategy will have on the progress and completion time of a given software project. Using the model, a manager can compare different strategies and choose the one which is best for his next project. As an illustration how to apply the simulation model, we systematically study the performance of various list policies for a small sample project. We provide a detailed analysis of the task assignments which actually occur in the simulations. In addition, the example provides clear evidence that strategies which are more adaptive to the current project state than list policies will yield improved schedules. This result suggests to apply dynamic optimization techniques when scheduling software projects.

We present a novel method for estimating the number of defects contained in a document using the results of an inspection of the document. The method is empirical, being based on observations made during past inspections of comparable documents. The method yields an interval estimate, that is, a whole range of values which is likely to contain the true value of the number of defects in the document. We also derive point estimates from the interval estimate. The method is validated using a known empirical inspection dataset and clearly outperforms existing approaches for estimating the defect content after inspections.

A probabilistic scheduling model for software projects is presented. The model explicitly takes a scheduling strategy as input. When the scheduling strategy is fired, the model outputs a probability distribution for the project completion time or cost. By applying stochastic optimization techniques, schedules for software projects can be computed which minimize the development time or cost with a given staff.