Empirical software engineering research needs research guidelines to improve the research and reporting processes. We propose a preliminary set of research guidelines aimed at stimulating discussion among software researchers. They are based on a review of research guidelines developed for medical researchers and on our own experience in doing and reviewing software engineering research. The guidelines are intended to assist researchers, reviewers, and meta-analysts in designing, conducting, and evaluating empirical studies. Editorial boards of software engineering journals may wish to use our recommendations as a basis for developing guidelines for reviewers and for framing policies for dealing with the design, data collection, and analysis and reporting of empirical studies.

Investigations into the comparative performance of mathematical software often involve collection and analysis of data under circumstances where the behavior of the software is not understood and unexpected results are likely to arise.The recently developed statistical techniques of exploratory data analysis are well suited to exposing important regularities of such data.Several of these techniques are explained and illustrated.Nonlinear optmuzation algorithms are comphcated by nature, and data on the performance of some of them provide an opportunity to apply exploratory techniques and other techniquos of modern data analysis.The focus of this study is on one relatively small body of data from selected measurements by K. E. Hillstrom on five nonlinear optimization routines in solving one test problem, starting from each of twenty randomly chosen starting points.The variability of performance across optimizers is described, and the effect of starting points is exposed.The concluding discussion examines some aspects of the design issues in studying behavior of mathematical software and related data analysis problems.

Abstract The optimization scores for Knuth's random sample of inner loops are analysed to provide a unified comparison of the five optimization levels. The techniques used are those of exploratory data analysis, and their role in the analysis is discussed. As a consequence of the analysis, five rough groups of programs with different optimization behaviour are identified and tentatively characterized.