This study aims to compare the use of targeted manual searches with broad automated searches, and to assess the importance of grey literature and breadth of search on the outcomes of SLRs. We used a participant-observer multi-case embedded case study. Our two cases were a tertiary study of systematic literature reviews published between January 2004 and June 2007 based on a manual search of selected journals and conferences and a replication of that study based on a broad automated search. Broad searches find more papers than restricted searches, but the papers may be of poor quality. Researchers undertaking SLRs may be justified in using targeted manual searches if they intend to omit low quality papers; if publication bias is not an issue; or if they are assessing research trends in research methodologies.

There is little empirical knowledge of the effectiveness of the object-oriented paradigm. To conduct a systematic review of the literature describing empirical studies of this paradigm. We undertook a Mapping Study of the literature. 138 papers have been identified and classified by topic, form of study involved, and source. The majority of empirical studies of OO (object oriented software) concentrate on metrics, relatively few consider effectiveness.

This workshop is concerned with defining the procedures that are needed to establish a sound empirical foundation for the practices of Software Engineering. Our goal is to begin building a community that will review, analyse, codify and promulgate software engineering experiences as well as to identify the processes and infrastructure that are needed to support these activities.

This paper reports two trials of an evaluation framework intended to evaluate novel software applications. The evaluation framework was originally developed to evaluate a risk-based software bidding model, and our first trial of using the framework was our evaluation of the bidding model. We found that the framework worked well as a validation framework but needed to be extended before it would be appropriate for evaluation. Subsequently, we compared our framework with a recently completed evaluation of a software tool undertaken as part of the Framework V CLARiFi project. In this case, we did not use the framework to guide the evaluation; we used the framework to see whether it would identify any weaknesses in the actual evaluation process. Activities recommended by the framework were not undertaken in the order suggested by the evaluation process and we found problems relating to that oversight surfaced during the tool evaluation activities. Our experiences suggest that the framework has some benefits but it also requires further practical testing.

This paper discusses the issues involved in evaluating a software bidding model. We found it difficult to assess the appropriateness of any model evaluation activities without a baseline or standard against which to assess them. This paper describes our attempt to construct such a baseline. We reviewed evaluation criteria used to assess cost models and an evaluation framework that was intended to assess the quality of requirements models. We developed an extended evaluation framework and an associated evaluation process that will be used to evaluate our bidding model. Furthermore, we suggest the evaluation framework might be suitable for evaluating other models derived from expert-opinion based influence diagrams.

We discuss a method of developing a software bidding model that allows users to visualize the uncertainty involved in pricing decisions and make appropriate bid/no bid decisions. We present a generic bidding model developed using the modeling method. The model elements were identified after a review of bidding research in software and other industries. We describe the method we developed to validate our model and report the main results of our model validation, including the results of applying the model to four bidding scenarios.

This paper proposes a method for specifying models of software data sets in order to capture the definitions and relationships among software measures. We believe a method of defining software data sets is necessary to ensure that software data are trustworthy. Software companies introducing a measurement program need to establish procedures to collect and store trustworthy measurement data. Without appropriate definitions it is difficult to ensure data values are repeatable and comparable. Software metrics researchers need to maintain collections of software data sets. Such collections allow researchers to assess the generality of software engineering phenomena. Without appropriate safeguards, it is difficult to ensure that data from different sources are analyzed correctly. These issues imply the need for a standard method of specifying software data sets so they are fully documented and can be exchanged with confidence. We suggest our method of defining data sets can be used as such a standard. We present our proposed method in terms of a conceptual entity-relationship data model that allows complex software data sets to be modeled and their data values stored. The standard can, therefore, contribute both to the definition of a company measurement program and to the exchange of data sets among researchers.