Support vector machines (SVM) are often applied in the context of machine learning analysis of various data. Given the nature of SVMs, these operate always in the sub-interpolation range as a machine learning method. Here we explore the impact of structural complexity on the performance and statistical reliability of SVMs applied for text mining. We set a theoretical framework for our analysis. We found experimentally that the statistical reliability and performance reduce exponentially with the increase of the structural complexity of the SVMs. This is an important result for the understanding of how the prediction error of SVM predictive data models behaves.

Other disciplines commonly employ secondary studies to address the needs of practitioners and policy-makers. Since being adopted by software engineering in 2004, many have been undertaken by researchers. To assess how the role of secondary studies in software engineering has evolved. We examined a sample of 131 secondary studies published in a set of five major software engineering journals for the years 2010, 2015 and 2020. These were categorised by their type (e.g. mapping study), their research focus (quantitative/qualitative and practice/methodological), as well as the experience of the first authors. Secondary studies are now a well-established research tool. They are predominantly qualitative and there is extensive use of mapping studies to profile research in particular areas. A significant number are clearly produced as part of postgraduate study, although experienced researchers also conduct many secondary studies. They are sometimes also used as part of a multi-method study. Existing guidelines largely focus upon quantitative systematic reviews. Based on our findings, we suggest that more guidance is needed on how to conduct, analyse, and report qualitative secondary studies.

Previous studies have raised concerns about the analysis and meta-analysis of crossover experiments and we were aware of several families of experiments that used crossover designs and meta-analysis. To identify families of experiments that used meta-analysis, to investigate their methods for effect size construction and aggregation, and to assess the reproducibility and validity of their results. We performed a systematic review (SR) of papers reporting families of experiments in high quality software engineering journals, that attempted to apply meta-analysis. We attempted to reproduce the reported meta-analysis results using the descriptive statistics and also investigated the validity of the meta-analysis process. Out of 13 identified primary studies, we reproduced only five. Seven studies could not be reproduced. One study which was correctly analyzed could not be reproduced due to rounding errors. When we were unable to reproduce results, we provide revised meta-analysis results. To support reproducibility of analyses presented in our paper, it is complemented by the reproducer R package. Meta-analysis is not well understood by software engineering researchers. To support novice researchers, we present recommendations for reporting and meta-analyzing families of experiments and a detailed example of how to analyze a family of 4-group crossover experiments.

Background Examples of questionable statistical practice, when published in high quality software engineering (SE) journals, may lead to novice researchers adopting incorrect statistical practices.

Background:: Statistical validity and model complexity are both important concepts to enhanced understanding and correctness assessment of computational models. However, information about these are often missing from publications applying machine learning.

There have been many changes in statistical theory in the past 30 years, including increased evidence that non-robust methods may fail to detect important results. The statistical advice available to software engineering researchers needs to be updated to address these issues. This paper aims both to explain the new results in the area of robust analysis methods and to provide a large-scale worked example of the new methods. We summarise the results of analyses of the Type 1 error efficiency and power of standard parametric and non-parametric statistical tests when applied to non-normal data sets. We identify parametric and non-parametric methods that are robust to non-normality. We present an analysis of a large-scale software engineering experiment to illustrate their use. We illustrate the use of kernel density plots, and parametric and non-parametric methods using four different software engineering data sets. We explain why the methods are necessary and the rationale for selecting a specific analysis. We suggest using kernel density plots rather than box plots to visualise data distributions. For parametric analysis, we recommend trimmed means, which can support reliable tests of the differences between the central location of two or more samples. When the distribution of the data differs among groups, or we have ordinal scale data, we recommend non-parametric methods such as Cliff’s δ or a robust rank-based ANOVA-like method.

Background: Since the introduction of the systematic review process to Software Engineering in 2004, researchers have investigated a number of ways to mitigate the amount of effort and time taken to filter through large volumes of literature.

Systematic review is a widely used research method in software engineering, and in other disciplines, for identifying and analysing empirical evidence. The method is data intensive and time consuming, and hence is usually supported by a wide range of software-based tools. However, systematic reviewers have found that finding and selecting tools can be quite challenging.

Background: A number of software tools are being developed to support systematic reviewers within the software engineering domain. However, at present, we are not sure which aspects of the review process can most usefully be supported by such tools or what characteristics of the tools are most important to reviewers. Aim: The aim of the study is to explore the scope and practice of tool support for systematic reviewers in other disciplines. Method: Researchers with experience of performing systematic reviews in Healthcare and the Social Sciences were surveyed. Qualitative data was collected through semi-structured interviews and data analysis followed an inductive approach. Results: 13 interviews were carried out. 21 software tools categorised into one of seven types were identified. Reference managers were the most commonly mentioned tools. Features considered particularly important by participants were support for multiple users, support for data extraction and support for tool maintenance. The features and importance levels identified by participants were compared with those proposed for tools to support systematic reviews in software engineering. Conclusions: Many problems faced by systematic reviewers in other disciplines are similar to those faced in software engineering. There is general consensus across domains that improved tools are needed.

Background The labour intensive and error prone nature of the systematic review process has led to the development and use of a range of tools to provide automated support.

Context: For the outcomes of systematic literature reviews to be of use for practitioners, we need to develop models for addressing the needs of Knowledge Translation (KT). Aim: To identify some of the key issues that need to be addressed by a KT process for software engineering (SE) and possible routes for achieving these. Method: We have examined some of the models used in other disciplines, and suggested a possible interpretation for software engineering. Results: We propose a model for achieving KT. Conclusions: Research with industry and commerce is needed to explore how this can be realised.

Context: Due to the lack of suitably skilled participants, software engineering experiments often lack the statistical power needed to detect the levels of effect that may be encountered. Aim: To investigate whether this can be remedied by running an experiment across multiple sites, organised as a single study rather than as a set of replications. Method: We performed a `trial' of the idea using a topic (structured abstracts) that some of us had studied previously and which required no participant training. We used five sites, each with 16 participants. Results: We were able to demonstrate the benefits of increased statistical power (and of structured abstracts). We report on our experiences with designing and conducting the study and identify some key lessons about how future studies of this form might be organised. Conclusions: The distributed model offers a flexible, robust form that is capable of delivering better statistical power than would be achieved by running a set of parallel replicated studies.

Background: Systematic literature reviews (SLRs) have become an established methodology in software engineering (SE) research however they can be very time consuming and error prone. Aim: The aims of this study are to identify and classify tools that can help to automate part or all of the SLR process within the SE domain. Method: A mapping study was performed using an automated search strategy plus snowballing to locate relevant papers. A set of known papers was used to validate the search string. Results: 14 papers were accepted into the final set. Eight presented text mining tools and six discussed the use of visualisation techniques. The stage most commonly targeted was study selection. Only two papers reported an independent evaluation of the tool presented. The majority were evaluated through small experiments and examples of their use. Conclusions: A variety of tools are available to support the SLR process although many are in the early stages of development and usage.

Context: Several text books and papers published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers over the time period 1993 to 2010. Method: Seventy experimental and quasi-experimental papers published in four general software engineering journals in the years 1992-2002 and 2006-2010 were each assessed for quality by three empirical software engineering researchers using two quality assessment methods (a questionnaire-based method and a subjective overall assessment). Regression analysis was used to assess the relationship between paper quality and the year of publication, publication date group (before 2003 and after 2005), source journal, average coauthor experience, citation of statistical text books and papers, and paper length. The results were validated both by removing papers for which the quality score appeared unreliable and using an alternative quality measure. Results: Paper quality was significantly associated with year, citing general statistical texts, and paper length (p <; 0.05). Paper length did not reach significance when quality was measured using an overall subjective assessment. Conclusions: The quality of experimental and quasi-experimental software engineering papers appears to have improved gradually since 1993.

Context: We have been undertaking a series of case studies to investigate the value of mapping (scoping) studies in software engineering. Our previous studies have assessed these using the subjective opinions of researchers. Objective: In order to provide a more objective assessment of value, for this study, we used the results of a systematic mapping study to investigate how well mapping studies identify clusters of related studies and to what extent such clusters are complete. Method: In this participant-observer case study, we undertook a mapping study of unit testing and regression testing empirical studies, which we compared with a previous expert literature review and with six other mapping studies and systematic literature reviews (SLRs) that addressed overlapping topics. Results: Our mapping study found more clusters than the expert literature review although it benefited from the set of studies identified by the expert review when refining our search process. The set of studies found by our searches were less complete than those found by SLRs addressing more specific topics, although we found some studies missed by those SLRs. Conclusions: Researchers undertaking systematic reviews and mapping studies should make use of related systematic reviews and mapping studies to identify known studies in order to refine search strings and validate search results. For completeness and traceability, mapping studies should keep a record of all multiple reports of a single study. Meta-analyses and other systematic literature reviews undertaking detailed aggregation should report on candidate primary studies that were rejected in the final screening process, as well as candidate studies that were included. This helps ensure the repeatability of aggregation results.

Context: There has been an increase in research into global software development (GSD) and in the number of systematic literature reviews (SLRs) addressing this topic. Objective: The aim of this research is to catalogue GSD SLRs in order to identify the topics covered, the active researchers, the publication vehicles, and to assess the quality of the SLRs identified. Method: We performed a broad automated search to find SLRs dealing with GSD. We differentiate between SLR studies and papers reporting those studies. Data relating to each of the following was extracted and synthesized from each study: authors and their affiliation at the time of publication, the journal or conference in which the paper was published, the quality of each study and the main GSD study topic. Results: Twenty-four GSD SLR studies and 37 papers reporting those studies were identified. Major GSD topics covered include: (1) organizational environment, (2) project execution, and (3) project planning and control. The main research groups are based in Brazil (17), Ireland (8), and Sweden (7). Conclusions: GSD SLR studies are most frequently reported in the International Conference on Global Software Engineering and IEEE Software; the two most popular topics for research are risk factors due to the organizational environment and the development process. The most active researchers are based in Brazil. The quality of the SLR studies has not changed over time.

Context: In teaching about software engineering we currently make little use of any empirical knowledge. Aim: To examine the outcomes available from the use of Evidence-Based Software Engineering (EBSE) practices, so as to identify where these can provide support for, and inform, teaching activities. Method: We have examined all known secondary studies published up to the end of 2009, together with those published in major journals to mid-2011, and identified where these provide practical results that are relevant to student needs. Results: Starting with 145 candidate systematic literature reviews (SLRs), we were able to identify and classify potentially useful teaching material from 43 of them. Conclusions: EBSE can potentially lend authority to our teaching, although the coverage of key topics is uneven. Additionally, mapping studies can provide support for research-led teaching.

During systematic literature reviews it is necessary to assess the quality of empirical papers. Current guidelines suggest that two researchers should independently apply a quality checklist and any disagreements must be resolved. However, there is little empirical evidence concerning the effectiveness of these guidelines. This paper investigates the three techniques that can be used to improve the reliability (i.e. the consensus among reviewers) of quality assessments, specifically, the number of reviewers, the use of a set of evaluation criteria and consultation among reviewers. We undertook a series of studies to investigate these factors. Two studies involved four research papers and eight reviewers using a quality checklist with nine questions. The first study was based on individual assessments, the second study on joint assessments with a period of inter-rater discussion. A third more formal randomised block experiment involved 48 reviewers assessing two of the papers used previously in teams of one, two and three persons to assess the impact of discussion among teams of different size using the evaluations of the “teams” of one person as a control. For the first two studies, the inter-rater reliability was poor for individual assessments, but better for joint evaluations. However, the results of the third study contradicted the results of Study 2. Inter-rater reliability was poor for all groups but worse for teams of two or three than for individuals. When performing quality assessments for systematic literature reviews, we recommend using three independent reviewers and adopting the median assessment. A quality checklist seems useful but it is difficult to ensure that the checklist is both appropriate and understood by reviewers. Furthermore, future experiments should ensure participants are given more time to understand the quality checklist and to evaluate the research papers.

Background: One of the anticipated benefits of systematic literature reviews (SLRs) is that they can be conducted in an auditable way to produce repeatable results. Aim: This study aims to identify under what conditions SLRs are likely to be stable, with respect to the primary studies selected, when used in software engineering. The conditions we investigate in this report are when novice researchers undertake searches with a common goal. Method: We undertook a participant-observer multi-case study to investigate the repeatability of systematic literature reviews. The 'cases' in this study were the early stages, involving identification of relevant literature, of two SLRs of unit testing methods. The SLRs were performed independently by two novice researchers. The SLRs were restricted to the ACM and IEEE digital libraries for the years 1986-2005 so their results could be compared with a published expert literature review of unit testing papers. Results: The two SLRs selected very different papers with only six papers out of 32 in common, and both differed substantially from a published secondary study of unit testing papers finding only three of 21 papers. Of the 29 additional papers found by the novice researchers, only 10 were considered relevant. The 10 additional relevant papers would have had an impact on the results of the published study by adding three new categories to the framework and adding papers to three, otherwise empty, cells. Conclusions: In the case of novice researchers, having broadly the same research question will not necessarily guarantee repeatability with respect to primary studies. Systematic reviews must be careful to report their search process fully or they will not be repeatable. Missing papers can have a significant impact on the stability of the results of a secondary study.

Background: Teaching programming to novices is a difficult task due to the complex nature of the subject, as negative stereotypes are associated with programming and because introductory programming courses often fail to encourage student understanding. Aim: This study investigates the effectiveness of using robots as tools to aid the process of teaching programming and to determine whether such technology can help to overcome the current barriers for learners in this context. Method: The Systematic Literature Review (SLR) methodology has been selected to discover how effective the use of robotics has been in the teaching of introductory programming concepts. Nine electronic databases, the proceedings from six conferences and two journals have been searched for literature relevant to the study. Results: After applying inclusion and exclusion criteria 34 articles have been accepted in the SLR. 74% of included literature report robots to be an effective teaching tool and one that can help novice programmers in their studies. Conclusion: Robots can be a powerful and effective tool when used in an introductory programming course but the potential remains to further investigate methods for their implementation. Thoughts on the use of the SLR methodology from the perspective of a PhD student are also given.

Abstract The Unified Modeling Language (UML) was created on the basis of expert opinion and has now become accepted as the ‘standard’ object‐oriented modelling notation. Our objectives were to determine how widely the notations of the UML, and their usefulness, have been studied empirically, and to identify which aspects of it have been studied in most detail. We undertook a mapping study of the literature to identify relevant empirical studies and to classify them in terms of the aspects of the UML that they studied. We then conducted a systematic literature review, covering empirical studies published up to the end of 2008, based on the main categories identified. We identified 49 relevant publications, and report the aggregated results for those categories for which we had enough papers—metrics,comprehension,model quality,methods and toolsandadoption. Despite indications that a number of problems exist with UML models, researchers tend to use the UML as a ‘given’ and seem reluctant to ask questions that might help to make it more effective. Copyright © 2010 John Wiley & Sons, Ltd.

Context: The authors wanted to assess whether the quality of published human-centric software engineering experiments was improving. This required a reliable means of assessing the quality of such experiments. Aims: The aims of the study were to confirm the usability of a quality evaluation checklist, determine how many reviewers were needed per paper that reports an experiment, and specify an appropriate process for evaluating quality. Method: With eight reviewers and four papers describing human-centric software engineering experiments, we used a quality checklist with nine questions. We conducted the study in two parts: the first was based on individual assessments and the second on collaborative evaluations. Results: The inter-rater reliability was poor for individual assessments but much better for joint evaluations. Four reviewers working in two pairs with discussion were more reliable than eight reviewers with no discussion. The sum of the nine criteria was more reliable than individual questions or a simple overall assessment. Conclusions: If quality evaluation is critical, more than two reviewers are required and a round of discussion is necessary. We advise using quality criteria and basing the final assessment on the sum of the aggregated criteria. The restricted number of papers used and the relatively extensive expertise of the reviewers limit our results. In addition, the results of the second part of the study could have been affected by removing a time restriction on the review as well as the consultation process.

We identify three challenges related to the provenance of the material we use in teaching software engineering. We suggest that these challenges can be addressed by using evidence-based software engineering (EBSE) and its primary tool of systematic literature reviews (SLRs). This paper aims to assess the educational and scientific value of undergraduate and postgraduate students undertaking a specific form of SLR called a mapping study. Using a case study methodology, we asked three postgraduate students and three undergraduates and their supervisor to complete a questionnaire concerning the educational value of mapping studies and any problems they experienced. Students found undertaking a mapping study to be a valuable experience providing both reusable research skills and a good overview of a research topic. Postgraduates found it useful as a starting point for their studies. Undergraduates reported problems undertaking the study in the required timescales. Searching and classifying the literature was difficult.

Access controls are not sufficient to prevent the release of secret information from an information system unless they address the problem of inference. An inference strategy is a method by which a user can infer secret information using the information which they are allowed to access through the access control mechanism. The aim of this paper is to collate and categorise the set of inference strategies in the existing literature. The systematic literature review (SLR) methodology is used to identify and categorise known inference strategies. The SLR search found 63 sources, and 127 inference strategies were extracted from these sources, which have been categorised into 11 different categories. Recording the inference strategy processes has abstracted the detail which ties inference strategies to an information system. Using this abstraction, it should be feasible to determine the level of inference protection offered by information systems in general.

The aims of this research were to investigate the applicability of the systematic literature review (SLR) process within the constraints of a 13-week masterpsilas level project and to aggregate evidence about the effectiveness of pair programming for teaching introductory programming. It was found that, with certain modifications to the process, it was possible to undertake an SLR within a limited time period and to produce valid results. Based on pre-defined inclusion and exclusion criteria, the student found 28 publications reporting empirical studies of pair programming, of which nine publications were used for data extraction and analysis. Results of the review indicates that whilst pair programming has little effect on the marks obtained for examinations and assignments, it can significantly improve the pass and retention rates and the studentspsila confidence and enjoyment of programming. Following the student study, experienced reviewers re-applied the inclusion and exclusion criteria to the 28 publications and carried out data extraction and synthesis using the resulting papers. A comparison of the studentpsilas results and those of the experienced reviewers is presented.

This study aims to compare the use of targeted manual searches with broad automated searches, and to assess the importance of grey literature and breadth of search on the outcomes of SLRs. We used a participant-observer multi-case embedded case study. Our two cases were a tertiary study of systematic literature reviews published between January 2004 and June 2007 based on a manual search of selected journals and conferences and a replication of that study based on a broad automated search. Broad searches find more papers than restricted searches, but the papers may be of poor quality. Researchers undertaking SLRs may be justified in using targeted manual searches if they intend to omit low quality papers; if publication bias is not an issue; or if they are assessing research trends in research methodologies.

There is little empirical knowledge of the effectiveness of the object-oriented paradigm. To conduct a systematic review of the literature describing empirical studies of this paradigm. We undertook a Mapping Study of the literature. 138 papers have been identified and classified by topic, form of study involved, and source. The majority of empirical studies of OO (object oriented software) concentrate on metrics, relatively few consider effectiveness.

A consequence of the growing number of empirical studies in software engineering is the need to adopt systematic approaches to assessing and aggregating research outcomes in order to provide a balanced and objective summary of research evidence for a particular topic. The paper reports experiences with applying one such approach, the practice of systematic literature review, to the published studies relevant to topics within the software engineering domain. The systematic literature review process is summarised, a number of reviews being undertaken by the authors and others are described and some lessons about the applicability of this practice to software engineering are extracted. The basic systematic literature review process seems appropriate to software engineering and the preparation and validation of a review protocol in advance of a review activity is especially valuable. The paper highlights areas where some adaptation of the process to accommodate the domain-specific characteristics of software engineering is needed as well as areas where improvements to current software engineering infrastructure and practices would enhance its applicability. In particular, infrastructure support provided by software engineering indexing databases is inadequate. Also, the quality of abstracts is poor; it is usually not possible to judge the relevance of a study from a review of the abstract alone.

Abstract The initial use of a service‐oriented system, which collates information from three databases, highlighted a performance problem with query response time for queries that selected more than ten attributes from a single data source. Our aim was to find the cause of the performance problem within the service‐oriented system. The challenges that were encountered when attempting to locate such a problem are described. Our approach to finding the performance problem, while addressing these challenges, was based on systematically devising and applying measurements to the system. We developed strategies that addressed the challenges and the inherent difficulty of measuring performance in a service‐oriented system. The cause of the performance problem was a buffer used by a JDBC‐ODBC driver to connect Java code to a Microsoft Access database. Three strategies successfully addressed the challenges of measuring performance in the service‐oriented system. In particular, routing queries to a local data source eliminated networking problems. Creating a model of the system allowed performance measurements to continue concurrently with demonstrations of the system, and with its further development. Finally, porting the system identified the part that had a detrimental effect on performance. Copyright © 2006 John Wiley & Sons, Ltd.

With increasing numbers of Web users, there is a necessity to improve their Web site navigation experience over the Internet and a range of Web applications have emerged recently for this purpose. Many researchers have stressed the importance of identifying semantic relatedness of Web pages in such Web applications as Web site navigation, automatic tour generation and adaptive Web applications. One approach to identifying semantic relatedness between documents is to use lexical databases and lexical chains. For example, an approach using lexical chains has been proposed by Green for identifying paragraph similarity in a document [1]. However, due to the unacceptable length of time needed for lexical chaining and the difficulty of global representation of documents, Green used synset weight vectors to compare semantic relatedness between two documents. But his approach to identifying paragraph similarities can be extended to identify semantic similarities between documents. In this study, an approach to identifying semantic similarity between Web pages incorporating weighted lexical chains (SRWLC) and document properties based on reiteration, density, length and semantic distance is proposed. The two approaches (the proposed approach and Green’s approach - SR Green ) were empirically compared by determining the semantic relatedness of Web pages using human subjects. The research hypothesis of this research is that the proposed approach identifies significantly more semantically related pages with a higher precision than the approach that has been proposed by Green. The null hypothesis is that there is no significant different in identification of semantically related pages between the two approaches. In this context precision is defined as the proportion of retrieved pages that are relevant. Web pages belonging to the Department of Computer Science, Keele University are used for the empirical evaluation of the two methods. The semantic relatedness of all pages was identified using both approaches and a Web-based page categorisation exercise using human subjects was carried out for this empirical evaluation. The evaluation is Web based, and therefore can be carried out on the subjects’ preferred Web browser at his or her preferred time & place. Therefore the distortion effects are minimised and the results of the evaluation are realistic and also can be reliably generalised to some extent. An invitation e-mail was sent to twenty subjects during the first week of October 2004 giving a link and guidelines for them to start the experiment. Once the link on the email was clicked, subjects were shown the initial Web page, giving an introduction to the experiment and instructions on how to continue. Twelve out of twenty invited subjects completed the experiment. Two subjects attempted the experiments but couldn’t finish because of network problems; another three subjects couldn’t finish because of time restrictions, and the other three subjects did not respond at all. Therefore responses from only twelve subjects are used for experimental evaluation. The Wilcoxon signed ranks test returns a p value of 0.004, indicating that the null hypothesis can be rejected, and that there is evidence to suggest that the SR WLC approach identifies significantly more semantically-related pages with higher precision than the SR Green approach. The SRWLC approach should be evaluated further using different Web site contents and language styles (e.g. American/British English). It would be interesting to use more subjects from different backgrounds to do the evaluation. This would determine whether the results of the evaluation are influenced by the human subject’s background, such as their status (student, staff, or other) his familiarity with the pages of the test database, gender differences or the level of English knowledge. Most importantly the approach is believed to be valid for more general browsing environments than a computer science Website and a wider study is desirable.

A systematic literature review is being carried out as part of a Ph.D. research project, which is focussing on the security problems caused by inference in distributed and autonomous information systems. Inference problems differ from other security problems because they do not directly violate access control mechanisms. For example, a user may infer an aircraft’s secret destination using unclassified values for the range of the aircraft and its direction of flight. The aim of the research is to investigate the effects of inference when information is collated from autonomous, heterogeneous and distributed information systems (Budgen et al., 2005).

Context: Making best use of the growing number of empirical studies in Software Engineering, for making decisions and formulating research questions, requires the ability to construct an objective summary of available research evidence. Adopting a systematic approach to assessing and aggregating the outcomes from a set of empirical studies is also particularly important in Software Engineering, given that such studies may employ very different experimental forms and be undertaken in very different experimental contexts.Objectives: To provide an introduction to the role, form and processes involved in performing Systematic Literature Reviews. After the tutorial, participants should be able to read and use such reviews, and have gained the knowledge needed to conduct systematic reviews of their own.Method: We will use a blend of information presentation (including some experiences of the problems that can arise in the Software Engineering domain), and also of interactive working, using review material prepared in advance.

WISER is a series of international workshops that focus on identifying and transferring techniques from other disciplines that might usefully be applied to software engineering research and practice.The workshops address this topic through presentations and discussions of both actual case studies and of ways in which potentially useful approaches can be identified, adapted and adopted within software engineering.

This paper describes an investigation of factors affecting the outcome of undergraduate student group projects. The data used relates to 2 cohorts of 2ndyear students. The factors addressed are group size, the range of abilities within a group, the presence/absence of women in a group and the mix of expertise (other than in computing) within a group. Results suggest that group size is not a significant factor in determining outcome, weak students do no better when grouped with more able students (than when grouped with those of similar ability), men only groups do not perform as well as groups having women in them and outcome is independent of the range of 'other' expertise within the group

This workshop is concerned with defining the procedures that are needed to establish a sound empirical foundation for the practices of Software Engineering. Our goal is to begin building a community that will review, analyse, codify and promulgate software engineering experiences as well as to identify the processes and infrastructure that are needed to support these activities.

We adapted a survey instrument developed by Timothy Lethbridge to assess the extent to which the education delivered by four UK universities matches the requirements of the software industry. We propose a survey methodology that we believe addresses the research question more appropriately than the one used by Lethbridge. In particular, we suggest that restricting the scope of the survey to address the question of whether the curricula for a specific university addressed the needs of its own students, allowed us to identify an appropriate target population. However, our own survey suffered from several problems. In particular the questions used in the survey are not ideal, and the response rate was poor. Although the poor response rate reduces the value of our results, our survey appears to confirm several of Lethbridge's observations with respect to the over-emphasis of mathematical topics and the under-emphasis on business topics. We also have a close agreement with respect to the relative importance of different software engineering topics. However the set of topics, that we found were taught far less than their importance would suggest, were quite different from the topics identified by Lethbridge.

We discuss what constitutes a 'software service model' and propose a framework that can accommodate both component-based and service-based systems, while also distinguishing between 'static' and 'dynamic' forms of service delivery. We then draw upon this, and upon our own experiences of developing service-based forms, to consider what characteristics a service architecture might possess

This paper reports on our experiences with using the emerging Web service technologies and tools to create a demonstration information broker system as part of our research into information management in a distributed environment. To provide a realistic context, we chose to study the use of information in the healthcare domain, and this context sets some challenging parameters and constraints for our research and for the demonstration system. In this paper, we both report on the extent to which existing Web service technologies have proved to be mature enough to meet these requirements, and also assess their current limitations.

We address the problem of large-scale data integration, where the data sources are unknown at design time, are from autonomous organisations, and may evolve. Experiments are described involving a demonstrator system in the field of health services data integration within the UK. Current Web services technology has been used extensively and largely successfully in these distributed prototype systems. The work shows that Web services provide a good infrastructure layer, but integration demands a higher level "broker" architectural layer; the paper identifies eight specific requirements for such an architecture that have emerged from the experiments, derived from an analysis of shortcomings which are collectively due to the static nature of the initial prototype. The way in which these are being met in the current version in order to achieve a more dynamic integration is described.

The design process in component-based software engineering needs to implicitly incorporate ways of reusing existing components. Reuse in turn implies the need to acquire knowledge about any components that might be incorporated into a system. Our paper describes an empirical study of how component documentation can be used to help develop that knowledge, using Unix processes as the source of documentation and components, and using both think-aloud and non-intrusive techniques to record subject behaviour. Our conclusions suggest how component documentation might be organised so as to assist with reuse, and identify further questions about its form.

As the software engineering community moves towards a programming philosophy in which integrators pull together disparate components from multiple sources, there is an increased need to control and record the configuration of systems as they evolve over time. This paper presents VACUUM, an approach to configuration management (CM) for components, and an architecture based around component warehouses, remote catalogues of components which can be accessed as plug-in services. It also describes a prototype implementation of the. VACUUM CM system and associated architecture. It is shown that this work begins to address those factors identified as important to growth in the component marketplace, and also meets many of the established requirements for CM systems.

The urgent need to change software easily to meet evolving business requirements requires a radical shift in the development of software, with a more demand-centric view leading to software which will be delivered as a service, within the framework of an open marketplace. We describe a service architecture and its rationale, in which components may be bound instantly, just at the time they are needed and then the binding may, be disengaged. This allows highly flexible software services to be evolved in "internet time". The paper focuses on early results: some of the aims have been demonstrated and amplified through an experimental implementation based on e-Speak, an existing and available technology. It is concluded that technology such as e-Speak provides a useful infrastructure that rapidly enabled us to demonstrate the basic operation and viability of our approach.

There is an urgent industrial need for new approaches to software evolution that will lead to far faster implementation of software changes. For the past 40 years, the techniques, processes and methods of software development have been dominated by supply side issues, and as a result the software industry is oriented towards developers rather than users. Existing software maintenance processes are simply too slow to meet the needs of many businesses. To achieve the levels of functionality, flexibility and time to market of changes and updates required by users, a radical shift is required in the development of software, with a more demand-centric view leading to software which will be delivered as a service, within the framework of an open marketplace. Although there are some signs that this approach is being adopted by industry, it is in a very limited and restricted form. We summarise research that has resulted in a long term strategic view of software engineering innovation. Based on this foundation, we describe more recent work that has resulted in an innovative demand-led model for the future of software. We describe a service architecture in which components may be bound instantly, just at the time they are needed and then the binding may be disengaged. Such ultra late binding requires that many non-functional attributes of the software are capable of automatic negotiation and resolution. Some of these attributes have been demonstrated and amplified through a prototype implementation based on existing and available technology.

For the past 40 years, the techniques, processes and methods of software development have been dominated by supply-side issues, giving rise to a software industry oriented towards developers rather than users. To achieve the levels of functionality, flexibility and time-to-market required by users, a radical shift is required in the development of software, with a more demand-centric view, leading to software which will be delivered as a service within the framework of an open marketplace. Already, there are some signs that this approach is being adopted by industry, but in a very limited way. We summarise research and a research method which has resulted in a long-term strategic view of software engineering innovation. Based on this foundation, we describe more recent work, which has resulted in an innovative demand-side model for the future of software. We propose a service architecture in which components may be bound instantly, just at the time they are needed, and then the binding may be discarded. A major benefit of this approach is that it leads to highly flexible and agile software that should be able to meet rapidly changing business needs.

Software engineering is no longer the preserve of individuals but is essentially a team-based activity involving a wide variety of stakeholders and thus making the need for communication and co-operation an inherent characteristic. Changes in support technology, economic factors and globalisation of the software process is resulting in the geographical separation of personnel. Where such distribution of personnel occurs, it is clear that the quality of the project will be dependent upon the degree of effective communication and co-operation. This paper presents the results from a study of communication and co-operation practices on a range of distributed software projects from an industrial and educational context. It analyses the advantages and disadvantages of distributed software engineering teams and presents the lessons learned from these studies, including team structure, project management, communication and procedures.

Distributed group working amongst teams of software engineers is increasingly evident in the 'real world'. Tools to support such working are at present limited to general-purpose groupware involving video, audio, chat, shared whiteboards and shared workspaces. Within software engineering education, group tasks have an established role in the curriculum. However in general, groups are local to a particular university or institution and are composed of students who have a significant shared history (in terms of technical background and social interaction) and who are able to meet face-to-face on a regular basis. This paper reports on work undertaken by three UK Universities to provide computer science students with the opportunity to experience group working across universities using low-cost tools to support distributed cooperative working.

Group projects have been widely and successfully adopted within software engineering education. Their particular strength lies in giving students experience of real-world working practices. We report on initial experiences of extending software engineering group working to encompass cooperative working across multiple organisations. In particular, we report on a set of case studies involving the execution of selected software engineering tasks by distributed groups of students from three UK Universities.

One of the factors limiting the wider acceptance of computer science (and hence software engineering) as a 'Science and Engineering' discipline is the lack of established experimental practices that can be used to provide empirical support for our ideas. While such techniques do exist, they are not widely known, and do not usually appear in any curricula. This paper described an introductory course on empirical practices that we taught to our final year undergraduate students, and provides a simple 'case study' illustration of our experiences.

Any attempt to automate software configuration management (SCM) should employ data representations and computational models which are powerful enough to handle abstraction, specialization, combinatorial variance, and genericity. The authors describe a working database and query engine based on first-order logic, with an extended data domain accommodating large items (such as source or object files) as pure values and an extended predicate domain accommodating external utilities (such as compilers) as pure functions. It is shown that several well-understood logic programming techniques (theorem proving, partial evaluation, abstract interpretation, and lemma reuse) provide a sound theoretical and practical basis for mechanized software building. A high-level computational mechanism (theorem proving in first-order logic) has been found which elegantly, if unexpectedly, supports software release engineering. The user interface is essentially declarative: to build a release, the user asks a question to which the desired product is an answer.>