Context: Recent papers have proposed the use ofgrey literature(GL) and multivocal reviews. These papers have raised issues about the practices used for systematic reviews (SRs) in software engineering (SE) and suggested that there should be changes to the current SR guidelines.Objective: To investigate whether current SR guidelines need to be changed to support GL and multivocal reviews.Method: We discuss the definitions of GL and the importance of GL and of industry-based field studies in SE SRs. We identify properties of SRs that constrain the material used in SRs: a) the nature of primary studies; b) the requirements of SRs to be auditable, traceable, and reproducible; and explain why these requirements restrict the use of blogs in SRs.Results: SR guidelines have always considered GL as a possible source of primary studies and have never supported exclusion of field studies that incorporate the practitioners’ viewpoint. However, the concept of GL, which was meant to refer to documents that were not formally published, is now being extended to information from sources such as blogs/tweets/Q&A posts. Thus, it might seem that SRs do not make full use of GL because they do not include such information. However, the unit of analysis for an SR is the primary study. Thus, it is not thesourcebut thetypeof information that is important. Any report describing a rigorous empirical evaluation is a candidate primary study. Whether it is actually included in an SR depends on the SR eligibility criteria. However, any study that cannot be guaranteed to be publicly available in the long term should not be used as a primary study in an SR. This does not prevent such information from being aggregated in surveys of social media and used in the context of evidence-based software engineering (EBSE).Conclusions: Current guidelines for SRs do not require extensions, but their scope needs to be better defined. SE researchers require guidelines for analysing social media posts (e.g., blogs, tweets, vlogs), but these should be based on qualitative primary (not secondary) study guidelines. SE researchers can use mixed-methods SRs and/or the fourth step of EBSE to incorporate findings from social media surveys with those from SRs and to develop industry-relevant recommendations.

Context: Several tertiary studies have criticized the reporting of software engineering secondary studies.Objective: Our objective is to identify guidelines for reporting software engineering (SE) secondary studies which would address problems observed in the reporting of software engineering systematic reviews (SRs).Method: We review the criticisms of SE secondary studies and identify the major areas of concern. We assess the PRISMA 2020 (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement as a possible solution to the need for SR reporting guidelines, based on its status as the reporting guideline recommended by the Cochrane Collaboration whose SR guidelines were a major input to the guidelines developed for SE. We report its advantages and limitations in the context of SE secondary studies. We also assess reporting guidelines for mapping studies and qualitative reviews, and compare their structure and content with that of PRISMA 2020.Results: Previous tertiary studies confirm that reports of secondary studies are of variable quality. However,ad hocrecommendations that amend reporting standards may result in unnecessary duplication of text. We confirm that the PRISMA 2020 statement addresses SE reporting problems, but is mainly oriented to quantitative reviews, mixed-methods reviews and meta-analyses. However, we show that the PRISMA 2020 item definitions can be extended to cover the information needed to report mapping studies and qualitative reviews.Conclusions: In this paper and its Supplementary Material, we present and illustrate an integrated set of guidelines called SEGRESS (Software Engineering Guidelines for REporting Secondary Studies), suitable for quantitative systematic reviews (building upon PRISMA 2020), mapping studies (PRISMA-ScR), and qualitative reviews (ENTREQ and RAMESES), that addresses reporting problems found in current SE SRs.

Other disciplines commonly employ secondary studies to address the needs of practitioners and policy-makers. Since being adopted by software engineering in 2004, many have been undertaken by researchers. To assess how the role of secondary studies in software engineering has evolved. We examined a sample of 131 secondary studies published in a set of five major software engineering journals for the years 2010, 2015 and 2020. These were categorised by their type (e.g. mapping study), their research focus (quantitative/qualitative and practice/methodological), as well as the experience of the first authors. Secondary studies are now a well-established research tool. They are predominantly qualitative and there is extensive use of mapping studies to profile research in particular areas. A significant number are clearly produced as part of postgraduate study, although experienced researchers also conduct many secondary studies. They are sometimes also used as part of a multi-method study. Existing guidelines largely focus upon quantitative systematic reviews. Based on our findings, we suggest that more guidance is needed on how to conduct, analyse, and report qualitative secondary studies.

Clinical data, such as evaluations, treatments, vital sign and lab test results, are usually observed and recorded in hospital systems. Making use of such data to help physicians to evaluate the mortality risk of in-hospital patients provides an invaluable source of information that can ultimately help with improving healthcare services. In particular, quick and accurate predictions of mortality can be valuable for physicians who are making decisions about interventions. In this work we introduce the use of a predictive Deep Learning model to help evaluate the mortality risk for in-hospital patients. Stacked Denoising Autoencoder (SDA) has been trained using a unique time-stamped dataset (King Abdullah International Research Center – KAIMRC) which is naturally imbalanced. The results are compared to those from common deep learning approaches, using different methods for data balancing. The proposed model demonstrated here aims to overcome the problem of imbalanced data, and outperforms common deep learning approaches with an accuracy of 77.13% for the Recall macro

There have been many changes in statistical theory in the past 30 years, including increased evidence that non-robust methods may fail to detect important results. The statistical advice available to software engineering researchers needs to be updated to address these issues. This paper aims both to explain the new results in the area of robust analysis methods and to provide a large-scale worked example of the new methods. We summarise the results of analyses of the Type 1 error efficiency and power of standard parametric and non-parametric statistical tests when applied to non-normal data sets. We identify parametric and non-parametric methods that are robust to non-normality. We present an analysis of a large-scale software engineering experiment to illustrate their use. We illustrate the use of kernel density plots, and parametric and non-parametric methods using four different software engineering data sets. We explain why the methods are necessary and the rationale for selecting a specific analysis. We suggest using kernel density plots rather than box plots to visualise data distributions. For parametric analysis, we recommend trimmed means, which can support reliable tests of the differences between the central location of two or more samples. When the distribution of the data differs among groups, or we have ordinal scale data, we recommend non-parametric methods such as Cliff’s δ or a robust rank-based ANOVA-like method.

The original ACM/IEEE Curriculum Guidelines for undergraduate degree programs in software engineering were issued in 2004. In 2010 the two professional bodies set up a task force to review these and see what changes might be needed, and following the report of this group, then set up another task force in 2012 to revise the guidelines. We describe the processes that the two task forces followed, and outline the main areas of change in the revised curriculum model that resulted from these. This is then followed by a review of the elements making up the model, to explain the roles that each of them can perform for those who may be developing a software engineering degree program. Finally, there are a few observations about both the processes and the outcome (the Curriculum Model).

We shall conduct a half-day workshop on needed revisions to Software Engineering 2004: Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering (SE 2004). A brief overview of the current guidelines and their revision status will be presented. Workshop attendees will share their experience using the current guidelines and suggest needed changes. We will provide a summary report from the workshop to other CSEE&T attendees at a Birds Of a Feather meeting later during the conference.

Context: For the outcomes of systematic literature reviews to be of use for practitioners, we need to develop models for addressing the needs of Knowledge Translation (KT). Aim: To identify some of the key issues that need to be addressed by a KT process for software engineering (SE) and possible routes for achieving these. Method: We have examined some of the models used in other disciplines, and suggested a possible interpretation for software engineering. Results: We propose a model for achieving KT. Conclusions: Research with industry and commerce is needed to explore how this can be realised.

Context: Due to the lack of suitably skilled participants, software engineering experiments often lack the statistical power needed to detect the levels of effect that may be encountered. Aim: To investigate whether this can be remedied by running an experiment across multiple sites, organised as a single study rather than as a set of replications. Method: We performed a `trial' of the idea using a topic (structured abstracts) that some of us had studied previously and which required no participant training. We used five sites, each with 16 participants. Results: We were able to demonstrate the benefits of increased statistical power (and of structured abstracts). We report on our experiences with designing and conducting the study and identify some key lessons about how future studies of this form might be organised. Conclusions: The distributed model offers a flexible, robust form that is capable of delivering better statistical power than would be achieved by running a set of parallel replicated studies.

This panel will engage participants in a discussion of recent changes in software engineering practice that should be reflected in curriculum guidelines for undergraduate software engineering programs. Current progress in revising the guidelines will be presented, including suggestions to update coverage of agile methods, security and service-oriented computing.

Software Engineering 2004: Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering (SE 2004) [1] is one volume in a set of computing curricula adopted and supported by the ACM and the IEEE Computer Society. In order to keep the software engineering guidelines up to date the two professional societies began a review and revision project in early 2011. This special session will present the results of the review, present a first draft of the revision, and provide time for discussion and input from the computing education community.

Context: Several text books and papers published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers over the time period 1993 to 2010. Method: Seventy experimental and quasi-experimental papers published in four general software engineering journals in the years 1992-2002 and 2006-2010 were each assessed for quality by three empirical software engineering researchers using two quality assessment methods (a questionnaire-based method and a subjective overall assessment). Regression analysis was used to assess the relationship between paper quality and the year of publication, publication date group (before 2003 and after 2005), source journal, average coauthor experience, citation of statistical text books and papers, and paper length. The results were validated both by removing papers for which the quality score appeared unreliable and using an alternative quality measure. Results: Paper quality was significantly associated with year, citing general statistical texts, and paper length (p <; 0.05). Paper length did not reach significance when quality was measured using an overall subjective assessment. Conclusions: The quality of experimental and quasi-experimental software engineering papers appears to have improved gradually since 1993.

Context: Service Oriented Architecture (SOA) has formed the basis for many workshops, conferences and books. However these do not always seem to have used a consistent set of concepts. Objectives: To identify the key characteristics of an SOA as identified from the literature. Method: We report on our experience of undertaking a mapping study (a form of systematic literature review) to identify the different definitions of SOA being used in the literature. Results: We found 921 candidate studies, of which only 98 were explicit about the definitions of SOA that were used. Conclusion: We have demonstrated that a mapping study can be usefully employed to identify inconsistencies and help to create a model of SOA.

Context: We have been undertaking a series of case studies to investigate the value of mapping (scoping) studies in software engineering. Our previous studies have assessed these using the subjective opinions of researchers. Objective: In order to provide a more objective assessment of value, for this study, we used the results of a systematic mapping study to investigate how well mapping studies identify clusters of related studies and to what extent such clusters are complete. Method: In this participant-observer case study, we undertook a mapping study of unit testing and regression testing empirical studies, which we compared with a previous expert literature review and with six other mapping studies and systematic literature reviews (SLRs) that addressed overlapping topics. Results: Our mapping study found more clusters than the expert literature review although it benefited from the set of studies identified by the expert review when refining our search process. The set of studies found by our searches were less complete than those found by SLRs addressing more specific topics, although we found some studies missed by those SLRs. Conclusions: Researchers undertaking systematic reviews and mapping studies should make use of related systematic reviews and mapping studies to identify known studies in order to refine search strings and validate search results. For completeness and traceability, mapping studies should keep a record of all multiple reports of a single study. Meta-analyses and other systematic literature reviews undertaking detailed aggregation should report on candidate primary studies that were rejected in the final screening process, as well as candidate studies that were included. This helps ensure the repeatability of aggregation results.

Context: A previous study has shown that software developers who are experienced with using design patterns hold some conflicting opinions about three of the more popular design patterns: Facade, Singleton and Visitor. Aim: To identify the characteristics of these three patterns that have caused them to generate such differing views.

Context: In teaching about software engineering we currently make little use of any empirical knowledge. Aim: To examine the outcomes available from the use of Evidence-Based Software Engineering (EBSE) practices, so as to identify where these can provide support for, and inform, teaching activities. Method: We have examined all known secondary studies published up to the end of 2009, together with those published in major journals to mid-2011, and identified where these provide practical results that are relevant to student needs. Results: Starting with 145 candidate systematic literature reviews (SLRs), we were able to identify and classify potentially useful teaching material from 43 of them. Conclusions: EBSE can potentially lend authority to our teaching, although the coverage of key topics is uneven. Additionally, mapping studies can provide support for research-led teaching.

During systematic literature reviews it is necessary to assess the quality of empirical papers. Current guidelines suggest that two researchers should independently apply a quality checklist and any disagreements must be resolved. However, there is little empirical evidence concerning the effectiveness of these guidelines. This paper investigates the three techniques that can be used to improve the reliability (i.e. the consensus among reviewers) of quality assessments, specifically, the number of reviewers, the use of a set of evaluation criteria and consultation among reviewers. We undertook a series of studies to investigate these factors. Two studies involved four research papers and eight reviewers using a quality checklist with nine questions. The first study was based on individual assessments, the second study on joint assessments with a period of inter-rater discussion. A third more formal randomised block experiment involved 48 reviewers assessing two of the papers used previously in teams of one, two and three persons to assess the impact of discussion among teams of different size using the evaluations of the “teams” of one person as a control. For the first two studies, the inter-rater reliability was poor for individual assessments, but better for joint evaluations. However, the results of the third study contradicted the results of Study 2. Inter-rater reliability was poor for all groups but worse for teams of two or three than for individuals. When performing quality assessments for systematic literature reviews, we recommend using three independent reviewers and adopting the median assessment. A quality checklist seems useful but it is difficult to ensure that the checklist is both appropriate and understood by reviewers. Furthermore, future experiments should ensure participants are given more time to understand the quality checklist and to evaluate the research papers.

Context. Although research in software engineering largely seeks to improve the practices and products of software development, many practices are based upon codification of expert knowledge, often with little or no underpinning from objective empirical evidence. Software design patterns seek to codify expert knowledge to share experience about successful design structures. Objectives. To investigate how extensively the use of software design patterns has been subjected to empirical study and what evidence is available about how and when their use can provide an effective mechanism for knowledge transfer about design. Method. We conducted a systematic literature review in the form of a mapping study, searching the literature up to the end of 2009 to identify relevant primary studies about the use of the 23 patterns catalogued in the widely referenced book by the “Gang of Four.” These studies were then categorized according to the forms of study employed, the patterns that were studied, as well as the context within which the study took place. Results. Our searches identified 611 candidate papers. Applying our inclusion/exclusion criteria resulted in a final set of 10 papers that described 11 instances of “formal” experimental studies of object-oriented design patterns. We augmented our analysis by including seven “experience” reports that described application of patterns using less rigorous observational forms. We report and review the profiles of the empirical evidence for those patterns for which multiple studies exist. Conclusions. We could not identify firm support for any of the claims made for patterns in general, although there was some support for the usefulness of patterns in providing a framework for maintenance, and some qualitative indication that they do not help novices learn about design. For future studies we recommend that researchers use case studies that focus upon some key patterns, and seek to identify the impact that their use can have upon maintenance.

Background: One of the anticipated benefits of systematic literature reviews (SLRs) is that they can be conducted in an auditable way to produce repeatable results. Aim: This study aims to identify under what conditions SLRs are likely to be stable, with respect to the primary studies selected, when used in software engineering. The conditions we investigate in this report are when novice researchers undertake searches with a common goal. Method: We undertook a participant-observer multi-case study to investigate the repeatability of systematic literature reviews. The 'cases' in this study were the early stages, involving identification of relevant literature, of two SLRs of unit testing methods. The SLRs were performed independently by two novice researchers. The SLRs were restricted to the ACM and IEEE digital libraries for the years 1986-2005 so their results could be compared with a published expert literature review of unit testing papers. Results: The two SLRs selected very different papers with only six papers out of 32 in common, and both differed substantially from a published secondary study of unit testing papers finding only three of 21 papers. Of the 29 additional papers found by the novice researchers, only 10 were considered relevant. The 10 additional relevant papers would have had an impact on the results of the published study by adding three new categories to the framework and adding papers to three, otherwise empty, cells. Conclusions: In the case of novice researchers, having broadly the same research question will not necessarily guarantee repeatability with respect to primary studies. Systematic reviews must be careful to report their search process fully or they will not be repeatable. Missing papers can have a significant impact on the stability of the results of a secondary study.

Abstract The Unified Modeling Language (UML) was created on the basis of expert opinion and has now become accepted as the ‘standard’ object‐oriented modelling notation. Our objectives were to determine how widely the notations of the UML, and their usefulness, have been studied empirically, and to identify which aspects of it have been studied in most detail. We undertook a mapping study of the literature to identify relevant empirical studies and to classify them in terms of the aspects of the UML that they studied. We then conducted a systematic literature review, covering empirical studies published up to the end of 2008, based on the main categories identified. We identified 49 relevant publications, and report the aggregated results for those categories for which we had enough papers—metrics,comprehension,model quality,methods and toolsandadoption. Despite indications that a number of problems exist with UML models, researchers tend to use the UML as a ‘given’ and seem reluctant to ask questions that might help to make it more effective. Copyright © 2010 John Wiley & Sons, Ltd.

Context: The authors wanted to assess whether the quality of published human-centric software engineering experiments was improving. This required a reliable means of assessing the quality of such experiments. Aims: The aims of the study were to confirm the usability of a quality evaluation checklist, determine how many reviewers were needed per paper that reports an experiment, and specify an appropriate process for evaluating quality. Method: With eight reviewers and four papers describing human-centric software engineering experiments, we used a quality checklist with nine questions. We conducted the study in two parts: the first was based on individual assessments and the second on collaborative evaluations. Results: The inter-rater reliability was poor for individual assessments but much better for joint evaluations. Four reviewers working in two pairs with discussion were more reliable than eight reviewers with no discussion. The sum of the nine criteria was more reliable than individual questions or a simple overall assessment. Conclusions: If quality evaluation is critical, more than two reviewers are required and a round of discussion is necessary. We advise using quality criteria and basing the final assessment on the sum of the aggregated criteria. The restricted number of papers used and the relatively extensive expertise of the reviewers limit our results. In addition, the results of the second part of the study could have been affected by removing a time restriction on the review as well as the consultation process.

We identify three challenges related to the provenance of the material we use in teaching software engineering. We suggest that these challenges can be addressed by using evidence-based software engineering (EBSE) and its primary tool of systematic literature reviews (SLRs). This paper aims to assess the educational and scientific value of undergraduate and postgraduate students undertaking a specific form of SLR called a mapping study. Using a case study methodology, we asked three postgraduate students and three undergraduates and their supervisor to complete a questionnaire concerning the educational value of mapping studies and any problems they experienced. Students found undertaking a mapping study to be a valuable experience providing both reusable research skills and a good overview of a research topic. Postgraduates found it useful as a starting point for their studies. Undergraduates reported problems undertaking the study in the required timescales. Searching and classifying the literature was difficult.

This study aims to compare the use of targeted manual searches with broad automated searches, and to assess the importance of grey literature and breadth of search on the outcomes of SLRs. We used a participant-observer multi-case embedded case study. Our two cases were a tertiary study of systematic literature reviews published between January 2004 and June 2007 based on a manual search of selected journals and conferences and a replication of that study based on a broad automated search. Broad searches find more papers than restricted searches, but the papers may be of poor quality. Researchers undertaking SLRs may be justified in using targeted manual searches if they intend to omit low quality papers; if publication bias is not an issue; or if they are assessing research trends in research methodologies.

Context: Although the Unified Modeling Language (UML) is now widely used, there is little empirical knowledge about the effectiveness of its different elements.

There is little empirical knowledge of the effectiveness of the object-oriented paradigm. To conduct a systematic review of the literature describing empirical studies of this paradigm. We undertook a Mapping Study of the literature. 138 papers have been identified and classified by topic, form of study involved, and source. The majority of empirical studies of OO (object oriented software) concentrate on metrics, relatively few consider effectiveness.

A consequence of the growing number of empirical studies in software engineering is the need to adopt systematic approaches to assessing and aggregating research outcomes in order to provide a balanced and objective summary of research evidence for a particular topic. The paper reports experiences with applying one such approach, the practice of systematic literature review, to the published studies relevant to topics within the software engineering domain. The systematic literature review process is summarised, a number of reviews being undertaken by the authors and others are described and some lessons about the applicability of this practice to software engineering are extracted. The basic systematic literature review process seems appropriate to software engineering and the preparation and validation of a review protocol in advance of a review activity is especially valuable. The paper highlights areas where some adaptation of the process to accommodate the domain-specific characteristics of software engineering is needed as well as areas where improvements to current software engineering infrastructure and practices would enhance its applicability. In particular, infrastructure support provided by software engineering indexing databases is inadequate. Also, the quality of abstracts is poor; it is usually not possible to judge the relevance of a study from a review of the abstract alone.

Abstract The initial use of a service‐oriented system, which collates information from three databases, highlighted a performance problem with query response time for queries that selected more than ten attributes from a single data source. Our aim was to find the cause of the performance problem within the service‐oriented system. The challenges that were encountered when attempting to locate such a problem are described. Our approach to finding the performance problem, while addressing these challenges, was based on systematically devising and applying measurements to the system. We developed strategies that addressed the challenges and the inherent difficulty of measuring performance in a service‐oriented system. The cause of the performance problem was a buffer used by a JDBC‐ODBC driver to connect Java code to a Microsoft Access database. Three strategies successfully addressed the challenges of measuring performance in the service‐oriented system. In particular, routing queries to a local data source eliminated networking problems. Creating a model of the system allowed performance measurements to continue concurrently with demonstrations of the system, and with its further development. Finally, porting the system identified the part that had a detrimental effect on performance. Copyright © 2006 John Wiley & Sons, Ltd.

Summary form only given. In software engineering, as in computing science, the topics that we teach to our students can be considered as falling into two broad categories: the deterministic, and the non-deterministic. Deterministic topics are those where a specific scenario or operation leads to outcomes that can be assessed in terms of true/false values, and so this classification encompasses large elements of computer architecture, databases, metrics and testing. However, much of the software engineering body of knowledge is really concerned with much more non-deterministic processes such as requirements elicitation, design, construction, maintenance etc. These are activities in which humans play a central role, making value judgements that result in outcomes that are more appropriately assessed by using some form of better/worse ranking than through a true/false categorisation. How much we recognise the existence of this distinction in our teaching is a moot point. Many of our students, educated in the classical science paradigm, will be familiar with the type of reasoning that leads to the outcomes for the deterministic elements. In my presentation, I examine some of the reasons why this experience may not be adequate when they encounter the non-deterministic elements of our subject, and hence why we may need to inculcate some degree of understanding of the evidence-based paradigm in order to support both our teaching and also their learning. I will discuss the nature of this paradigm, present some experiences of how it may be adapted for use in Software Engineering, and review some of the questions that it raises.

Context: Making best use of the growing number of empirical studies in Software Engineering, for making decisions and formulating research questions, requires the ability to construct an objective summary of available research evidence. Adopting a systematic approach to assessing and aggregating the outcomes from a set of empirical studies is also particularly important in Software Engineering, given that such studies may employ very different experimental forms and be undertaken in very different experimental contexts.Objectives: To provide an introduction to the role, form and processes involved in performing Systematic Literature Reviews. After the tutorial, participants should be able to read and use such reviews, and have gained the knowledge needed to conduct systematic reviews of their own.Method: We will use a blend of information presentation (including some experiences of the problems that can arise in the Software Engineering domain), and also of interactive working, using review material prepared in advance.

This workshop is concerned with defining the procedures that are needed to establish a sound empirical foundation for the practices of Software Engineering. Our goal is to begin building a community that will review, analyse, codify and promulgate software engineering experiences as well as to identify the processes and infrastructure that are needed to support these activities.

We adapted a survey instrument developed by Timothy Lethbridge to assess the extent to which the education delivered by four UK universities matches the requirements of the software industry. We propose a survey methodology that we believe addresses the research question more appropriately than the one used by Lethbridge. In particular, we suggest that restricting the scope of the survey to address the question of whether the curricula for a specific university addressed the needs of its own students, allowed us to identify an appropriate target population. However, our own survey suffered from several problems. In particular the questions used in the survey are not ideal, and the response rate was poor. Although the poor response rate reduces the value of our results, our survey appears to confirm several of Lethbridge's observations with respect to the over-emphasis of mathematical topics and the under-emphasis on business topics. We also have a close agreement with respect to the relative importance of different software engineering topics. However the set of topics, that we found were taught far less than their importance would suggest, were quite different from the topics identified by Lethbridge.

No curriculum, however good, will solve our problems on its own. We need to take and use this effectively (and hence need measures of what is effective too). My position is therefore that the CCSE is a useful and quite important stepping-stone, but it is not a watershed in the education processnfortunately, I think there are many more tasks ahead of us. The SEI's programme to develop curriculum modules back in the late 1980s represented a determined attempt to bootstrap software engineering education, using expert judgement with some degree of consultation. The process used to develop the CCSE has moved on a step from this, but was still heavily dependent upon expert judgement-even if this has drawn upon a larger community and longer experience. For the next step, we therefore need to begin to replace the use of expert judgement with evidence that has been carefully assembled and systematically reviewed

We discuss what constitutes a 'software service model' and propose a framework that can accommodate both component-based and service-based systems, while also distinguishing between 'static' and 'dynamic' forms of service delivery. We then draw upon this, and upon our own experiences of developing service-based forms, to consider what characteristics a service architecture might possess

This paper reports on our experiences with using the emerging Web service technologies and tools to create a demonstration information broker system as part of our research into information management in a distributed environment. To provide a realistic context, we chose to study the use of information in the healthcare domain, and this context sets some challenging parameters and constraints for our research and for the demonstration system. In this paper, we both report on the extent to which existing Web service technologies have proved to be mature enough to meet these requirements, and also assess their current limitations.

We address the problem of large-scale data integration, where the data sources are unknown at design time, are from autonomous organisations, and may evolve. Experiments are described involving a demonstrator system in the field of health services data integration within the UK. Current Web services technology has been used extensively and largely successfully in these distributed prototype systems. The work shows that Web services provide a good infrastructure layer, but integration demands a higher level "broker" architectural layer; the paper identifies eight specific requirements for such an architecture that have emerged from the experiments, derived from an analysis of shortcomings which are collectively due to the static nature of the initial prototype. The way in which these are being met in the current version in order to achieve a more dynamic integration is described.

The design process in component-based software engineering needs to implicitly incorporate ways of reusing existing components. Reuse in turn implies the need to acquire knowledge about any components that might be incorporated into a system. Our paper describes an empirical study of how component documentation can be used to help develop that knowledge, using Unix processes as the source of documentation and components, and using both think-aloud and non-intrusive techniques to record subject behaviour. Our conclusions suggest how component documentation might be organised so as to assist with reuse, and identify further questions about its form.

The Software Engineering Institute (SEI) at Carnegie Mellon University started its first contract with a carte blanche opportunity and generous funding to improve the state of software engineering education. Norm Gibbs, the first Director of Education at the SEI guided efforts is thus area. One of his innovations, discussed here, were the "curriculum modules" encapsulating software engineering knowledge.

The urgent need to change software easily to meet evolving business requirements requires a radical shift in the development of software, with a more demand-centric view leading to software which will be delivered as a service, within the framework of an open marketplace. We describe a service architecture and its rationale, in which components may be bound instantly, just at the time they are needed and then the binding may, be disengaged. This allows highly flexible software services to be evolved in "internet time". The paper focuses on early results: some of the aims have been demonstrated and amplified through an experimental implementation based on e-Speak, an existing and available technology. It is concluded that technology such as e-Speak provides a useful infrastructure that rapidly enabled us to demonstrate the basic operation and viability of our approach.

There is an urgent industrial need for new approaches to software evolution that will lead to far faster implementation of software changes. For the past 40 years, the techniques, processes and methods of software development have been dominated by supply side issues, and as a result the software industry is oriented towards developers rather than users. Existing software maintenance processes are simply too slow to meet the needs of many businesses. To achieve the levels of functionality, flexibility and time to market of changes and updates required by users, a radical shift is required in the development of software, with a more demand-centric view leading to software which will be delivered as a service, within the framework of an open marketplace. Although there are some signs that this approach is being adopted by industry, it is in a very limited and restricted form. We summarise research that has resulted in a long term strategic view of software engineering innovation. Based on this foundation, we describe more recent work that has resulted in an innovative demand-led model for the future of software. We describe a service architecture in which components may be bound instantly, just at the time they are needed and then the binding may be disengaged. Such ultra late binding requires that many non-functional attributes of the software are capable of automatic negotiation and resolution. Some of these attributes have been demonstrated and amplified through a prototype implementation based on existing and available technology.

For the past 40 years, the techniques, processes and methods of software development have been dominated by supply-side issues, giving rise to a software industry oriented towards developers rather than users. To achieve the levels of functionality, flexibility and time-to-market required by users, a radical shift is required in the development of software, with a more demand-centric view, leading to software which will be delivered as a service within the framework of an open marketplace. Already, there are some signs that this approach is being adopted by industry, but in a very limited way. We summarise research and a research method which has resulted in a long-term strategic view of software engineering innovation. Based on this foundation, we describe more recent work, which has resulted in an innovative demand-side model for the future of software. We propose a service architecture in which components may be bound instantly, just at the time they are needed, and then the binding may be discarded. A major benefit of this approach is that it leads to highly flexible and agile software that should be able to meet rapidly changing business needs.

Although developing systems with reusable software components offers considerable scope to speed the delivery of a system, their use also requires us to revise our ideas about the design process. We report on a series of laboratory-based studies in which we observed the processes by which simple component-based designs were developed, paying particular attention to when the documentation for particular components was accessed and what information was being sought at each stage in the development of the solution. We observed quite extensive access to documentation, both for the components that were eventually used and also for those that were discarded during the process.

One of the factors limiting the wider acceptance of computer science (and hence software engineering) as a 'Science and Engineering' discipline is the lack of established experimental practices that can be used to provide empirical support for our ideas. While such techniques do exist, they are not widely known, and do not usually appear in any curricula. This paper described an introductory course on empirical practices that we taught to our final year undergraduate students, and provides a simple 'case study' illustration of our experiences.

Abstract This paper examines how the ‘MASCOT machine’ and the Modula‐2 language may be used together as a basis for engineering a range of real‐time systems. It is shown that Modula‐2 provides the features required for a satisfactory MASCOT implementation language without the normal need to add extra features or use preprocessing. The paper also describes how the MASCOT design tools can be interpreted more widely than has previously been considered in order to provide support for a large range of problems.