Evidence-based software engineering (EBSE) aims to improve research utilization in practice. It relies on systematic methods to identify, appraise, and synthesize existing research findings to answer questions of interest for practice. However, the lack of practitioners’ involvement in these studies’ design, execution, and reporting indicates a lack of appreciation for the need for knowledge exchange between researchers and practitioners. The resultant systematic literature studies often lack relevance for practice. This paper explores the use of Rapid Reviews (RRs), in fostering knowledge exchange between academia and industry. Through the lens of two case studies, we delve into the practical application and experience of conducting RRs. We analyzed the conduct of two rapid reviews by two different groups of researchers and practitioners. We collected data through interviews, and the documents produced during the review (like review protocols, search results, and presentations). The interviews were analyzed using thematic analysis. We report how the two groups of researchers and practitioners performed the rapid reviews. We observed some benefits, like promoting dialogue and paving the way for future collaborations. We also found that practitioners entrusted the researchers to develop and follow a rigorous approach and were more interested in the applicability of the findings in their context. The problems investigated in these two cases were relevant but not the most immediate ones. Therefore, rapidness was not a priority for the practitioners. The study illustrates that rapid reviews can support researcher-practitioner communication and industry-academia collaboration. Furthermore, the recommendations based on the experiences from the two cases complement the detailed guidelines researchers and practitioners may follow to increase interaction and knowledge exchange.

Eye tracking has been used as part of software engineering and computer science research for a long time, and during this time new techniques for machine learning (ML) have emerged. Some of those techniques are applicable to the analysis of eye-tracking data, and to some extent have been applied. However, there is no structured summary available on which ML techniques are used for analysis in different types of eye-tracking research studies.

Context: Software development is moving towards a place where data about development is gathered in a systematic fashion in order to improve the practice, for example, in tuning of static code analysis. However, this kind of data gathering has so far primarily happened within organizations, which is unfortunate as it tends to favor larger organizations with more resources for maintenance of developer tools. Objective: Over the years, we have seen a lot of benefits from open source and recently there has been a lot of development in open data. We see this as an opportunity for cross-organisation community building and wonder to what extent the views on using and sharing open source software developer tools carry across to open data-driven tuning of software development tools. Method: An exploratory study with 11 participants divided into 3 focus groups discussing using and sharing of static code analyzers and data about these analyzers. Results: While using and sharing open-source code (analyzers in this case) is perceived in a positive light as part of the practice of modern software development, sharing data is met with skepticism and uncertainty. Developers are concerned about threats to the company brand, exposure of intellectual property, legal liabilities, and to what extent data is context-specific to a certain organisation. Conclusions: Sharing data in software development is different from sharing data about software development. We need to better understand how we can provide solutions for sharing of software development data in a fashion that reduces risk and enables openness.

Abstract Empirical software engineering research relies on good communication with industrial partners. Conducting joint research both requires and contributes to bridging the communication gap between industry and academia (IA) in software engineering. This study aims to explore communication between the two parties in such a setting. To better understand what facilitates good IA communication and what project outcomes such communication promotes, we performed a case study, in the context of a long‐term IA joint project, followed by a validating survey among practitioners and researchers with experience of working in similar settings. We identified five facilitators of IA communication and nine project outcomes related to this communication. The facilitators concern the relevance of the research, practitioners' attitude and involvement in research, frequency of communication and longevity of the collaboration. The project outcomes promoted by this communication include, for researchers, changes in teaching and new scientific venues, and for practitioners, increased awareness, changes to practice, and new tools and source code. Besides, both parties gain new knowledge and develop social‐networks through IA communication. Our study presents empirically based insights that can provide advise on how to improve communication in IA research projects and thus the co‐creation of software engineering knowledge that is anchored in both practice and research.

The HAVOSS (Handling Vulnerabilities in OSS) maturity model describes important processes for managing security vulnerabilities in OSS modules in developed products. So far, the model has not been evaluated in any real assessment process. Here we present a study where the model was evaluated by using it in assessments of processes for two product types in one organization. Each assessment was conducted in a focus group meeting where their procedures were analyzed. The evaluation was conducted by posing specific questions about the model during the focus group meetings and by investigating how difficult it was to assess the maturity of practices from the transcribed text. It was found that some practices were easy to assess, while other could be analysed separately for different parts of the products. Further work can be conducted on how assessments can be conducted and how they can be combined with other software security initiatives.

Abstract Background Assessing and communicating software engineering research can be challenging. Design science is recognized as an appropriate research paradigm for applied research, but is rarely explicitly used as a way to present planned or achieved research contributions in software engineering. Applying the design science lens to software engineering research may improve the assessment and communication of research contributions. Aim The aim of this study is 1) to understand whether the design science lens helps summarize and assess software engineering research contributions, and 2) to characterize different types of design science contributions in the software engineering literature. Method In previous research, we developed a visual abstract template, summarizing the core constructs of the design science paradigm. In this study, we use this template in a review of a set of 38 award winning software engineering publications to extract, analyze and characterize their design science contributions. Results We identified five clusters of papers, classifying them according to their different types of design science contributions. Conclusions The design science lens helps emphasize the theoretical contribution of research output—in terms of technological rules—and reflect on the practical relevance, novelty and rigor of the rules proposed by the research.

Software products are rarely developed from scratch and vulnerabilities in such products might reside in parts that are either open source software or provided by another organization. Hence, the total cybersecurity of a product often depends on cooperation, explicit or implicit, between several organizations. We study the attitudes and practices of companies in software ecosystems towards sharing vulnerability information. Furthermore, we compare these practices to contemporary cybersecurity recommendations. This is performed through a questionnaire-based qualitative survey. The questionnaire is divided into two parts: the providers' perspective and the acquirers' perspective. The results show that companies are willing to share information with each other regarding vulnerabilities. Sharing is not considered to be harmful neither to the cybersecurity nor their business, even though a majority of the respondents consider vulnerability information sensitive. However, the companies, despite being open to sharing, are less inclined to proactively sharing vulnerability information. Furthermore, the providers do not perceive that there is a large interest in vulnerability information from their customers. Hence, the companies' overall attitude to sharing vulnerability information is passive but open. In contrast, contemporary cybersecurity guidelines recommend active disclosure and sharing among actors in an ecosystem.

Background: In software engineering, industryacademia is a symbiotic relationship. Researchers need to be aware of the industry to produce relevant research, while practitioners are educated in academia and could take advantage of empirical research. The SERP taxonomy architecture is designed to support communication between practitioners and researchers in software engineering. Objective: The purpose of this study is to analyze to what extent the SERP taxonomy architecture is useful for improving communication between researchers and practitioners in IoT vulnerability management. Method: We developed a SERP taxonomy for IoT vulnerability management, SERP-MENTION, in an incremental way. Along the development, we evaluated the developed taxonomy in a project of industry academia collaboration. Results: In addition to the taxonomy itself we elaborate on the taxonomy development process and the potential of SERP-MENTION to support communication between researchers and practitioners within the area. Conclusion: The SERP architecture can be used in a new field, it is perceived as useful by potential users to better describe and communicate research outputs and practical challenges in software vulnerability management.

As our society becomes more and more dependent on IT systems, failures of these systems can harm more and more people and organizations. Diligently performing risk and hazard analysis helps to minimize the potential harm of IT system failures on the society and increases the probability of their undisturbed operation. Risk and hazard analysis is an important activity for the development and operation of critical software intensive systems, but the increased complexity and size puts additional requirements on the effectiveness of risk and hazard analysis methods. This paper presents a qualitative comparison of two hazard analysis methods, failure mode and effect analysis (FMEA) and system theoretic process analysis (STPA), using case study research methodology. Both methods have been applied on the same forward collision avoidance system to compare the effectiveness of the methods and to investigate what are the main differences between them. Furthermore, this study also evaluates the analysis process of both methods by using a qualitative criteria derived from the technology acceptance model (TAM). The results of the FMEA analysis were compared to the results of the STPA analysis, which were presented in a previous study. Both analyses were conducted on the same forward collision avoidance system. The comparison shows that FMEA and STPA deliver similar analysis results.

Empirical software engineering research aims to generate prescriptive knowledge that can help software engineers improve their work and overcome their challenges, but deriving these insights from real-world problems can be challenging. In this paper, we promote design science as an effective way to produce and communicate prescriptive knowledge. We propose using a visual abstract template to communicate design science contributions and highlight the main problem/solution constructs of this area of research, as well as to present the validity aspects of design knowledge. Our conceptualization of design science is derived from existing literature and we illustrate its use by applying the visual abstract to an example use case. This is work in progress and further evaluation by practitioners and researchers will be forthcoming.

Implementation of open source development practices within commercial settings can bring benefits such as improved source code quality, lower maintenance costs, and increased innovation.However, a widespread in-house implementation of the practices has not been observed.The goal of this research is to understand factors which hinder the implementation.For the purpose, development practices of a large, global software and hardware organization that bases its products on open source software, and has over a decade long experience of contributing to various open source projects were studied.The results were validated through a set of structured interviews and a focus group meeting.It is found that the initial implementation of the process has not been carried out in a planned and systematic way within the company.The results of the follow-up focus group meeting show that while the company's practices acquired a higher degree of alignment over a two-year period, the change was necessitated by a need to have a more efficient development effort across new, globally distributed, development sites.

Context: The increasing dependence on critical IT systems makes them more and more complex, which results in increased complexity and size. Risk analysis is an important activity for the development and operation of critical IT systems, but the increased complexity and size put additional requirements on the effectiveness of risk analysis methods. There complexity means that there is a need to involve different perspectives into risk analysis. Objective: The objective of the research carried out in this study is to investigate the effectiveness of perspective-based risk analysis (PBRA) methods compared to traditional risk analysis (TRA) methods. Method: A controlled experiment was designed and carried out. 43 subjects performed risk analysis of a software-controlled train door system using either TRA or PBRA. Results: The results suggest that PBRA helps to identify more relevant risks than TRA. On the other hand, our experiment failed to provide supporting evidence that PBRA helps to identify fewer non-relevant risks. This study also found that PBRA is more difficult to use than TRA. Conclusions: Some potential benefits of using perspective-based risk analysis are uncovered and experimentally confirmed. In particular, it was discovered that PBRA is more effective than the traditional method and identifies more relevant risks.

Context: At the same time as our dependence on IT systems increases, the number of reports of problems caused by failures of critical IT systems has also increased. This means that there is a need for risk analysis in the development of this kind of systems. Risk analysis of technical systems has a long history in mechanical and electrical engineering. Objective: Even if a number of methods for risk analysis of technical systems exist, the failure behavior of information systems is typically very different from mechanical systems. Therefore, risk analysis of IT systems requires different risk analysis techniques, or at least adaptations of traditional approaches. This means that there is a need to understand what types of methods are available for IT systems and what research that has been conducted on these methods. Method: In this paper we present a systematic mapping study on risk analysis for IT systems. 1086 unique papers were identified in a database search and 57 papers were identified as relevant for this study. These papers were classified based on 5 different criteria. Results: This classification, for example, shows that most of the discussed risk analysis methods are qualitative and not quantitative and that most of the risk analysis methods that are presented in these papers are developed for IT systems in general and not for specific types of IT system. Conclusions: The results show that many new risk analysis methods have been proposed in the last decade but even more that there is a need for more empirical evaluations of the different risk analysis methods. Many papers were identified that propose new risk analysis methods, but few papers discuss a systematic evaluation of these methods or a comparison of different methods based on empirical data.

Service Level Agreements (SLA) are considered a good practice not only for IT outsourcing but also for IT management within an organisation. In this paper we study the usage of SLAs in municipal IT management. Municipal IT management traditionally involves a large organisation, often with a low IT maturity, but with high requirements on software quality and information security. In this study, a series of interviews was conducted with IT managers from a number of Swedish municipalities which shows a need for more support for municipalities trying to write and improve their SLAs. Based on a number of SLAs collected, this study proposes a specific SLA template for municipal IT management that focuses especially on information security issues. In a final step, the proposed template was evaluated in a focus group meeting with practitioners from Swedish municipalities.

Context: Several text books and papers published between 2000 and 2002 have attempted to introduce experimental design and statistical methods to software engineers undertaking empirical studies. Objective: This paper investigates whether there has been an increase in the quality of human-centric experimental and quasi-experimental journal papers over the time period 1993 to 2010. Method: Seventy experimental and quasi-experimental papers published in four general software engineering journals in the years 1992-2002 and 2006-2010 were each assessed for quality by three empirical software engineering researchers using two quality assessment methods (a questionnaire-based method and a subjective overall assessment). Regression analysis was used to assess the relationship between paper quality and the year of publication, publication date group (before 2003 and after 2005), source journal, average coauthor experience, citation of statistical text books and papers, and paper length. The results were validated both by removing papers for which the quality score appeared unreliable and using an alternative quality measure. Results: Paper quality was significantly associated with year, citing general statistical texts, and paper length (p <; 0.05). Paper length did not reach significance when quality was measured using an overall subjective assessment. Conclusions: The quality of experimental and quasi-experimental software engineering papers appears to have improved gradually since 1993.

During systematic literature reviews it is necessary to assess the quality of empirical papers. Current guidelines suggest that two researchers should independently apply a quality checklist and any disagreements must be resolved. However, there is little empirical evidence concerning the effectiveness of these guidelines. This paper investigates the three techniques that can be used to improve the reliability (i.e. the consensus among reviewers) of quality assessments, specifically, the number of reviewers, the use of a set of evaluation criteria and consultation among reviewers. We undertook a series of studies to investigate these factors. Two studies involved four research papers and eight reviewers using a quality checklist with nine questions. The first study was based on individual assessments, the second study on joint assessments with a period of inter-rater discussion. A third more formal randomised block experiment involved 48 reviewers assessing two of the papers used previously in teams of one, two and three persons to assess the impact of discussion among teams of different size using the evaluations of the “teams” of one person as a control. For the first two studies, the inter-rater reliability was poor for individual assessments, but better for joint evaluations. However, the results of the third study contradicted the results of Study 2. Inter-rater reliability was poor for all groups but worse for teams of two or three than for individuals. When performing quality assessments for systematic literature reviews, we recommend using three independent reviewers and adopting the median assessment. A quality checklist seems useful but it is difficult to ensure that the checklist is both appropriate and understood by reviewers. Furthermore, future experiments should ensure participants are given more time to understand the quality checklist and to evaluate the research papers.

SUMMARY Maturity models are widely used in process improvement. The users of a maturity model should be confident that the weak points of the assessed processes can be found, and that the most valuable changes are introduced. Therefore, the evaluation of maturity models is an important activity. In this paper, a mapping study of the literature on the evaluation of maturity models is presented. Two databases are searched resulting in a set of relevant papers. The identified papers can be classified according to six categories, namely the maturity model under evaluation, type of evaluation, relation of the evaluators/authors to the maturity model, level of objectivity, main purpose of the paper, and size of study. Further, a framework of different evaluations of maturity models is developed, and the relevant papers are mapped to the framework. Finally, the relevant research on the evaluation of the maturity models in the Capability Maturity Model family is discussed in more detail. The result of this mapping study is a clear overview of how the evaluation of maturity models has been done, and some discussions are provided for further research on the evaluation of commonly used or newly developed maturity models. Copyright © 2011 John Wiley & Sons, Ltd.

Synthesis of case studies is different from synthesis of purely quantitative studies, for example, in that sampling and analysis in primary studies have been carried out differently, and that primary results are of a different nature. The objective of this research is to identify what challenges should be considered when choosing and using a method for synthesis of case studies. We collected experience from independent synthesis of two published case studies (on trust in outsourcing) by two teams, one team applied cross-case analysis, the other team applied thematic synthesis. The two teams reached both supporting and complimentary conclusions. Identified challenges relate to the goals and research questions of the cases to be synthesized, the number of case studies, temporal and spatial variations, and access to raw data.

Context: The authors wanted to assess whether the quality of published human-centric software engineering experiments was improving. This required a reliable means of assessing the quality of such experiments. Aims: The aims of the study were to confirm the usability of a quality evaluation checklist, determine how many reviewers were needed per paper that reports an experiment, and specify an appropriate process for evaluating quality. Method: With eight reviewers and four papers describing human-centric software engineering experiments, we used a quality checklist with nine questions. We conducted the study in two parts: the first was based on individual assessments and the second on collaborative evaluations. Results: The inter-rater reliability was poor for individual assessments but much better for joint evaluations. Four reviewers working in two pairs with discussion were more reliable than eight reviewers with no discussion. The sum of the nine criteria was more reliable than individual questions or a simple overall assessment. Conclusions: If quality evaluation is critical, more than two reviewers are required and a round of discussion is necessary. We advise using quality criteria and basing the final assessment on the sum of the aggregated criteria. The restricted number of papers used and the relatively extensive expertise of the reviewers limit our results. In addition, the results of the second part of the study could have been affected by removing a time restriction on the review as well as the consultation process.

There has been much recent interest in how to help students in higher education develop their generic skills, especially since this is a focus of the Bologna process that aims to standardize European higher education. However, even though the Master thesis is the final and often crucial part of a graduate degree and requires many generic skills very little research has directly focused on them. In particular, there is a lack of such knowledge for engineering education programs. In this paper we present results from a survey where we asked 23 students from three different Swedish Universities about which generic skills are needed and developed in a Master thesis project in software engineering. One outcome of our analysis is that there is a lack of understanding on how to define, and thus examine, generic skills in software engineering thesis projects.

Case study is a suitable research methodology for software engineering research since it studies contemporary phenomena in its natural context. However, the understanding of what constitutes a case study varies, and hence the quality of the resulting studies. This paper aims at providing an introduction to case study methodology and guidelines for researchers conducting case studies and readers studying reports of such studies. The content is based on the authors’ own experience from conducting and reading case studies. The terminology and guidelines are compiled from different methodology handbooks in other research domains, in particular social science and information systems, and adapted to the needs in software engineering. We present recommended practices for software engineering case studies as well as empirically derived and evaluated checklists for researchers and readers of case study research.

The medical device industry is a constantly growing domain which makes use of more and more software products. Given the importance to this industry of dependable software components, rigorous software engineering techniques would seem to have an important role to play. However, in a recent survey of the industry we found a lower than expected rate of adoption of certain sound software engineering practices. To ensure and improve the quality of developed software products, whether they are standalone applications or embedded in complex systems, both the medical device industry as well as software engineers have to take action. Our survey is a first step in this direction and may help in identifying appropriate programs and future research topics.

Case study is an important research methodology for software engineering. We have identified the need for checklists supporting researchers and reviewers in conducting and reviewing case studies. We derived checklists for researchers and reviewers respectively, using systematic qualitative procedures. Based on nine sources on case studies, checklists are derived and validated, and hereby presented for further use and improvement.

To measure the reliability of a website from a user's point of view, the uncertainly on the usage of the website has to be taken into account. In this paper we investigate the influence of this uncertainly on the reliability estimate for a web server. For this purpose a session based Markov model is used to model the usage extracted from the server's logfiles. From these logfiles a complete user profile can be extracted together with an estimate of the uncertainty on this user profile. This paper investigates the applicability of this kind of Markov model on web server reliability and discusses the difficulties with data extraction from the logfiles. Advantages and disadvantages of this approach are discussed and the approach is applied to data from a university department's web server to demonstrate its applicability.

There is a need to identify factors that affect the result of empirical studies in software engineering research. It is still the case that seemingly identical replications of controlled experiments result in different conclusions due to the fact that all factors describing the experiment context are not clearly defined and hence controlled. In this article, a scheme for describing the participants of controlled experiments is proposed and evaluated. It consists of two main factors, the incentives for participants in the experiment and the experience of the participants. The scheme has been evaluated by classifying a set of previously conducted experiments from literature. It can be concluded that the scheme was easy to use and understand. It is also found that experiments that are classified in the same way to a large extent point at the same results, which indicates that the scheme addresses relevant factors.

In market-driven development of product software, a crucial decision for each candidate requirement is whether or not to select it for implementation in the next release. We present an analytical model of the requirements selection process, which is used for reasoning about decision quality. A network of queues, with two classes of jobs, models the selection of requirements of different quality. The feasibility of model parameter estimation is validated in a survey involving product managers and system engineers. The results show that some of the respondents have made internally consistent parameter estimations, indicating that the model is relevant and its parameters understandable. It is also shown that a majority of the consistent respondents estimate that most of their implemented product requirements are incorrectly selected. The main objective of the model is to provide tools for evaluation of improvement proposals by estimating the impact of process change.

Abstract In order to understand the state of test process practices in the software industry, we have conducted a qualitative survey, covering software development departments at 11 companies in Sweden of different sizes and application domains. The companies develop products in an evolutionary manner, which means either new versions are released regularly, or new product variants under new names are released. The survey was conducted through workshop and interview sessions, loosely guided by a questionnaire scheme. The main conclusions of the survey are that the documented development process is emphasized by larger organizations as a key asset, while smaller organizations tend to lean more on experienced people. Further, product volution is performed primarily as new product variants for embedded systems, and as new versions for packaged software. The development is structured using incremental development or a daily build approach; increments are used among more process‐focused organizations, and daily build is more frequently utilized in less process‐focused organizations. Test automation is performed using scripts for products with focus on functionality, and recorded data for products with focus on non‐functional properties. Test automation is an issue which most organizations want to improve; handling the legacy parts of the product and related documentation presents a common problem in improvement efforts for product evolution. Copyright © 2003 John Wiley & Sons, Ltd.

Empirical methods are important in software engineering. It is important to be able to evaluate new techniques and methods in a structured way before they are introduced in the software process. This paper presents how empirical methods may be taught by letting students take part in the execution and analysis of empirical investigations in projects. The projects that the students carry out include a role-play where teachers play the roles of managers for the students and responsible for ordering the work of the students. It is found that the project is well received and that it is probable that the students reach the related learning objectives of the course.

In order to increase reuse, a number of product versions may be developed based on the same software platform. The platform must, however, be managed and updated according to new requirements if it should be reusable in a series of releases. This means that the platform is constantly changed during its lifecycle, and changes can result in degradation of the platform. In this paper, a measurement approach is proposed as a means of tracking the degradation of a software platform and consequently in the product line. The tracking approach is evaluated in a case study where it is applied to a series of different releases of a product. The result of the case study indicates that the presented approach is feasible.

Estimation of the defect content is important to enable quality control throughout the software development process. Capture-recapture methods and curve fitting methods have been suggested as tools to estimate the defect content after a review. The methods are highly reliant on the quality of the data. If the number of reviewers is fairly small, it becomes difficult or even impossible to get reliable estimates. This paper presents a comprehensive study of estimates based on two reviewers, using real data from reviews. Three experience-based defect content estimation methods are evaluated vs. methods that use data only from the current review. Some models are possible to distinguish from each other in terms of statistical significance. In order to gain an even better understanding, the best models are compared subjectively. It is concluded that the experience-based methods provide some good opportunities to estimate the defect content after a review.

The required effort of a task can be estimated subjectively in interviews with experts in an organization in different ways. Interview techniques dealing with which type of questions to ask are evaluated and techniques for combining estimates from individuals into one estimate are compared in an experiment. The result shows that the interview technique is not as important as the combination technique. The estimate which is best with respect to mean value and standard deviation of the effort is based on an equal weighting of all individual estimates. The experiment is performed within the Personal Software Process (PSP).