The research volume is continuously increasing. Manual analysis of large topic scopes and continuously updating literature studies with the newest research results is effort intensive and, therefore, difficult to achieve. To discuss possibilities and next steps for using LLMs (e.g., GPT-4) in the mapping study process. The research can be classified as a solution proposal. The solution was iteratively designed and discussed among the authors based on their experience with LLMs and literature reviews. We propose strategies for the mapping process, outlining the use of agents and prompting strategies for each step. Given the potential of LLMs in literature studies, we should work on a holistic solutions for LLM-supported mapping studies.

The adoption or acceptance of new technologies or ways of working in software development activities is a recurrent topic in the software engineering literature. The topic has, therefore, been empirically investigated extensively. It is, however, unclear which theoretical frames of reference are used in this research to explain acceptance behaviors. In this study, we explore how major theories and models of acceptance behavior have been used in the software engineering literature to empirically investigate acceptance behavior. We conduct a systematic mapping study of empirical studies using acceptance behavior theories in software engineering. We identified 47 primary studies covering 56 theory uses. The theories were categorized into six groups. Technology acceptance models (TAM and its extensions) were used in 29 of the 47 primary studies, innovation theories in 10, and the theories of planned behavior/ reasoned action (TPB/TRA) in six. All other theories were used in at most two of the primary studies. The usage and operationalization of the theories were, in many cases, inconsistent with the underlying theories. Furthermore, we identified 77 constructs used by these studies of which many lack clear definitions. Our results show that software engineering researchers are aware of some of the leading theories and models of acceptance behavior, which indicates an attempt to have more theoretical foundations. However, we identified issues related to theory usage that make it difficult to aggregate and synthesize results across studies. We propose mitigation actions that encourage the consistent use of theories and emphasize the measurement of key constructs.

Rainer and Wohlin showed that case studies are not well understood by reviewers and authors and thus they say that a given research is a case study when it is not. Rainer and Wohlin proposed a smell indicator (inspired by code smells) to identify case studies based on the frequency of occurrences of words, which performed better than human classifiers. With the emergence of ChatGPT, we evaluate ChatGPT to assess its performance in accurately identifying case studies. We also reflect on the results’ implications for mapping studies, specifically data extraction. We used ChatGPT with the model GPT-4 to identify case studies and compared the result with the smell indicator for precision, recall, and accuracy. GPT-4 and the smell indicator perform similarly, with GPT-4 performing slightly better in some instances and the smell indicator (SI) in others. The advantage of GPT-4 is that it is based on the definition of case studies and provides traceability on how it reaches its conclusions. As GPT-4 performed well on the task and provides traceability, we should use and, with that, evaluate it on data extraction tasks, supporting us as authors.

Double-counting in a literature review occurs when the same data, population, or evidence is erroneously counted multiple times during synthesis. Detecting and mitigating the threat of double-counting is particularly challenging in tertiary studies. Although this topic has received much attention in the health sciences, it seems to have been overlooked in software engineering. We describe issues with double-counting in tertiary studies, investigate the prevalence of the issue in software engineering, and propose ways to identify and address the issue. We analyze 47 tertiary studies in software engineering to investigate in which ways they address double-counting and whether double-counting might be a threat to validity in them. In 19 of the 47 tertiary studies, double-counting might bias their results. Of those 19 tertiary studies, only 5 consider double-counting a threat to their validity, and 7 suggest strategies to address the issue. Overall, only 9 of the 47 tertiary studies, acknowledge double-counting as a potential general threat to validity for tertiary studies. Double-counting is an overlooked issue in tertiary studies in software engineering, and existing design and evaluation guidelines do not address it sufficiently. Therefore, we propose recommendations that may help to identify and mitigate double-counting in tertiary studies.

Software engineering research aims to establish software development practice on a scientific basis. However, the evidence of the efficacy of technology is insufficient to ensure its uptake in industry. In the absence of a theoretical frame of reference, we mainly rely on best practices and expert judgment from industry-academia collaboration and software process improvement research to improve the acceptance of the proposed technology. To identify acceptance models and theories and discuss their applicability in the research of acceptance behavior related to software development. We analyzed literature reviews within an interdisciplinary team to identify models and theories relevant to software engineering research. We further discuss acceptance behavior from the human information processing perspective of automatic and affect-driven processes (“fast” system 1 thinking) and rational and rule-governed processes (“slow” system 2 thinking). We identified 30 potentially relevant models and theories. Several of them have been used in researching acceptance behavior in contexts related to software development, but few have been validated in such contexts. They use constructs that capture aspects of (automatic) system 1 and (rational) system 2 oriented processes. However, their operationalizations focus on system 2 oriented processes indicating a rational view of behavior, thus overlooking important psychological processes underpinning behavior. Software engineering research may use acceptance behavior models and theories more extensively to understand and predict practice adoption in the industry. Such theoretical foundations will help improve the impact of software engineering research. However, more consideration should be given to their validation, overlap, construct operationalization, and employed data collection mechanisms when using these models and theories.

Practitioners working in large-scale software development face many challenges in regression testing activities. One of the reasons is the lack of a structured regression testing process. In this regard, checklists can help practitioners keep track of essential regression testing activities and add structure to the regression testing process to a certain extent. This study aims to introduce regression testing checklists so test managers/teams can use them: (1) to assess whether test teams/members are ready to begin regression testing, and (2) to keep track of essential regression testing activities while planning and executing regression tests. We used interviews, workshops, and questionnaires to design, evolve, and evaluate regression testing checklists. In total, 25 practitioners from 12 companies participated in creating the checklist. Twenty-three of them participated in checklists evolution and evaluation. We identified activities practitioners consider significant while planning, performing, and analyzing regression testing. We designed regression testing checklists based on these activities to help practitioners make informed decisions during regression testing. With the help of practitioners, we evolved these checklists into two iterations. Finally, the practitioners provided feedback on the proposed checklists. All respondents think the proposed checklists are useful and customizable for their environments, and 80% think checklists cover aspects essential for regression testing. The proposed regression testing checklists can be useful for test managers to assess their team/team members’ readiness and decide when to start and stop regression testing. The checklists can be used to record the steps required while planning and executing regression testing. Further, these checklists can provide a basis for structuring the regression testing process in varying contexts.

Abstract Regression testing is challenging because of its complexity and the amount of effort and time it requires, especially in large‐scale environments with continuous integration and delivery. Regression test selection and prioritization techniques have been proposed in the literature to address the regression testing challenges, but adoption rates of these techniques in industry are not encouraging. One of the possible reasons could be the disparity in the regression testing goals in industry and literature. This work compares the research perspective to industry practice on regression testing goals, corresponding information needs, and metrics required to evaluate these goals. We have conducted a literature review of 44 research papers and a survey with 56 testing practitioners. The survey comprises 11 interviews and 45 responses to an online questionnaire. We identified that industry and research accentuate different regression testing goals. For instance, the literature emphasizes increasing the fault detection rates of test suites and early identification of critical faults. In contrast, the practitioners' focus is on test suite maintenance, controlled fault slippage, and awareness of changes. Similarly, the literature suggests maintaining information needs from test case execution histories to evaluate regression testing techniques based on various metrics, whereas, at large, the practitioners do not use the metrics suggested in the literature. To bridge the research and practice gap, based on the literature and survey findings, we have created a goal–question–metric (GQM) model that maps the regression testing goals, associated information needs, and metrics from both perspectives. The GQM model can guide researchers in proposing new techniques closer to industry contexts. Practitioners can benefit from information needs and metrics presented in the literature and can use GQM as a tool to follow their regression testing goals.

Abstract Replication studies help solidify and extend knowledge by evaluating previous studies’ findings. Software engineering literature showed that too few replications are conducted focusing on software artifacts without the involvement of humans. This study aims to replicate an artifact-based study on software testing to address the gap related to replications. In this investigation, we focus on (i) providing a step-by-step guide of the replication, reflecting on challenges when replicating artifact-based testing research and (ii) evaluating the replicated study concerning the validity and robustness of the findings. We replicate a test case prioritization technique proposed by Kwon et al. We replicated the original study using six software programs, four from the original study and two additional software programs. We automated the steps of the original study using a Jupyter notebook to support future replications. Various general factors facilitating replications are identified, such as (1) the importance of documentation; (2) the need for assistance from the original authors; (3) issues in the maintenance of open-source repositories (e.g., concerning needed software dependencies, versioning); and (4) availability of scripts. We also noted observations specific to the study and its context, such as insights from using different mutation tools and strategies for mutant generation. We conclude that the study by Kwon et al. is partially replicable for small software programs and could be automated to facilitate software practitioners, given the availability of required information. However, it is hard to implement the technique for large software programs with the current guidelines. Based on lessons learned, we suggest that the authors of original studies need to publish their data and experimental setup to support the external replications.

Behavior-driven development (BDD) is a variant of test-driven development where specifications are described in a structured domain-specific natural language. Although refactoring is a crucial activity of BDD, little research is available on the topic. To support practitioners in refactoring BDD specifications by (1) proposing semi-automated approaches to identify refactoring candidates; (2) defining refactoring techniques for BDD specifications; and (3) evaluating the proposed identification approaches in an industry context. Using Action Research, we have developed an approach for identifying refactoring candidates in BDD specifications based on two measures of similarity and applied the approach in two projects of a large software organization. The accuracy of the measures for identifying refactoring candidates was then evaluated against an approach based on machine learning and a manual approach based on practitioner perception. We proposed two measures of similarity to support the identification of refactoring candidates in a BDD specification base; (1) normalized compression similarity (NCS) and (2) similarity ratio (SR). A semi-automated approach based on NCS and SR was developed and applied to two industrial cases to identify refactoring candidates. Our results show that our approach can identify candidates for refactoring 6o times faster than a manual approach. Our results furthermore showed that our measures accurately identified refactoring candidates compared with a manual identification by software practitioners and outperformed an ML-based text classification approach. We also described four types of refactoring techniques applicable to BDD specifications; merging candidates, restructuring candidates, deleting duplicates, and renaming specification titles. Our results show that NCS and SR can help practitioners in accurately identifying BDD specifications that are suitable candidates for refactoring, which also decreases the time for identifying refactoring candidates.

Large-scale software projects require interaction between many stakeholders. Behavior-driven development (BDD) facilitates collaboration between stakeholders, and an adapted BDD process can help improve cooperation in a large-scale project. The objective of this study is to propose and empirically evaluate a BDD based process adapted for large-scale projects. A technology transfer model was used to propose a BDD based process for large-scale projects. We conducted six workshop sessions to understand the challenges and benefits of BDD. Later, an industrial evaluation was performed for the process with the help of practitioners. From our investigations, understanding of a business aspect of requirements, their improved quality, a guide to system-level use-cases, reuse of artifacts, and help for test organization are found as benefits of BDD. Practitioners identified the following challenges: specification and ownership of behaviors, adoption of new tools, the software projects’ scale, and versioning of behaviors. We proposed a process to address these challenges and evaluated the process with the help of practitioners. The evaluation proved that BDD could be adapted and used to facilitate interaction in large-scale software projects in the software industry. The feedback from the practitioners helped in improving the proposed process.

Abstract Software engineering is decision intensive. Evidence‐based software engineering is suggested for decision‐making concerning the use of methods and technologies when developing software. Software development often includes the reuse of software assets, for example, open‐source components. Which components to use have implications on the quality of the software (e.g., maintainability). Thus, research is needed to support decision‐making for composite software. This paper presents a roadmap for research required to support evidence‐based decision‐making for choosing and integrating assets in composite software systems. The roadmap is developed as an output from a 5‐year project in the area, including researchers from three different organizations. The roadmap is developed in an iterative process and is based on (1) systematic literature reviews of the area; (2) investigations of the state of practice, including a case survey and a survey; and (3) development and evaluation of solutions for asset identification and selection. The research activities resulted in identifying 11 areas in need of research. The areas are grouped into two categories: areas enabling evidence‐based decision‐making and those related to supporting the decision‐making. The roadmap outlines research needs in these 11 areas. The research challenges and research directions presented in this roadmap are key areas for further research to support evidence‐based decision‐making for composite software.

Abstract Model‐based test case generation techniques provide a mechanism to derive tests systematically. This study provides a systematic mapping of test case generation techniques based on UML interaction diagrams. The study compares the test case generation techniques regarding their capabilities and limitations, and it also assesses the reporting quality of the primary studies. We can conclude that the studies presenting test case generation techniques using UML interaction diagrams were not following the guidelines for research methods (eg, case studies or experiments). Solutions were not empirically evaluated in industrial contexts. Our study revealed that better tool support is needed to introduce the UML interaction diagram–based test case generation techniques in the industry.

Abstract Modern software development relies on a combination of development and re-use of technical asset, e.g., software components, libraries, and APIs. In the past, re-use was mostly conducted with internal assets but today external; open source, customer off-the-shelf (COTS), and assets developed through outsourcing are also common. This access to more asset alternatives presents new challenges regarding what assets to optimally chose and how to make this decision. To support decision-makers, decision theory has been used to develop decision models for asset selection. However, very little industrial data has been presented in literature about the usefulness, or even perceived usefulness, of these models. Additionally, only limited information has been presented about what model characteristics determine practitioner preference toward one model over another. The objective of this work is to evaluate what characteristics of decision models for asset selection determine industrial practitioner preference of a model when given the choice of a decision model of high precision or a model with high speed. An industrial questionnaire survey is performed where a total of 33 practitioners, of varying roles, from 18 companies are tasked to compare two decision models for asset selection. Textual analysis and formal and descriptive statistics are then applied on the survey responses to answer the study’s research questions. The study shows that the practitioners had clear preference toward the decision model that emphasized speed over the one that emphasized decision precision. This conclusion was determined to be because one of the models was perceived faster, had lower complexity, was more flexible in use for different decisions, and was more agile on how it could be used in operation, its emphasis on people, its emphasis on “good enough” precision and ability to fail fast if a decision was a failure. Hence, we found seven characteristics that the practitioners considered important for their acceptance of the model. Industrial practitioner preference, which relates to acceptance, of decision models for asset selection is dependent on multiple characteristics that must be considered when developing a model for different types of decisions such as operational day-to-day decisions as well as more critical tactical or strategic decisions. The main contribution of this work are the seven identified characteristics that can serve as industrial requirements for future research on decision models for asset selection.

Context: Electroencephalograms (EEG) have been used in a few cases in the context of software engineering (SE). EEGs allow capturing emotions and cognitive functioning. Such human factors have already shown to be important to understand software engineering tasks. Therefore, it is essential to gain experience in the community to utilize EEG as a research tool. Objective: To report experiences of using EEG in the context of a software engineering education (review of master theses proposals). We provide our reflections and lessons learned of (1) how to plan an EEG study, (2) how to conduct and execute (e.g., tools), (3) how to analyze. Method: We carried out an experiment using an EEG headset to measure the participants' attention rate. The experiment task includes reviewing three master thesis project plans. Results: We describe how we evolved our understanding of experimentation practices to collect and analyze psychological and cognitive data. We also provide a set of lessons learned regarding the application of EEG technology for research. Conclusions: We believe that that EEG could benefit software engineering research to collect cognitive information under certain conditions. The lessons learned reported here should be used as inputs for future experiments in software engineering, where human aspects are of interest.

Abstract DevOps as a new way of thinking for software development and operations has received much attention in the industry, while it has not been thoroughly investigated in academia yet. The objective of this study is to characterize DevOps by exploring its central components in terms of principles, practices and their relations to the principles, challenges of DevOps adoption, and benefits reported in the peer‐reviewed literature. As a key objective, we also aim to realize the relations between DevOps practices and benefits in a systematic manner. A systematic literature review was conducted. Also, we used the concept of benefits dependency network to synthesize the findings, in particular, to specify dependencies between DevOps practices and link the practices to benefits. We found that in many cases, DevOps characteristics, ie, principles, practices, benefits, and challenges, were not sufficiently defined in detail in the peer‐reviewed literature. In addition, only a few empirical studies are available, which can be attributed to the nascency of DevOps research. Also, an initial version of the DevOps benefits dependency network has been derived. The definition of DevOps principles and practices should be emphasized given the novelty of the concept. Further empirical studies are needed to improve the benefits dependency network presented in this study.

The choice of which software component to use influences the success of a software system. Only a few empirical studies investigate how the choice of components is conducted in industrial practice. This is important to understand to tailor research solutions to the needs of the industry. Existing studies focus on the choice for off-the-shelf (OTS) components. It is, however, also important to understand the implications of the choice of alternative component sourcing options (CSOs), such as outsourcing versus the use of OTS. Previous research has shown that the choice has major implications on the development process as well as on the ability to evolve the system. The objective of this study is to explore how decision making took place in industry to choose among CSOs. Overall, 22 industrial cases have been studied through a case survey. The results show that the solutions specifically for CSO decisions are deterministic and based on optimization approaches. The non-deterministic solutions proposed for architectural group decision making appear to suit the CSO decision making in industry better. Interestingly, the final decision was perceived negatively in nine cases and positively in seven cases, while in the remaining cases it was perceived as neither positive nor negative.

This paper introduces a decision canvas for capturing architecture decisions in software and systems engineering. The canvas leverages a dedicated taxonomy, denoted GRADE, meant for establishing the basics of the vocabulary for assessing and choosing architectural assets in the development of software-intensive systems. The canvas serves as a template for practitioners to discuss and document architecture decisions, i.e., capture, understand and communicate decisions among decision-makers and to others. It also serves as a way to reflect on past decision-making activities devoted to both tentative and concluding decisions in the development of software-intensive systems. The canvas has been assessed by means of preliminary internal and external evaluations with four scenarios. The results are promising as the canvas fulfills its intended objectives while satisfying most of the needs of the subjects participating in the evaluation.

During exploratory testing sessions the tester simultaneously learns, designs and executes tests. The activity is iterative and utilizes the skills of the tester and provides flexibility and creativity. Test charters are used as a vehicle to support the testers during the testing. The aim of this study is to support practitioners in the design of test charters through checklists. We aimed to identify factors allowing practitioners to critically reflect on their designs and contents of test charters to support practitioners in making informed decisions of what to include in test charters. The factors and contents have been elicited through interviews. Overall, 30 factors and 35 content elements have been elicited.

In Agile Software Development (ASD) effort estimation plays an important role during release and iteration planning. The state of the art and practice on effort estimation in ASD have been recently identified. However, this knowledge has not yet been organized. The aim of this study is twofold: (1) To organize the knowledge on effort estimation in ASD and (2) to use this organized knowledge to support practice and the future research on effort estimation in ASD. We applied a taxonomy design method to organize the identified knowledge as a taxonomy of effort estimation in ASD. The proposed taxonomy offers a faceted classification scheme to characterize estimation activities of agile projects. Our agile estimation taxonomy consists of four dimensions: estimation context, estimation technique, effort predictors and effort estimate. Each dimension in turn has several facets. We applied the taxonomy to characterize estimation activities of 10 agile projects identified from the literature to assess whether all important estimation-related aspects are reported. The results showed that studies do not report complete information related to estimation. The taxonomy was also used to characterize the estimation activities of four agile teams from three different software companies. The practitioners involved in the investigation found the taxonomy useful in characterizing and documenting the estimation sessions.

Background: Cost avoidance through reuse shows the benefits gained by the software organisations when reusing an artefact. Cost avoidance captures benefits that are not captured by cost savings e.g. spending that would have increased in the absence of the cost avoidance activity. This type of benefit can be combined with quality aspects of the product e.g. costs avoided because of defect prevention. Cost avoidance is a key driver for software reuse. Objectives: The main objectives of this study are: (1) To assess the status of capturing cost avoidance through reuse in the academia; (2) Based on the first objective, propose improvements in capturing of reuse cost avoidance, integrate these into an instrument, and evaluate the instrument in the software industry. Method: The study starts with a systematic literature review (SLR) on capturing of cost avoidance through reuse. Later, a solution is proposed and evaluated in the industry to address the shortcomings identified during the systematic literature review. Results: The results of a systematic literature review describe three previous studies on reuse cost avoidance and show that no solution, to capture reuse cost avoidance, was validated in industry. Afterwards, an instrument and a data collection form are proposed that can be used to capture the cost avoided by reusing any type of reuse artefact. The instrument and data collection form (describing guidelines) were demonstrated to a focus group, as part of static evaluation. Based on the feedback, the instrument was updated and evaluated in industry at 6 development sites, in 3 different countries, covering 24 projects in total. Conclusion: The proposed solution performed well in industrial evaluation. With this solution, practitioners were able to do calculations for reuse costs avoidance and use the results as decision support for identifying potential artefacts to reuse.

Background: Survey is a method of research aiming to gather data from a large population of interest. Despite being extensively used in software engineering, survey-based research faces several challenges, such as selecting a representative population sample and designing the data collection instruments.

Efficient development of high quality software is tightly coupled to the ability of quickly taking complex decisions based on trustworthy facts. In component-based software engineering, the decisions related to selecting the most suitable component among functionally-equivalent ones are of paramount importance. Despite sharing the same functionality, components differ in terms of their extra-functional properties. Therefore, to make informed selections, it is crucial to evaluate extra-functional properties in a systematic way. To date, many properties and evaluation methods that are not necessarily compatible with each other exist. The property model ontology presented in this paper represents the first step towards providing a systematic way to describe extra-functional properties and their evaluation methods, and thus making them comparable. This is beneficial from two perspectives. First, it aids researchers in identifying comparable property models as a guide for empirical evaluations. Second, practitioners are supported in choosing among alternative evaluation methods for the properties of their interest. The use of the ontology is illustrated by instantiating a subset of property models relevant in the automotive domain.

Abstract Context: Global software development (GSD) projects can have distributed teams that work independently in different locations or team members that are dispersed. The various development settings in GSD can influence quality during product evolution. When evaluating quality using defects as a proxy, the development settings have to be taken into consideration. Objective: The aim is to provide a systematic method for supporting investigations of the implication of GSD contexts on defect data as a proxy for quality. Method: A method engineering approach was used to incrementally develop the proposed method. This was done through applying the method in multiple industrial contexts and then using lessons learned to refine and improve the method after application. Results: A measurement instrument and visualization was proposed incorporating an understanding of the release history and understanding of GSD contexts. Conclusion: The method can help with making accurate inferences about development settings because it includes details on collecting and aggregating data at a level that matches the development setting in a GSD context and involves practitioners at various phases of the investigation. Finally, the information that is produced from following the method can help practitioners make informed decisions when planning to develop software in comparable circumstances. Copyright © 2016 John Wiley & Sons, Ltd.

Background: Systematic literature studies are commonly used in software engineering. There are two main ways of conducting the searches for these type of studies; they are snowballing and database searches. In snowballing, the reference list (backward snowballing - BSB) and citations (forward snowballing - FSB) of relevant papers are reviewed to identify new papers whereas in a database search, different databases are searched using predefined search strings to identify new papers. Objective: Snowballing has not been in use as extensively as database search. Hence it is important to evaluate its efficiency and reliability when being used as a search strategy in literature studies. Moreover, it is important to compare it to database searches. Method: In this paper, we applied snowballing in a literature study, and reflected on the outcome. We also compared database search with backward and forward snowballing. Database search and snowballing were conducted independently by different researchers. The searches of our literature study were compared with respect to the efficiency and reliability of the findings. Results: Out of the total number of papers found, snowballing identified 83% of the papers in comparison to 46% of the papers for the database search. Snowballing failed to identify a few relevant papers, which potentially could have been addressed by identifying a more comprehensive start set. Conclusion: The efficiency of snowballing is comparable to database search. It can potentially be more reliable than a database search however, the reliability is highly dependent on the creation of a suitable start set.

Although citation counts are often considered a measure of academic impact, they are criticized for failing to evaluate impact as intended. In this paper we propose that software engineering citations may be classified according to how the citation is used by the author of the citing paper, and that through this classification of citation behaviour it is possible to achieve a more refined understanding of the cited paper's impact. Our objective in this work is to conduct an initial evaluation using the citation behaviour taxonomy proposed by Bornmann and Daniel. We independently classified citations to ten highly-cited papers published at the International Symposium on Empirical Software Engineering and Measurement (ESEM). The degree to which classifications were consistent between researchers was analyzed in order to assess the clarity of Bornmann and Daniel's taxonomy. We found poor to fair agreement between researchers even though the taxonomy was perceived as relatively easy to apply for the majority of citations. We were nevertheless able to identify clear differences in the profile of citation behaviors between the cited papers. We conclude that an improved taxonomy is required if classification is to be reliable, and that a degree of automation would improve reliability as well as reduce the time taken to make a classification.

Abstract Context Software maintenance process in globally distributed settings brings significant management challenges to software organizations. Objectives Investigate the factors specific to managing software maintenance process in globally distributed settings and best practices in software organizations. Method A systematic literature review and interviews with industry practitioners were conducted. For analysis and synthesis, the grounded theory method was used. Results We identified a number of management challenges and mitigation strategies and then classified them under people, process, product, and technology factors. Overall, a structure of challenges and solutions, the conceptual framework, has been developed that may be used to understand and classify global maintenance challenges. Conclusions Distributed software maintenance process has specific management challenges in relation to process, people, product, and technology. Therefore, companies performing maintenance in distributed settings should consider these factors, which are not present in the general global software development literature, although many lessons apply to both. Copyright © 2015 John Wiley & Sons, Ltd.

Test-Driven Development (TDD) is a software development approach where test cases are written before actual development of the code in iterative cycles. Context: TDD has gained attention of many software practitioners during the last decade since it has contributed several benefits to the software development process. However, empirical evidence of its dominance in terms of internal code quality, external code quality and productivity is fairly limited. Objective: The aim behind conducting this controlled experiment with professional Java developers is to see the impact of Test-Driven Development (TDD) on internal code quality, external code quality and productivity compared to Test-Last Development (TLD). Results: Experiment results indicate that values found related to number of acceptance test cases passed, McCabe's Cyclomatic complexity, branch coverage, number of lines of code per person hours, number of user stories implemented per person hours are statistically insignificant. However, static code analysis results were found statistically significant in the favor of TDD. Moreover, the results of the survey revealed that the majority of developers in the experiment prefer TLD over TDD, given the lesser required level of learning curve as well as the minimum effort needed to understand and employ TLD compared to TDD.

Context: The study selection process is critical to improve the reliability of secondary studies. Goal: To evaluate the selection strategies commonly employed in secondary studies in software engineering. Method: Building on these strategies, a study selection process was formulated and evaluated in a systematic review. Results: The selection process used a more inclusive strategy than the one typically used in secondary studies, which led to additional relevant articles. Conclusions: The results indicates that a good-enough sample could be obtained by following a less inclusive but more efficient strategy, if the articles identified as relevant for the study are a representative sample of the population, and there is a homogeneity of results and quality of the articles.

Time pressure is prevalent in the software industry in which shorter and shorter deadlines and high customer demands lead to increasingly tight deadlines. However, the effects of time pressure have received little attention in software engineering research. We performed a controlled experiment on time pressure with 97 observations from 54 subjects. Using a two-by-two crossover design, our subjects performed requirements review and test case development tasks. We found statistically significant evidence that time pressure increases efficiency in test case development (high effect size Cohen’s d=1.279) and in requirements review (medium effect size Cohen’s d=0.650). However, we found no statistically significant evidence that time pressure would decrease effectiveness or cause adverse effects on motivation, frustration or perceived performance. We also investigated the role of knowledge but found no evidence of the mediating role of knowledge in time pressure as suggested by prior work, possibly due to our subjects. We conclude that applying moderate time pressure for limited periods could be used to increase efficiency in software engineering tasks that are well structured and straight forward.

Value stream mapping (VSM) is one of the several Lean practices, which has recently attracted interest in the software engineering community. In other contexts (such as military, health and production), VSM has achieved considerable improvements in processes and products. The goal is to capitalize on these benefits in the software intensive product development context. The primary contribution is that we are extending the definition of waste to fit in the software intensive product development context. As traditionally in VSM everything that is not considered valuable is waste, we do this practically by looking at value beyond the customer perspective and using the software value map. An evaluation has been conducted through an industrial case study. First, the instantiation and motivations for selecting certain strategies have been provided. Second, the outcome of the VSM is described in detail. The instantiation of VSM via workshops was considered good as workshops allowed active interaction and discussion stakeholders' groups that are distant from each other in the regular work. With respect to waste and improvement identification, the participants were able to identify similar improvement suggestions. In a retrospective, the value stream approach was perceived positively by the practitioners with respect to process and outcome. Copyright © 2014 John Wiley & Sons, Ltd.

System of systems are of high complexity, and for each system, many different requirements are implemented in parallel. Systems are developed with some degree of managerial independence but later on have to work together. In this situation, many requirements are written, implemented, and tested in parallel for different systems that are to be integrated. This makes identifying bottlenecks challenging, and visualizations often used on project level (such as Kanban boards or burndown charts) have to be extended/complemented to cope with the increased complexity. In response to these challenges, the contributions of this study are to propose the following: (i) a visualization for early identification and proactive removal of bottlenecks; (ii) a visualization to check on the success of bottleneck resolution; and (iii) to provide an industry evaluation of the visualizations in a case study of a system of systems developed at Ericsson AB in Sweden. The feedback by the practitioners showed that the visualizations were perceived as useful in improving throughput and lead time. The quantitative analysis showed that the visualizations were able in identifying bottlenecks and showing improvements or the lack thereof. On the basis of the qualitative and quantitative data collected, we conclude that the visualizations are useful in bottleneck identification and resolution. Copyright © 2014 John Wiley & Sons, Ltd.

SUMMARY Given the current state of the art in research, practitioners are faced with the challenge of choosing scripted testing (ST) or exploratory testing (ET). This study aims at systematically incorporating strengths of ET and ST in a hybrid testing process to overcome the weaknesses of each. We utilized systematic review and practitioner interviews to identify strengths and weaknesses of ET and ST. Strengths of ET were mapped to weaknesses of ST and vice versa. Noblit and Hare's lines‐of‐argument method was used for data analysis. The results of the mapping were used as input to codesign a hybrid process with experienced practitioners. We found a clear need to create a hybrid process as follows: (i) both ST and ET provide strengths and weaknesses, and these depend on some particular conditions, which prevents preference of one approach to another; and (ii) the mapping showed that it is possible to address the weaknesses in one process by the strengths of the other in a hybrid form. With the input from literature and industry experts, a flexible and iterative hybrid process was designed. Practitioners can clearly benefit from using a hybrid process given the mapping of advantages and disadvantages. Copyright © 2013 John Wiley & Sons, Ltd.

The link between maintenance and product quality, as well as the high cost of software maintenance, highlights the importance of efficient maintenance processes. Sustaining maintenance work efficiency in a global software development setting that involves a transfer is a challenging endeavor. Studies report on the negative effect of transfers on efficiency. However, empirical evidence on the magnitude of the change in efficiency is scarce. In this study we used a lean indicator to visualize variances in defect resolution cycles for two large products during evolution, before, during and after a transfer. Focus group meetings were also held for each product. Study results show that during and immediately after the transfer the defect inflow is higher, bottlenecks are more visible, and defect resolution cycles are longer, as compared to before the transfer. Furthermore we highlight the factors that influenced the change in defect resolution cycles before, during, and after the transfer.

Large open and closed source organizations like Google, Facebook and Mozilla are migrating their products towards rapid releases. While this allows faster time-to-market and user feedback, it also implies less time for testing and bug fixing. Since initial research results indeed show that rapid releases fix proportionally less reported bugs than traditional releases, this paper investigates the changes in software testing effort after moving to rapid releases. We analyze the results of 312,502 execution runs of the 1,547 mostly manual system level test cases of Mozilla Fire fox from 2006 to 2012 (5 major traditional and 9 major rapid releases), and triangulated our findings with a Mozilla QA engineer. In rapid releases, testing has a narrower scope that enables deeper investigation of the features and regressions with the highest risk, while traditional releases run the whole test suite. Furthermore, rapid releases make it more difficult to build a large testing community, forcing Mozilla to increase contractor resources in order to sustain testing for rapid releases.

Background - Validity threats should be considered and consistently reported to judge the value of an empirical software engineering research study. The relevance of specific threats for a particular research study depends on the worldview or philosophical worldview of the researchers of the study. Problem/Gap - In software engineering, different categorizations exist, which leads to inconsistent reporting and consideration of threats. Contribution - In this paper, we relate different worldviews to software engineering research methods, identify generic categories for validity threats, and provide a categorization of validity threats with respect to their relevance for different world views. Thereafter, we provide a checklist aiding researchers in identifying relevant threats. Method - Different threat categorizations and threats have been identified in literature, and are reflected on in relation to software engineering research. Results - Software engineering is dominated by the pragmatist worldviews, and therefore use multiple methods in research. Maxwell's categorization of validity threats has been chosen as very suitable for reporting validity threats in software engineering research. Conclusion - We recommend to follow a checklist approach, and reporting first the philosophical worldview of the researcher when doing the research, the research methods and all threats relevant, including open, reduced, and mitigated threats.

SUMMARY Software security can be improved by identifying and correcting vulnerabilities. In order to reduce the cost of rework, vulnerabilities should be detected as early and efficiently as possible. Static automated code analysis is an approach for early detection. So far, only few empirical studies have been conducted in an industrial context to evaluate static automated code analysis. A case study was conducted to evaluate static code analysis in industry focusing on defect detection capability, deployment, and usage of static automated code analysis with a focus on software security. We identified that the tool was capable of detecting memory related vulnerabilities, but few vulnerabilities of other types. The deployment of the tool played an important role in its success as an early vulnerability detector, but also the developers perception of the tools merit. Classifying the warnings from the tool was harder for the developers than to correct them. The correction of false positives in some cases created new vulnerabilities in previously safe code. With regard to defect detection ability, we conclude that static code analysis is able to identify vulnerabilities in different categories. In terms of deployment, we conclude that the tool should be integrated with bug reporting systems, and developers need to share the responsibility for classifying and reporting warnings. With regard to tool usage by developers, we propose to use multiple persons (at least two) in classifying a warning. The same goes for making the decision of how to act based on the warning. Copyright © 2012 John Wiley & Sons, Ltd.

Context: Systems of systems (SoS) are highly complex and are integrated on multiple levels (unit, component, system, system of systems). Many of the characteristics of SoS (such as operational and managerial independence, integration of system into system of systems, SoS comprised of complex systems) make their development and testing challenging.

Increasing the number of persons working on quality assurance (QA) tasks, e.g., reviews and testing, increases the number of defects detected -- but it also increases the total effort unless effort is controlled with fixed effort budgets. Our research investigates how QA tasks should be configured regarding two parameters, i.e., time and number of people. We define an optimization problem to answer this question. As a core element of the optimization problem we discuss and describe how defect detection probability should be modeled as a function of time. We apply the formulas used in the definition of the optimization problem to empirical defect data of an experiment previously conducted with university students. The results show that the optimal choice of the number of persons depends on the actual defect detection probabilities of the individual defects over time, but also on the size of the effort budget. Future work will focus on generalizing the optimization problem to a larger set of parameters, including not only task time and number of persons but also experience and knowledge of the personnel involved, and methods and tools applied when performing a QA task.

Study selection in systematic reviews is prone to bias and there exist no commonly defined strategies of how to reduce the bias and resolve disagreement between researchers. This study aims at identifying strategies for bias reduction and disagreement resolution. A review of existing systematic reviews is conducted for study selection strategy identification. In total 13 different strategies have been identified.

Abstract Responsiveness to customer needs is an important goal in agile and lean software development. One major aspect is to have a continuous and smooth flow that quickly delivers value to the customer. In this paper we apply cumulative flow diagrams to visualize the flow of lean software development. The main contribution is the definition of novel measures connected to the diagrams to achieve the following goals: (1) increase throughput and reduce lead‐time to achieve high responsiveness to customers' needs and (2) to provide a tracking system that shows the progress/status of software product development. An evaluation of the measures in an industrial case study showed that practitioners found them useful and identify improvements based on the measurements, which were in line with lean and agile principles. Furthermore, the practitioners found the measures useful in seeing the progress of development for complex products where many tasks are executed in parallel. The measures are now an integral part of the improvement work at the studied company. Copyright © 2010 John Wiley & Sons, Ltd.

Code reviews with static analysis tools are today recommended by several security development processes. Developers are expected to use the tools' output to detect the security threats they themselves have introduced in the source code. This approach assumes that all developers can correctly identify a warning from a static analysis tool (SAT) as a security threat that needs to be corrected. We have conducted an industry experiment with a state of the art static analysis tool and real vulnerabilities. We have found that average developers do not correctly identify the security warnings and only developers with specific experiences are better than chance in detecting the security vulnerabilities. Specific SAT experience more than doubled the number of correct answers and a combination of security experience and SAT experience almost tripled the number of correct security answers.

In order to draw valid conclusions when aggregating evidence it is important to describe the context in which industrial studies were conducted. This paper structures the context for empirical industrial studies and provides a checklist. The aim is to aid researchers in making informed decisions concerning which parts of the context to include in the descriptions. Furthermore, descriptions of industrial studies were surveyed.

Reading techniques help to guide reviewers during individual software inspections. In this experiment, we completely transfer the principle of statistical usage testing to inspection reading techniques for the first time. Statistical usage testing relies on a usage profile to determine how intensively certain parts of the system shall be tested from the users' perspective. Usage-based reading applies statistical usage testing principles by utilizing prioritized use cases as a driver for inspecting software artifacts (e.g., design). In order to reflect how intensively certain use cases should be inspected, time budgets are introduced to usage-based reading where a maximum inspection time is assigned to each use case. High priority use cases receive more time than low priority use cases. A controlled experiment is conducted with 23 Software Engineering M.Sc. students inspecting a design document. In this experiment, usage-based reading without time budgets is compared with time controlled usage-based reading. The result of the experiment is that time budgets do not significantly improve inspection performance. In conclusion, it is sufficient to only use prioritized use cases to successfully transfer statistical usage testing to inspections.