Testing and verification of self-driving cars are essential for ensuring their safety and reliability. In the context of the ICST 2024 self-driving cars testing tool competition, we present ITS4SDC, our tool for selecting roads that challenge lane-keeping assist systems by leading the car off the road. ITS4SDC leverages a long short-term memory-based model for the classification of roads as safe and unsafe and subsequently selects unsafe roads for testing.

ABSTRACT Background High‐level system testing of applications that use data from e‐Government services as input requires test data that is real‐life‐like but where the privacy of personal information is guaranteed. Applications with such strong requirement include information exchange between countries, medicine, banking, and so on. This review aims to synthesise the current state‐of‐the‐practice in this domain. Objectives The objective of this Systematic Review is to identify existing approaches for creating and evolving synthetic test data without using real‐life raw data. Methods We followed well‐known methodologies for conducting systematic literature reviews, including the ones from Kitchenham and PRISMA as well as guidelines for analysing the limitations of our review and its threats to validity. Results A variety of methods and tools exist for creating privacy‐preserving test data. Our search found 1013 publications in IEEE Xplore, ACM Digital Library, and SCOPUS. We extracted data from 75 of those publications and identified 37 approaches that answer our research question partly. A common prerequisite for using these methods and tools is direct access to real‐life data for data anonymization or synthetic test data generation. Nine existing synthetic test data generation approaches were identified that were closest to answering our research question. Nevertheless, further work would be needed to add the ability to evolve synthetic test data to the existing approaches. Conclusions None of the publications covered our requirements completely, only partially. Synthetic test data evolution is a field that has not received much attention from researchers but needs to be explored in Digital Government Solutions, especially since new legal regulations are being put in force in many countries.

Metamorphic Testing (MT) addresses the test oracle problem by defining how program outputs should change in response to specific input changes. The relations between input changes and their corresponding output changes are called Metamorphic Relations (MRs). Generating suitable MRs is complex and often requires deep domain knowledge. Our previous work introduced MetaTrimmer, a test-data-driven approach for selecting and constraining MRs, involving three steps: Test Data (TD) Generation, MT Process, and MR Analysis. MR Analysis is done to decide whether the violation of an MR for a specific input data pair (original and changed) indicates a failure or simply means that the MR does not apply for the chosen inputs. In this paper, we present an association-rule-based approach that semi-automatically extracts constraints dividing the input space into valid/invalid data during the MR Analysis step of MetaTrimmer. We validate our approach using 44 methods to which six predefined MRs are applied. Our results indicate that the proposed method efficiently identifies correct input data space constraints. More studies are needed to provide additional evidence that MetaTrimmer with the enhanced MR Analysis step is scalable and generalisable.

Regression testing is crucial in ensuring that pure code refactoring does not adversely affect existing software functionality, but it can be expensive, accounting for half the cost of software maintenance. Automated test case generation reduces effort but may generate weak test suites. Test amplification is a promising solution that enhances tests by generating additional or improving existing ones, increasing test coverage, but it faces the test oracle problem. To address this, we propose a test oracle derivation approach that uses object state data produced during System Under Test (SUT) test execution to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT’s intended behaviour. Our preliminary evaluation shows that the proposed approach can enhance the detection of behaviour changes substantially, providing initial evidence of its effectiveness.

Identifying and selecting high-quality Metamorphic Relations (MRs) is a challenge in Metamorphic Testing (MT). While some techniques for automatically selecting MRs have been proposed, they are either domain-specific or rely on strict assumptions about the applicability of a pre-defined MRs. This paper presents a preliminary evaluation of MetaTrimmer, a method for selecting and constraining MRs based on test data. MetaTrimmer comprises three steps: generating random test data inputs for the SUT (Step 1), performing test data transformations and logging MR violations (Step 2), and conducting manual inspections to derive constraints (Step 3). The novelty of MetaTrimmer is its avoidance of complex prediction models that require labeled datasets regarding the applicability of MRs. Moreover, MetaTrimmer facilitates the seamless integration of MT with advanced fuzzing for test data generation. In a preliminary evaluation, MetaTrimmer shows the potential to overcome existing limitations and enhance MR effectiveness.

Metamorphic Testing (MT) addresses the test oracle problem by examining the relationships between input-output pairs in consecutive executions of the System Under Test (SUT). These relations, known as Metamorphic Relations (MRs), specify the expected output changes resulting from specific input changes. However, achieving full automation in generating, selecting, and understanding MR violations poses challenges. Our research aims to develop methods and tools that assist testers in generating MRs, defining constraints, and providing explainability for MR outcomes. In the MR generation phase, we explore automated techniques that utilise a domain-specific language to generate and describe MRs. The MR constraint definition focuses on capturing the nuances of MR applicability by defining constraints. These constraints help identify the specific conditions under which MRs are expected to hold. The evaluation and validation involve conducting empirical studies to assess the effectiveness of the developed methods and validate their applicability in real-world regression testing scenarios. Through this research, we aim to advance the automation of MR generation, enhance the understanding of MR violations, and facilitate their effective application in regression testing.

Metamorphic Testing (MT) is a testing technique that can effectively alleviate the oracle problem. MT uses Metamorphic Relations (MRs) to determine if a test case passes or fails. MRs specify how the outputs should vary in response to specific input changes when executing the System Under Test (SUT). If a particular MR is violated for at least one test input (and its change), there is a high probability that the SUT has a fault. On the other hand, if a particular MR is not violated, it does not guarantee that the SUT is fault free. However, deciding if the MR is being violated due to a bug or because the MR does not hold/fit for particular conditions generated by specific inputs remains a manual task and unexplored. In this paper, we develop a method for refining MRs to offer hints as to whether a violation results from a bug or arises from the MR not being matched to certain test data under specific circumstances. In our initial proof-of-concept, we derive the relevant information from rules using the Association Rule Mining (ARM) technique. In our initial proof-of-concept, we validate our method on a toy example and discuss the lessons learned from our experiments. Our proof-of-concept demonstrates that our method is applicable and that we can provide suggestions that help strengthen the test suite for regression testing purposes.

We use mobile apps on a daily basis and there is an app for everything. We trust these applications with our most personal data. It is therefore important that these apps are as secure and well usable as possible. So far most studies on the maintenance and security of mobile applications have been done on Android applications. We do, however, not know how well these results translate to iOS.This research project aims to close this gap by analysing iOS applications with regards to maintainability and security. Regarding maintainability, we analyse code smells in iOS applications, the evolution of code smells in iOS applications and compare code smell distributions in iOS and Android applications. Regarding security, we analyse the evolution of the third-party library dependency network for the iOS ecosystem. Additionally, we analyse how publicly reported vulnerabilities spread in the library dependency network.Regarding maintainability, we found that the distributions of code smells in iOS and Android applications differ. Code smells in iOS applications tend to correspond to smaller classes, such as Lazy Class. Regarding security, we found that the library dependency network of the iOS ecosystem is not growing as fast as in some other ecosystems. There are less dependencies on average than for example in the npm ecosystem and, therefore, vulnerabilities do not spread as far.

Third party libraries are used to integrate existing solutions for common problems and help speed up development. The use of third party libraries, however, can carry risks, for example through vulnerabilities in these libraries. Studying the dependency networks of package managers lets us better understand and mitigate these risks. So far, the dependency networks of the three most important package managers of the Apple ecosystem, CocoaPods, Carthage and Swift PM, have not been studied. We analysed the dependencies for all publicly available open source libraries up to December 2021 and compiled a dataset containing the dependency networks of all three package managers. The dependency networks can be used to analyse how vulnerabilities are propagated through transitive dependencies. In order to ease the tracing of vulnerable libraries we also queried the NVD database and included publicly reported vulnerabilities for these libraries in the dataset.

Metamorphic testing (TM) examines the relations between inputs and outputs of test runs. These relations are known as metamorphic relations (MR). Currently, MRs are handpicked and require in-depth knowledge of the System Under Test (SUT), as well as its problem domain. As a result, the identification and selection of high-quality MRs is a challenge. Kanewala et al. suggested the Predicting Metamorphic Relations (PMR) approach for automatic prediction of applicable MRs picked from a predefined list. PMR is based on a Support Vector Machine (SVM) model using features derived from the Control Flow Graphs (CFGs) of 100 Java methods. The original study of Kanewala et al. showed encouraging results, but developing classification models from CFG-related features is costly. In this paper, we aim at developing a PMR approach that is less costly without losing performance. We complement the original PMR approach by considering other than CFG-related features. We define 21 features that can be directly extracted from source code and build several classifiers, including SVM models. Our results indicate that using the original CFG-based method-level features, in particular for a SVM with random walk kernel (RWK), achieve better predictions in terms of AUC-ROC for most of the candidate MRs than our models. However, for one of the candidate MRs, using source code features achieved the best AUC-ROC result (greater than 0.8).

Metamorphic Testing (MT) addresses the test oracle problem by examining the relations between inputs and outputs of test executions. Such relations are known as Metamorphic Relations (MRs). In current practice, identifying and selecting suitable MRs is usually a challenging manual task, requiring a thorough grasp of the SUT and its application domain. Thus, Kanewala et al. proposed the Predicting Metamorphic Relations (PMR) approach to automatically suggest MRs from a list of six pre-defined MRs for testing newly developed methods. PMR is based on a classification model trained on features extracted from the control-flow graph (CFG) of 100 Java methods. In our replication study, we explore the generalizability of PMR. First, since not all details necessary for a replication are provided, we rebuild the entire preprocessing and training pipeline and repeat the original study in a close replication to verify the reported results and establish the basis for further experiments. Second, we perform a conceptual replication to explore the reusability of the PMR model trained on CFGs from Java methods in the first step for functionally identical methods implemented in Python and C++. Finally, we retrain the model on the CFGs from the Python and C++ methods to investigate the dependence on programming language and implementation details. We were able to successfully replicate the original study achieving comparable results for the Java methods set. However, the prediction performance of the Java-based classifiers significantly decreases when applied to functionally equivalent Python and C++ methods despite using only CFG features to abstract from language details. Since the performance improved again when the classifiers were retrained on the CFGs of the methods written in Python and C++, we conclude that the PMR approach can be generalized, but only when classifiers are developed starting from code artefacts in the used programming language.

Together with many success stories, promises such as the increase in production speed and the improvement in stakeholders’ collaboration have contributed to making agile a transformation in the software industry in which many companies want to take part. However, driven either by a natural and expected evolution or by contextual factors that challenge the adoption of agile methods as prescribed by their creator(s), software processes in practice mutate into hybrids over time. Are these still agile? In this article, we investigate the question: what makes a software development method agile? We present an empirical study grounded in a large-scale international survey that aims to identify software development methods and practices that improve or tame agility. Based on 556 data points, we analyze the perceived degree of agility in the implementation of standard project disciplines and its relation to used development methods and practices. Our findings suggest that only a small number of participants operate their projects in a purely traditional or agile manner (under 15 percent). That said, most project disciplines and most practices show a clear trend towards increasing degrees of agility. Compared to the methods used to develop software, the selection of practices has a stronger effect on the degree of agility of a given discipline. Finally, there are no methods or practices that explicitly guarantee or prevent agility. We conclude that agility cannot be defined solely at the process level. Additional factors need to be taken into account when trying to implement or improve agility in a software company. Finally, we discuss the field of software process-related research in the light of our findings and present a roadmap for future research.

Abstract FinTech companies are challenging established financial institutions' dominance by offering the same products with a superior customer experience and delivering new features faster. The adoption of agile software development partially enables this competitive advantage. In response to this challenge, banks explore how they can improve their agile processes. LHV, a mid‐sized bank, uses agile practices but faces the challenge of further improvement to stay competitive with FinTech companies. In this article, we explore how LHV can improve its agile software development process. We conduct a case study at LHV where we first derive eight change proposals based on a literature review and interviews. Then, we report on how LHV implemented the change proposals and their perceived impact. Our results stress the importance of taking a coherent approach to improving agile processes by considering both business units and operations involved in the product life‐cycle. It is also necessary to align organizational structures to enable team autonomy by, for instance, decentralizing decision authority. Finally, it is beneficial to adapt agile practices to their context and have an IT architecture and technology supporting the agile approach.

Abstract Hardly any software development process is used as prescribed by authors or standards. Regardless of company size or industry sector, a majority of project teams and companies use hybrid development methods (short: hybrid methods) that combine different development methods and practices. Even though such hybrid methods are highly individualized, a common understanding of how to systematically construct synergetic practices is missing. In this article, we make a first step towards a statistical construction procedure for hybrid methods. Grounded in 1467 data points from a large‐scale practitioner survey, we study the question: What are hybrid methods made of and how can they be systematically constructed? Our findings show that only eight methods and few practices build the core of modern software development. Using an 85% agreement level in the participants' selections, we provide examples illustrating how hybrid methods can be characterized by the practices they are made of. Furthermore, using this characterization, we develop an initial construction procedure, which allows for defining a method frame and enriching it incrementally to devise a hybrid method using ranked sets of practice.

Reviews on test-driven development (TDD) studies suggest that the conflicting results reported in the literature are due to unobserved factors, such as the tasks used in the experiments, and highlight that there are very few industry experiments conducted with professionals. The goal of this study is to investigate the impact of a new factor, the chosentask, and thedevelopment approachon external quality in an industrial experimental setting with 17 professionals. The participants are junior to senior developers in programming with Java, beginner to novice in unit testing, JUnit, and they have no prior experience in TDD. The experimental design is a$2\times 2$cross-over, i.e., we use two tasks for each of the two approaches, namely TDD and incremental test-last development (ITLD). Our results reveal that bothdevelopment approachandtaskare significant factors with regards to the external quality achieved by the participants. More specifically, the participants produce higher quality code during ITLD in which splitting user stories into subtasks, coding, and testing activities are followed, compared to TDD. The results also indicate that the participants produce higher quality code during the implementation of Bowling Score Keeper, compared to that of Mars Rover API, although they perceived both tasks as of similar complexity. An interaction between thedevelopment approachandtaskcould not be observed in this experiment. We conclude that variables that have not been explored so often, such as the extent to which the task is specified in terms of smaller subtasks, and developers’ unit testing experience might be critical factors in TDD experiments. The real-world appliance of TDD and its implications on external quality still remain to be challenging unless these uncontrolled and unconsidered factors are further investigated by researchers in both academic and industrial settings.

Due to high demand, many cloud data centers have been developed across the world, consuming a large amount of energy. Making cloud datacenters energy efficient has become essential. Energy consumption of data centers can be minimized by designing energy-efficient hardware, software, and infrastructure. In this paper, we aim at giving an overview of software techniques, affected stakeholders, performance features, datasets, and tools used to make cloud data centers energy-efficient. To achieve this goal, we conducted a systematic mapping study using five online databases. After applying inclusion/exclusion and quality criteria, we selected 58 publications for further analysis. Our results indicate that all publications are solution and validation type of publications. We did not find publications containing evaluations in industry. We found that workload scheduling is the most frequently proposed technique used to improve cloud datacenters' energy efficiency. We found that not considering violations of service level agreements mostly affects end-users of cloud data centers. When analyzing how suggested solutions are validated, we identified the need to develop a standardized set of performance measures to benchmark software techniques proposed to make cloud data centers greener.

Productivity in software development has been studied for a long time and is still a topic of interest. Many factors, ranging from team size to music listened by developers, have been studied regarding their effect on productivity. Surprisingly, little is known about how the dynamics of open-source projects that use agile practices are related to the productivity of the developer teams. Our study aims to close this gap by analyzing the productivity of open-source projects using measures that are popular in the context of agile software development. To do this, we study data from seven open-source projects and calculated both the individual velocity of developers and focus factor of teams per iteration. First, we applied statistical process control to identify iterations with out-of-control velocity and focus factor values. Then, we studied these iterations regarding four context factors that partly characterize the dynamics of open-source projects, i.e., iteration length, turnover of developers who left, turnover of new developers, and team stability index. Our results suggest that high team stability and low turnover are strongly associated with iterations showing high velocity.

Energy consumption of mobile apps is receiving a lot of attention from researchers. Recent studies indicate that energy consumption of mobile devices could be lowered by improving the quality of mobile apps. Frequent refactoring is one way of achieving this goal. We explore the performance and energy impact of several common code refactorings in Android apps. Experimental results indicate that some code smell refactorings positively impact the energy consumption of Android apps. Refactoring of the code smells 'Duplicated code' and 'Type checking' reduce energy consumption by up to 10.8%. Significant reduction in energy consumption, however, does not seem to be directly related to the increase or decrease of execution time. In addition, the energy impact over permutations of code smell refactorings in the selected Android apps was small. When analyzing the order in which refactorings were made across code smell types, it turned out that some permutations resulted in a reduction and some in an increase of energy consumption for the analyzed apps.

To (semi-)automatically classify user needs collected from online open sources, we propose the Open Innovation in Requirements Engineering (OIRE) method. OIRE is mimicking the well-known Kano-model exclusively using data from online reviews instead of conducting interviews with select focus groups. In our previous research, we introduced the design, implementation and preliminary validation of the OIRE method. In this paper, we introduce the tool support for the OIRE method, OIRE-System (OIRE-S), and evaluate the OIRE method using OIRE-S. For that purpose, we conducted a case study with two Chinese companies using OIRE-S as tool support when applying the OIRE method. Based on the results of the case study, we conclude that the analysis results of the OIRE method provide valuable information for stakeholders and, thus, is useful to decision-makers in the industry, in particular as a complement to traditional requirement elicitation and analysis activities.

The quality of automatic app feature extraction from app reviews depends on various aspects, e.g. the feature extraction method, training and evaluation datasets, evaluation method etc. Annotation guidelines used to guide the annotation of training and evaluation datasets can have a considerable impact to the quality of the whole system but it is one of the aspects that is often overlooked. We conducted a study in which we explore the effects of annotation guidelines to the quality of app feature extraction. We propose several changes to the existing annotation guidelines with the goal of making the extracted app features more useful to app developers. We test the proposed changes via simulating the application of the new annotation guidelines and evaluating the performance of the supervised machine learning models trained on datasets annotated with initial and simulated annotation guidelines. While the overall performance of automatic app feature extraction remains the same as compared to the model trained on the dataset with initial annotations, the features extracted by the model trained on the dataset with simulated new annotations are less noisy and more informative to app developers.

Among the multitude of software development processes available, hardly any is used by the book. Regardless of company size or industry sector, a majority of project teams and companies use customized processes that combine different development methods--so-called hybrid development methods. Even though such hybrid development methods are highly individualized, a common understanding of how to systematically construct synergetic practices is missing. In this paper, we make a first step towards devising such guidelines. Grounded in 1,467 data points from a large-scale online survey among practitioners, we study the current state of practice in process use to answer the question: What are hybrid development methods made of? Our findings reveal that only eight methods and few practices build the core of modern software development. This small set allows for statistically constructing hybrid development methods. Using an 85% agreement level in the participants' selections, we provide two examples illustrating how hybrid development methods are characterized by the practices they are made of. Our evidence-based analysis approach lays the foundation for devising hybrid development methods.

Research collaboration between industry and academia supports improvement and innovation in industry and helps ensure the industrial relevance of academic research. However, many researchers and practitioners in the community believe that the level of joint industry-academia collaboration (IAC) projects in Software Engineering (SE) research is relatively low, creating a barrier between research and practice. The goal of the empirical study reported in this paper is to explore and characterize the state of IAC with respect to industrial needs, developed solutions, impacts of the projects and also a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review (SLR) study. To address the above goal, we conducted an opinion survey among researchers and practitioners with respect to their experience in IAC. Our dataset includes 101 data points from IAC projects conducted in 21 different countries. Our findings include: (1) the most popular topics of the IAC projects, in the dataset, are: software testing, quality, process, and project managements; (2) over 90% of IAC projects result in at least one publication; (3) almost 50% of IACs are initiated by industry, busting the myth that industry tends to avoid IACs; and (4) 61% of the IAC projects report having a positive impact on their industrial context, while 31% report no noticeable impacts or were “not sure”. To improve this situation, we present evidence-based recommendations to increase the success of IAC projects, such as the importance of testing pilot solutions before using them in industry. This study aims to contribute to the body of evidence in the area of IAC, and benefit researchers and practitioners. Using the data and evidence presented in this paper, they can conduct more successful IAC projects in SE by being aware of the challenges and how to overcome them, by applying best practices (patterns), and by preventing anti-patterns.

Abstract The emergence of agile methods and practices has not only changed the development processes but might also have affected how companies conduct software process improvement (SPI). Through a set of complementary studies, we aim to understand how SPI has changed in times of agile software development. Specifically, we aim (a) to identify and characterize the set of publications that connect elements of agility to SPI, (b) to explore to which extent agile methods/practices have been used in the context of SPI, and (c) to understand whether the topics addressed in the literature are relevant and useful for industry professionals. To study these questions, we conducted an in‐depth analysis of the literature identified in a previous mapping study, an interview study, and an analysis of the responses given by industry professionals to SPI‐related questions stemming from an independently conducted survey study. Regarding the first question, we identified 55 publications that focus on both SPI and agility of which 48 present and discuss how agile methods/practices are used to steer SPI initiatives. Regarding the second question, we found that the two most frequently mentioned agile methods in the context of SPI are Scrum and Extreme Programming (XP), while the most frequently mentioned agile practices are integrate often, test‐first, daily meeting, pair programming, retrospective, on‐site customer, and product backlog. Regarding the third question, we found that a majority of the interviewed and surveyed industry professionals see SPI as a continuous activity. They agree with the agile SPI literature that agile methods/practices play an important role in SPI activities but that the importance given to specific agile methods/practices does not always coincide with the frequency with which these methods/practices are mentioned in the literature.

Software engineering researchers have, over the years, proposed different critical success factors (CSFs) which are believed to be critically correlated with the success of software projects. To conduct an empirical investigation into the correlation of CSFs with success of software projects, we adapt and extend in this work an existing contingency fit model of CSFs. To archive the above objective, we designed an online survey and gathered CSF-related data for 101 software projects in the Turkish software industry. Among our findings is that the top three CSFs having the most significant associations with project success were: (1) team experience with the software development methodologies, (2) team’s expertise with the task, and (3) project monitoring and controlling. A comprehensive correlation analysis between the CSFs and project success indicates positive associations between the majority of the factors and variables, however, in most of the cases at non-significant levels. By adding to the body of evidence in this field, the results of the study will be useful for a wide audience. Software managers can use the results to prioritize the improvement opportunities in their organizations w.r.t. the discussed CSFs. Software engineers might use the results to improve their skills in different dimensions, and researchers might use the results to prioritize and conduct follow-up in-depth studies on those factors.

Requirements Engineering (RE) has established itself as a software engineering discipline over the past decades. While researchers have been investigating the RE discipline with a plethora of empirical studies, attempts to systematically derive an empirical theory in context of the RE discipline have just recently been started. However, such a theory is needed if we are to define and motivate guidance in performing high quality RE research and practice. We aim at providing an empirical and externally valid foundation for a theory of RE practice, which helps software engineers establish effective and efficient RE processes in a problem-driven manner. We designed a survey instrument and an engineer-focused theory that was first piloted in Germany and, after making substantial modifications, has now been replicated in 10 countries worldwide. We have a theory in the form of a set of propositions inferred from our experiences and available studies, as well as the results from our pilot study in Germany. We evaluate the propositions with bootstrapped confidence intervals and derive potential explanations for the propositions. In this article, we report on the design of the family of surveys, its underlying theory, and the full results obtained from the replication studies conducted in 10 countries with participants from 228 organisations. Our results represent a substantial step forward towards developing an empirical theory of RE practice. The results reveal, for example, that there are no strong differences between organisations in different countries and regions, that interviews, facilitated meetings and prototyping are the most used elicitation techniques, that requirements are often documented textually, that traces between requirements and code or design documents are common, that requirements specifications themselves are rarely changed and that requirements engineering (process) improvement endeavours are mostly internally driven. Our study establishes a theory that can be used as starting point for many further studies for more detailed investigations. Practitioners can use the results as theory-supported guidance on selecting suitable RE methods and techniques.

Context: To lower the cost of Requirements Engineering (RE) and to help software organizations, in particular, small / start-up companies with small teams and little marketing and customer research budget, gain a better understanding of user satisfaction with existing products, we propose the Open Innovation in Requirements Engineering (OIRE) method to systematically analyze information that is openly available in the Internet. Goal: In this article, we describe the components of the OIRE method, explain what it can do and how it provides useful information to help software engineers and managers make better informed decisions. In addition, we present three typical use cases and a proof-of-concept of the method demonstrating the applicability of the method using real-world data collected from the Internet. Method: First, we choose app reviews from app stores and app monitoring platforms as the sources of input. Then, based on a list of features, we extract feature-related text as the input. Next, we manually apply the OIRE method and verify the accuracy of the outputs of each component of the OIRE method. Result: We found the following: a) The accuracy of component 1 and component 2 of the OIRE method exceeds 63% and the highest number is 88%; b) The overall accuracy of the OIRE method is between 50% and 61%; c) For the use cases chosen, the OIRE method has the potential to help software engineers and managers make better informed RE decisions.

Effort estimation is important to correctly plan the use of resources in a software project. In agile projects, a correct effort estimation helps decide which issues have to be fixed or finished during the next iteration. However, estimating issues can be a complex task and developers may make inaccurate estimates. Therefore, the use of automatic approaches that aim to support developers in the estimation process is worth to be studied. We explore the use of a predictive model that use developers' features to assign story points to issue reports. The performance of the model is compared with the performance of models based on features extracted from the text of issues. We assessed the models with different performance metrics including Accuracy, Mean Absolute Error, and Standardized Accuracy. The preliminary results show that the model that uses developers' features sightly outperforms the models based on text features, indicating a promising research direction.

Deciding about the content and readiness when shipping a new product release can have a strong impact on the success (or failure) of the product. Having formerly analyzed the state-of-the art in this area, the objective for this paper was to better understand the process and rationale of real-world release decisions and to what extent research on release readiness is aligned with industrial needs. We designed two rounds of surveys with focus on the current (Survey-A) and the desired (Survey-B) process of how to make release readiness decisions. We received 49 and 40 valid responses for Survey-A and Survey-B, respectively.

Research collaboration between industry and academia supports improvement and innovation in industry and helps to ensure industrial relevance in academic research. However, many researchers and practitioners believe that the level of joint industry-academia collaboration (IAC) in software engineering (SE) research is still relatively low, compared to the amount of activity in each of the two communities. The goal of the empirical study reported in this paper is to exploratory characterize the state of IAC with respect to a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review study. To address the above goal, we gathered the opinions of researchers and practitioners w.r.t. their experiences in IAC projects. Our dataset includes 47 opinion data points related to a large set of projects conducted in 10 different countries. We aim to contribute to the body of evidence in the area of IAC, for the benefit of researchers and practitioners in conducting future successful IAC projects in SE. As an output, the study presents a set of empirical findings and evidence-based recommendations to increase the success of IAC projects.

Sustainability in software engineering is a relatively new and fast growing field of research. Green software engineering aims to produce sustainable software products with minimum negative impact on the environment. In order to make greener software products, software practitioners need actionable timely information, to make useful trade-offs between energy efficiency and other quality attributes, like performance, during development. Software analytics could be used to provide this support, as it combines information from different software artifacts and converts it into useful information. The objective of this paper is to provide an overview of the sub-domains, contribution types, research types, research methods, future research potentials and the role of software analytics in the field of green software engineering in 2015-16. We applied the systematic mapping method and conducted a search for studies in six online databases. Screening of papers was done according to inclusion/exclusion criteria and 50 selected studies were classified after analysis and data extraction. We found that there are many validation studies but hardly any evaluation and experience papers in the domain of green software engineering. Only 11 out of 50 papers in the green software engineering domain used software analytics techniques to foster green software engineering. Our results indicate the need to develop new/improved automated software analytics tools for software practitioners along with metrics explaining the correlation between energy usage and other quality attributes.

Software developers often include available open-source software packages into their projects to minimize redundant effort. However, adding a package to a project can also introduce risks, which can propagate through multiple levels of dependencies. Currently, not much is known about the structure of open-source package ecosystems of popular programming languages and the extent to which transitive bug propagation is possible. This paper analyzes the dependency network structure and evolution of the JavaScript, Ruby, and Rust ecosystems. The reported results reveal significant differences across language ecosystems. The results indicate that the number of transitive dependencies for JavaScript has grown 60% over the last year, suggesting that developers should look more carefully into their dependencies to understand what exactly is included. The study also reveals that vulnerability to a removal of the most popular package is increasing, yet most other packages have a decreasing impact on vulnerability. The findings of this study can inform the development of dependency management tools.

Predicting the resolution times of issue reports in software development is important, because it helps allocate resources adequately. However, issue resolution time (IRT) prediction is difficult and prediction quality is limited. A common approach in industry is to base predictions on expert knowledge. While this manual approach requires the availability and effort of experts, automated approaches using data mining and machine learning techniques require a small upfront investment for setting up the data collection and analysis infrastructure as well as the availability of sufficient past data for model building. Several approaches for automated IRT prediction have been proposed and evaluated. The aim of our study was (1) to compare the prediction quality of expert-based IRT prediction in a software company located in Estonia with that of various fully automated IRT prediction approaches proposed and used by other researchers, including k-means clustering, k-nearest neighbor classification, Naïve Bayes classification, decision trees, random forest (RF) and ordered logistic regression (OLR), and (2) to improve the current IRT prediction quality in the company at hand. For our study, we analyzed issue reports collected by the company in the period from April 2011 to January 2015. Regarding our first goal, we found that experts in the case company were able to predict IRTs approximately 50% of the time within the range of ±10% of the actual IRTs. In addition, 67% of the experts' predictions have an absolute error that is less or equal 0.5 hours. When applying the automated approaches used by other researchers to the company's data, we observed lower predictive quality as compared to IRT predictions made by the company's experts, even for the best-performing approaches RF and OLR. Regarding our second goal, after unsuccessfully experimenting with improvements to the RF and OLR based approaches, we managed to develop models based on text analysis that achieved a prediction quality at par or better than that achieved by company experts.

Context: To survive in a highly competitive software market, product managers are striving for frequent, incremental releases in ever shorter cycles. Release decisions are characterized by high complexity and have a high impact on project success. Under such conditions, using the experience from past releases could help product managers to take more informed decisions.

Methods for predicting issue lifetime can help software project managers to prioritize issues and allocate resources accordingly. Previous studies on issue lifetime prediction have focused on models built from static features, meaning features calculated at one snapshot of the issue's lifetime based on data associated to the issue itself. However, during its lifetime, an issue typically receives comments from various stakeholders, which may carry valuable insights into its perceived priority and difficulty and may thus be exploited to update lifetime predictions. Moreover, the lifetime of an issue depends not only on characteristics of the issue itself, but also on the state of the project as a whole. Hence, issue lifetime prediction may benefit from taking into account features capturing the issue's context (contextual features). In this work, we analyze issues from more than 4000 GitHub projects and build models to predict, at different points in an issue's lifetime, whether or not the issue will close within a given calendric period, by combining static, dynamic and contextual features. The results show that dynamic and contextual features complement the predictive power of static ones, particularly for long-term predictions.

Abstract Software test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, because test automation usually requires substantial upfront investments, automation is not always more cost‐effective than manual testing. To support decision‐makers in finding the optimal degree of test automation in a given project, we recently proposed a process simulation model using the System Dynamics modeling technique and used the simulation model in the context of a case study with a software company in Calgary, Canada. With the help of the simulation model, we were able to evaluate the performance of test processes with varying degrees of automation of test activities and help testers choose the most optimal cases. The goal of the earlier study was to investigate how the simulation model can help decision‐makers decide whether and to what degree the company should automate their test processes. In this article, we present further details of the System Dynamics model, its usage scenarios and examples of simulation experiments independent from a specific company context. Copyright © 2015 John Wiley & Sons, Ltd.

International audience

This paper describes preliminary results regarding research on Open Innovation (OI) in peer reviewed literature. Special focus is put on the field of computer science as compared to other fields using the categorization scheme of Thomson Reuters’ Web of Science Core Collection (ISI Core Collection). The study was triggered by the observation that peer reviewed literature on OI seems to have a less distinct growth in recent years than publications included in Google Scholar which involves non-peer reviewed literature to a much larger extent. When analyzing peer reviewed literature on OI, we observed that the field of Computer Science seems to have significantly less diversity than all other fields with more than ten publications indexed in the ISI Core Collection. Our preliminary interpretation of this observation is that the topic ‘Open Innovation’ has not yet been researched and discussed in depth and breadth comparable to other fields, in particular Business Economics, Engineering, and Operations Research Management Science.

Context: Not releasing software on time can cause substantial loss in revenue. Continuous awareness of the product release status is required. Release readiness is a time-dependent attribute of the status of the product release, which aggregates the degree of satisfaction of a portfolio of release process and product measures.

Context: Exploratory Testing has experienced a rise in popularity in the industry with the emergence of agile development practices, yet it remains unclear, in which domains and how it is used in practice.

Software test processes are complex and costly. To reduce testing effort without compromising effectiveness and product quality, automation of test activities has been adopted as a popular approach in software industry. However, since test automation usually requires substantial upfront investments, automation is not always more cost-effective than manual testing. To support decision-makers in finding the optimal degree of test automation in a given project, we propose in this paper a simulation model using the System Dynamics (SD) modeling technique. With the help of the simulation model, we can evaluate the performance of test processes with varying degrees of automation of test activities and help testers choose the most optimal cases. As the case study, we describe how we used our simulation model in the context of an Action Research (AR) study conducted in collaboration with a software company in Calgary, Canada. The goal of the study was to investigate how the simulation model can help decision-makers decide whether and to what degree the company should automate their test processes. As a first step, we compared the performances of the current fully manual testing with several cases of partly automated testing as anticipated for implementation in the partner company. The development of the simulation model as well as the analysis of simulation results helped the partner company to get a deeper understanding of the strengths and weaknesses of their current test process and supported decision-makers in the cost effective planning of improvements of selected test activities.

Scientific and engineering research is heavily dependent on effective development and use of software artifacts. Many of these artifacts are produced by the scientists themselves, rather than by trained software engineers. To address the challenges in this area, a research community often referred to as "Development of Scientific Software" has emerged in the last few decades. As this research area has matured, there has been a sharp increase in the number of papers and results made available, and it has thus become important to summarize and provide an overview about those studies. Through a systematic mapping and bibliometrics study, we have reviewed 130 papers in this area. We present the results of our study in this paper. Also we have made the mapping data available on an online repository which is planned to be updated on a regular basis. The results of our study seem to suggest that many software engineering techniques and activities are being used in the development of scientific software. However, there is still a need for further exploration of the usefulness of specific software engineering techniques (e.g., regarding software maintenance, evolution, refactoring, re(v)-engineering, process and project management) in the scientific context. It is hoped that this article will help (new) researchers get an overview of the research space and help them to understand the trends in the area.

Increasing the number of persons working on quality assurance (QA) tasks, e.g., reviews and testing, increases the number of defects detected -- but it also increases the total effort unless effort is controlled with fixed effort budgets. Our research investigates how QA tasks should be configured regarding two parameters, i.e., time and number of people. We define an optimization problem to answer this question. As a core element of the optimization problem we discuss and describe how defect detection probability should be modeled as a function of time. We apply the formulas used in the definition of the optimization problem to empirical defect data of an experiment previously conducted with university students. The results show that the optimal choice of the number of persons depends on the actual defect detection probabilities of the individual defects over time, but also on the size of the effort budget. Future work will focus on generalizing the optimization problem to a larger set of parameters, including not only task time and number of persons but also experience and knowledge of the personnel involved, and methods and tools applied when performing a QA task.

A key component in successfully managing software products is to properly, and in a timely manner, identify and secure competitive advantage by innovation via feature differentiation. Although open source software (OSS) is not a new idea, several product development companies that operate in a market-driven context have started to use open source solutions as core software components in their products. Adopting open source core components implies a lower degree of control over software development and increased business risk associated with integrating differentiating contributions into the core release stream. Whether and how to adjust the current requirements management practices after the adoption of OSS components to fully benefit from the concept of open innovation has not yet been empirically explored. We outline experiences and challenges related to leveraging open innovation via engaging in OSS identified during 19 interviews with practitioners occupying different roles in the requirements management process at a large company followed by four validation interviews with other practitioners. We then propose a research agenda for requirements and decision management in the open innovation context and suggest which challenges in requirements engineering open innovation affects.

During systematic literature reviews it is necessary to assess the quality of empirical papers. Current guidelines suggest that two researchers should independently apply a quality checklist and any disagreements must be resolved. However, there is little empirical evidence concerning the effectiveness of these guidelines. This paper investigates the three techniques that can be used to improve the reliability (i.e. the consensus among reviewers) of quality assessments, specifically, the number of reviewers, the use of a set of evaluation criteria and consultation among reviewers. We undertook a series of studies to investigate these factors. Two studies involved four research papers and eight reviewers using a quality checklist with nine questions. The first study was based on individual assessments, the second study on joint assessments with a period of inter-rater discussion. A third more formal randomised block experiment involved 48 reviewers assessing two of the papers used previously in teams of one, two and three persons to assess the impact of discussion among teams of different size using the evaluations of the “teams” of one person as a control. For the first two studies, the inter-rater reliability was poor for individual assessments, but better for joint evaluations. However, the results of the third study contradicted the results of Study 2. Inter-rater reliability was poor for all groups but worse for teams of two or three than for individuals. When performing quality assessments for systematic literature reviews, we recommend using three independent reviewers and adopting the median assessment. A quality checklist seems useful but it is difficult to ensure that the checklist is both appropriate and understood by reviewers. Furthermore, future experiments should ensure participants are given more time to understand the quality checklist and to evaluate the research papers.

Context: The authors wanted to assess whether the quality of published human-centric software engineering experiments was improving. This required a reliable means of assessing the quality of such experiments. Aims: The aims of the study were to confirm the usability of a quality evaluation checklist, determine how many reviewers were needed per paper that reports an experiment, and specify an appropriate process for evaluating quality. Method: With eight reviewers and four papers describing human-centric software engineering experiments, we used a quality checklist with nine questions. We conducted the study in two parts: the first was based on individual assessments and the second on collaborative evaluations. Results: The inter-rater reliability was poor for individual assessments but much better for joint evaluations. Four reviewers working in two pairs with discussion were more reliable than eight reviewers with no discussion. The sum of the nine criteria was more reliable than individual questions or a simple overall assessment. Conclusions: If quality evaluation is critical, more than two reviewers are required and a round of discussion is necessary. We advise using quality criteria and basing the final assessment on the sum of the aggregated criteria. The restricted number of papers used and the relatively extensive expertise of the reviewers limit our results. In addition, the results of the second part of the study could have been affected by removing a time restriction on the review as well as the consultation process.

Decision-making in requirements engineering often needs to be done in the presence of uncertainty. Rigorous methods can help to increase the probability of making the most appropriate decisions under the given circumstances. Robustness is a measure for the degree of stability of a solution in case of changes in the problem parameters. This paper presents a method (called DECIDERelease) that applies simulation-based analysis and multi-criteria decision analysis on top of the existing strategic release planning approach EVOLVE*. The purpose of DECIDEReleaseis to qualify decision-making by pro-actively exploring the robustness of the operational plans of upcoming releases. Based on this analysis, the strategic release plan that is the most robust against assumed changes in planning parameters at operational level can be selected. As a proof-of-concept, the applicability of DECIDERelease is demonstrated by an illustrative case study. Results from a survey administered to managers and engineers in industry indicate that the proposed method is perceived as useful in practice.

Software process simulation modeling (SPSM) research has increased since the first ProSim workshop held in 1998 and Kellner, Madachy and Raffo (KMR) discussed the "why, what and how" of process simulation. This paper aims to assess how SPSM has evolved during the past 10 years in particular whether the reasons for SPSM, the simulation paradigms, tools, problem domains, and model scopes have changed. We performed a systematic literature review of software process simulation papers from the ProSim series publications in the last decade. We identified 96 studies from the sources and included them in this review. The papers were categorized into four major types and data needed to address each research question was extracted. We found a need for refining the reasons and the classification scheme for SPSM introduced by KMR. More emerging SPSM paradigms and model scopes were added to enhance KMR's discussion. Trends over time showed that interest in continuous modeling was decreasing and interest in micro-processes was increasing. Hybrid models were based primarily on system dynamics and discrete event simulation and were all implemented by vertical integration. We recommend SPSM research concentrate more on recent software processes and on making SPSM more reusable and thus easier to build.

Software Process Simulation (SPS) research has increased since 1998 when the first ProSim Workshop was held. This paper aims to reveal how SPS has evolved during the past 10 years based on the preliminary results from the systematic literature review of SPS publications from 1998 to 2007. Trends over the period showed that interest in continuous modelling was decreasing and interest in micro-processes was increasing. Hybrid models were based primarily on system dynamics and discrete event simulation and were all implemented by vertical integration.

Evaluation, prioritization and selection of candidate requirements are of tremendous importance and impact for subsequent software development. Effort, time as well as quality constraints have to be taken into account. Typically, different stakeholders have conflicting priorities and the requirements of all these stakeholders have to be balanced in an appropriate way to ensure maximum value of the final set of requirements. Trade-off analysis is needed to proactively explore the impact of certain decisions in terms of all the criteria and constraints. The proposed method called Quantitative WinWin uses an evolutionary approach to provide support for requirements negotiations. The novelty of the presented idea is four-fold. Firstly, it iteratively uses the Analytical Hierarchy Process (AHP) for a stepwise analysis with the aim to balance the stakeholders' preferences related to different classes of requirements. Secondly, requirements selection is based on predicting and rebalancing its impact on effort, time and quality. Both prediction and rebalancing uses the simulation model prototype GENSIM. Thirdly, alternative solution sets offered for decision-making are developed incrementally based on thresholds for the degree of importance of requirements and heuristics to find a best fit to constraints. Finally, trade-off analysis is used to determine non-dominated extensions of the maximum value that is achievable under resource and quality constraints. As a main result, quantitative WinWin proposes a small number of possible sets of requirements from which the actual decision-maker can finally select the most appropriate solution.

Combining technology enabled learning and knowledge management is a very promising strategy to systematic competence building in software organizations. In order to set-up a combined learning and knowledge management environment and to support organization members in packaging small chunks of knowledge into courseware modules, Fraunhofer IESE developed the IntView-KM methodology. IntView-KM defines how to provide the organizational, methodological, and technological prerequisites that enable software engineers to produce small courseware modules (contents) as a byproduct of their daily work.This paper presents the two phases of the IntView-KM methodology. Furthermore, tool support for courseware module production is recommended and the existing tool support summarized. Finally, experiences in applying IntView-KM and results of a first evaluation of the tool support for IntView-KM are described.

Defining, prioritizing, and selecting requirements are problems of tremendous importance. In this paper, a new approach called Quantitative WinWin for decision support in requirements negotiation is studied. The difference to Boehm's WinWin groupware-based negotiation support is the inclusion of quantitative methods as a backbone for better and more objective decisions. Like Boehm's original WinWin, Quantitative WinWin uses an iterative approach, with the aim to increase knowledge about the requirements during each iteration. The novelty of the presented idea is three-fold. Firstly, it uses the Analytical Hierarchy Process for a stepwise determination of the stakeholders' preferences in quantitative terms. Secondly, these results are combined with methods for early effort estimation, in our case using the simulation prototype GENSIM, to evaluate the feasibility of alternative requirements subsets in terms of their related implementation efforts. Thirdly, it reflects the increasing knowledge gained about the requirements during each iteration, in a similar way as it is done in Boehm's spiral model for software development. As main result, quantitative WinWin offers decision support for selecting the most appropriate requirements based on the preferences of the stakeholders, the business value of requirements and a given maximum development effort.

System Dynamics (SD) is a method that allows for integrated modelling of technical as well as managerial aspects responsible for the dynamic complexity of systems. Therefore, simulation of SD models can also be a tool for problem analysis within software organisations. In this paper, experience with SD modelling of software processes and projects within Siemens is reported. Special focus is put on problems encountered during knowledge acquisition for SD model building, like inadequate guidance while conducting SD modelling projects, and insufficient methodical support for re-using available or generating missing knowledge. Both problems were addressed in a research project, jointly conducted by Fraunhofer IESE and Siemens Corporate Technology. One of the results of this project is a prescriptive process model for building SD models. This process model, which is briefly outlined in the paper, provides guidance for a systematic development of SD models in software organisations.

The use of test-coverage measures (e.g., block-coverage) to control the software test process has become an increasingly common practice. This is justified by the assumption that higher test-coverage helps achieve higher defect-coverage and therefore improves software quality. In practice, data often show that defect-coverage and test-coverage grow over time, as additional testing is performed. However, it is unclear whether this phenomenon of concurrent growth can be attributed to a causal dependency, or if it is coincidental, simply due to the cumulative nature of both measures. Answering such a question is important as it determines whether a given test-coverage measure should be monitored for quality control and used to drive testing. Although it is no general answer to this problem, a procedure is proposed to investigate whether any test-coverage criterion has a genuine additional impact on defect-coverage when compared to the impact of just running additional test cases. This procedure applies in typical testing conditions where the software is tested once, according to a given strategy, coverage measures are collected as well as defect data. This procedure is tested on published data, and the results are compared with the original findings. The study outcomes do not support the assumption of a causal dependency between test-coverage and defect-coverage, a result for which several plausible explanations are provided.

The use of test coverage measures (e.g. block coverage) to control the software test process has become an increasingly common practice. This is justified by the assumption that higher test coverage helps achieve higher defect coverage and therefore improves software quality. In practice, data often shows that defect coverage and test coverage grow over time, as additional testing is performed. However, it is unclear whether this phenomenon of concurrent growth can be attributed to a causal dependency or if it is coincidental, simply due to the cumulative nature of both measures. Answering such a question is important as it determines whether a given test coverage measure should be monitored for quality control and used to drive testing. Although this is no general answer to the problem above, we propose a procedure to investigate whether any test coverage criterion has a genuine additional impact on defect coverage when compared to the impact of just running additional test cases. This procedure is applicable in typical testing conditions where the software is tested once, according to a given strategy and where coverage measures are collected as well as defect data. We then test the procedure on published data and compare our results with the original findings. The study outcomes do not support the assumption of a causal dependency between test coverage and defect coverage, a result for which several plausible explanations are provided.