Novel aircraft are increasingly designed by combining numerous discipline-specific simulation and modeling tools. To determine whether existing tools can be used to design a given aircraft, manufacturers require in-depth knowledge about their tools’ capabilities. The research software used in digital design workflows, however, often lacks formal capability specifications.

• The SAFT-GT toolchain enables semi-automatic Attack-Fault Tree generation for enhanced safety and security assessment in self-adaptive systems. • The toolchain efficiently integrates into the feedback loop of self-adaptive systems, allowing for dynamic updates based on security assessments. • A user study with domain experts confirms the toolchain’s relevance and practical applicability in real-world scenarios. • Performance experiments demonstrate that the Attack-Fault Tree generation pipeline operates within feasible time constraints, supporting real-time applications. • The complete toolchain and resources are provided for download, fostering further research and collaboration in the field. In the rapidly evolving landscape of software engineering, the demand for robust and secure systems has become increasingly critical. This is especially true for self-adaptive systems due to their complexity and the dynamic environments in which they operate. To address this issue, we designed and developed the SAFT-GT toolchain that tackles the multifaceted challenges associated with ensuring both safety and security. This paper provides a comprehensive description of the toolchain’s architecture and functionalities, including the Attack-Fault Trees generation and model combination approaches. We emphasize the toolchain’s ability to integrate seamlessly with existing systems, allowing for enhanced safety and security analyses without requiring extensive modifications and domain knowledge. Our proposed approach can address evolving security threats, including both known vulnerabilities and emerging attack vectors that could compromise the system. As a use case for the toolchain, we integrate it into the feedback loop of self-adaptive systems. Finally, to validate the practical applicability of the toolchain, we conducted an extensive user study involving domain experts, whose insights and feedback underscore the toolchain’s relevance and usability in real-world scenarios. Our findings demonstrate the toolchain’s effectiveness in real-world applications while highlighting areas for future improvements. The toolchain and associated resources are available in an open-source repository to promote reproducibility and encourage further research in this field.

As space exploration advances and the commercialization and militarization of space technologies expand, ensuring the security of space assets has become a paramount concern. A key factor contributing to this challenge is the growing reliance on off-the-shelf hardware and software. While such components accelerate the adoption and commercial use of space technologies, they also introduce new vulnerabilities and broaden the attack surface. This paper highlights the critical importance of integrating cybersecurity concepts throughout the entire design lifecycle of space systems. It examines key dimensions of secure space system development, including secure engineering practices, comprehensive testing methodologies, strategies for cyber resiliency, and the role of standardization in fostering a consistent and robust security posture across the industry. By addressing these essential aspects, the paper underscores the need for a holistic, lifecycle-driven approach to safeguarding space systems against evolving cyber threats.

Abrupt and unexpected terminations of software are termed as software crashes. They can be challenging to analyze. Finding the root cause requires extensive manual effort and expertise to connect information sources like stack traces, source code, and logs. Typical approaches to fault localization require either test failures or source code. Crashes occurring in production environments, such as that of SAP HANA, provide solely crash logs and stack traces. We present a novel approach to localize faults based only on the stack trace information and no additional runtime information, by fine-tuning large language models (LLMs). We address complex cases where the root cause of a crash differs from the technical cause, and is not located in the innermost frame of the stack trace. As the number of historic crashes is insufficient to fine-tune LLMs, we augment our dataset by leveraging code mutators to inject synthetic crashes into the code base. By fine-tuning on 64,369 crashes resulting from 4.1 million mutations of the HANA code base, we can correctly predict the root cause location of a crash with an accuracy of 66.9% while baselines only achieve 12.6% and 10.6%. We substantiate the generalizability of our approach by evaluating on two additional open-source databases, SQLite and DuckDB, achieving accuracies of 63% and 74%, respectively. Across all our experiments, fine-tuning consistently outperformed prompting non-finetuned LLMs for localizing faults in our datasets.

The increasing complexity and adoption of machine learning (ML) pipelines has led to a rising demand for effective visualization tools. This paper presents a comprehensive review of existing tools for visualizing data flow in machine learning (ML) pipelines. We highlight the tools’ purposes, integration methods, and visualization techniques. We collected and analyzed 22 open-source tools and concepts, analyzing their features and classifying them based on their primary purpose. Our analysis revealed five main purposes of visualization tools: exploration, explanation, visual development, comparison, monitoring. We provide an analysis of their integration methods, from standalone visual interfaces to code-level libraries, as well as a review of various visualization techniques, including Directed Acyclic Graphs (DAGs), pipeline matrices, and annotated visualizations. Our findings highlight the importance of visualization in enhancing the interpretability and efficiency of ML workflows. Moreover, the paper provides key limitations and challenges in current visualization methods to promote future research directions enhancing the usability and functionality of ML pipeline visualization tools.

Abstract Question-and-answer platforms such as Stack Overflow are an important way for software developers to share and retrieve knowledge. However, reusing poorly understood code can lead to serious problems, such as bugs or security vulnerabilities. To better understand how code comments affect the perceived helpfulness of Stack Overflow answers, we conducted an online experiment simulating a Stack Overflow environment (n=91). The results indicate that both block and inline comments are perceived as significantly more helpful than uncommented source code. Moreover, novices rated code snippets with block comments as more helpful than those with inline comments. Interestingly, other surface features, such as the position of an answer and its answer score, were considered less important. Moreover, the content of Stack Overflow has been a major source for training large language models. AI-based coding assistants such as GitHub Copilot, which are based on these models, are changing the way Stack Overflow is used. However, our findings have implications beyond Stack Overflow. First, they may help to improve the relevance also of other community-driven platforms, which provide human advice and explanations of code solutions, complementing AI-based support for software developers. Second, since chat-based AI tools can be prompted to generate code in different ways, knowing which properties influence perceived helpfulness can lead to more targeted prompting strategies to generate readable code snippets.

Machine learning (ML)-enabled systems are being increasingly adopted by companies aiming to enhance their products and operational processes. This paper aims to deliver a comprehensive overview of the current status quo of engineering ML-enabled systems and lay the foundation to steer practically relevant and problem-driven academic research. We conducted an international survey to collect insights from practitioners on the current practices and problems in engineering ML-enabled systems. We received 188 complete responses from 25 countries. We conducted quantitative statistical analyses on contemporary practices using bootstrapping with confidence intervals and qualitative analyses on the reported problems using open and axial coding procedures. Our survey results reinforce and extend existing empirical evidence on engineering ML-enabled systems, providing additional insights into typical ML-enabled systems project contexts, the perceived relevance and complexity of ML life cycle phases, and current practices related to problem understanding, model deployment, and model monitoring. Furthermore, the qualitative analysis provides a detailed map of the problems practitioners face within each ML life cycle phase and the problems causing overall project failure. The results contribute to a better understanding of the status quo and problems in practical environments. We advocate for the further adaptation and dissemination of software engineering practices to enhance the engineering of ML-enabled systems. • International survey gathering insights from 188 practitioners across 25 countries. • Overview of current practices and challenges in engineering ML-enabled systems. • Inferential quantitative analysis reporting the status quo with confidence intervals. • Qualitative analysis mapping ML life cycle challenges and causes of project failure.

Abstract Ensuring the correct and safe behavior of a spacecraft is a main objective in space-system design. Since spacecraft consist of highly complex and tightly integrated components developed by large teams of engineers from various different disciplines, this is a challenging task. Increasingly, formal verification methods such as model checking are applied to establish the correctness of safety-critical parts or subsystems. Generally, the often limited scalability of model checking due to the state-space explosion problem hinders the wide-spread adoption of this technique. In this paper, we systematically examine the scalability of model checking for verifying behavioral models that arise within early space-system design phases. For this, we created a representative model for the mode management of a satellite that can be scaled in terms of its size and the complexity of interactions between system components. The model can be transformed into the input languages of various model-checking tools, which enables a comparative study of various model-checking algorithms and also facilitates analyzing the impact of different communication schemes on the scalability. The evaluation shows promising results regarding the applicability of model checking within the spacecraft design process.

Abstract Text mining techniques, particularly those leveraging machine learning for natural language processing, have gained significant attention for qualitative data analysis in software testing. However, their complexity and lack of transparency can pose challenges, especially in safety-critical domains where simpler, interpretable solutions are often preferred unless accuracy is heavily compromised. This study investigates the trade-offs between complexity, effort, accuracy, and utility in text mining and clustering techniques, focusing on their application for detecting functional dependencies among manual integration test cases in safety-critical systems. Using empirical data from an industrial testing project at ALSTOM Sweden, we evaluate various string distance methods, NCD compressors, and machine learning approaches. The results highlight the impact of preprocessing techniques, such as tokenization, and intrinsic factors, such as text length, on algorithm performance. Findings demonstrate how text mining and clustering can be optimized for safety-critical contexts, offering actionable insights for researchers and practitioners aiming to balance simplicity and effectiveness in their testing workflows.

Microservice-based systems have emerged as an effective architecture for countless industry applications. They provide applications as small, independent, and modular services. With the increasing interest in such systems, it is important to tackle challenges related to their quality assurance. However, to advance research in this area, systems are required to evaluate new approaches and tools. In this paper, we perform a systematic literature search for systems used in research for testing and monitoring microservice-based systems to aid future research. We provide an overview of the found studies and the systems used in their evaluation. We compose a list of publicly available systems and their characteristics, like size, available tests, and technologies used. Finally, we investigated the context in which these systems were used to provide insights in their usage and additional data that is available for them.

Smart manufacturing systems adapt to changes in the environment, which are detected via sensors. Decisions are then made by the control software of such manufacturing systems, e.g., by integrated AI-based algorithms. The design and evaluation of such algorithms require the availability of high-quality datasets. This paper provides an overview of the existing publicly available manufacturing datasets, offering a detailed exploration of the current landscape of shared data resources in the manufacturing sector and highlighting the utility of these datasets. The review identifies nine notable datasets with extensive documentation and comprehensive data coverage. Each of these datasets is described in detail and can be used for developing intelligent manufacturing systems, assessing their quality, and reporting on open gaps.

Data preprocessing, which includes data integration, cleaning, and transformation, is often a time and effort-intensive step due to its fundamental importance. This crucial phase is integral for ensuring the quality and suitability of data for sub-sequent stages, such as feature engineering and model training in Machine Learning-enabled and data-driven systems. This paper provides an extensive overview of data preprocessing functions in Python and examines their application and prevalence in computational notebooks by analyzing 149,048 computational notebooks collected from Kaggle. Despite the crucial role played by data preprocessing in model performance, our results expose a significant lack of emphasis on data preprocessing activities in the examined notebooks. Notably, users holding the highest rankings tend to skip data preprocessing steps and focus on model-related activities. Although other users exhibit more frequent incorporation of data preprocessing methods, the overall prevalence remains relatively limited. We discovered that data preparation practices such as missing values are present in 20 % to 60 % of the notebooks depending on the competition, whereas outliers handling is only present in less than 20% of the analyzed scripts. The most frequently and consistently applied practices are the data transformation methods.

Deviations from expected behavior during runtime, known as anomalies, have become more common due to the systems' complexity, especially for microservices. Consequently, analyzing runtime monitoring data, such as logs, traces for microservices, and metrics, is challenging due to the large volume of data collected. Developing effective rules or AI algorithms requires a deep understanding of this data to reliably detect unfore-seen anomalies. This paper seeks to comprehend anomalies and current anomaly detection approaches across diverse industrial sectors. Additionally, it aims to pinpoint the parameters necessary for identifying anomalies via runtime monitoring data. Therefore, we conducted semi-structured interviews with fifteen industry participants who rely on anomaly detection during runtime. Additionally, to supplement information from the interviews, we performed a literature review focusing on anomaly detection approaches applied to industrial real-life datasets. Our paper (1) demonstrates the diversity of interpretations and examples of software anomalies during runtime and (2) explores the reasons behind choosing rule-based approaches in the industry over self-developed AI approaches. AI-based approaches have become prominent in published industry-related papers in the last three years. Furthermore, we (3) identified key monitoring parameters collected during runtime (logs, traces, and metrics) that assist practitioners in detecting anomalies during runtime without introducing bias in their anomaly detection approach due to inconclusive parameters.

Since 2020, 33 literature reviews have systematically synthesized research on the continuous development of AI, also known as Machine Learning Operations (MLOps), reflecting the increasing prevalence of AI models across various fields and the multifaceted challenges in their development, integration, and deployment. Yet, the lack of comprehensive analysis of these literature reviews and their covered topics complicates selecting relevant ones and anticipating future trends and research. In addition, these literature reviews gathered related 1397 primary sources to describe aspects of AI's continuous development, integration, and deployment, posing a hidden gem to gain insights into the past and present work and derive insights into the future of AI's continuous development. With this work, we 1) systematically collected and summarised 33 literature reviews via a Multivocal Literature Review (MLR) that focus on the continuous development, deployment, and integration of AI models. 2) Due to minimal overlap between the literature reviews' primary sources, we offer holistic insights into and interrelations of frequently addressed topics. These topics encompass the AI development pipeline, respective Software Engineering (SE) practices, and associated challenges. 3) We discuss future research directions for AI's continuous development, integration, and deployment. Therefore, we base our arguments on identified clusters in the primary sources of literature reviews. This discussion focuses on AI model reliability and resource consumption, emphasizing the interrelation of proposed future work and the effects on the whole pipeline.

Infrastructure-as-Code (IaC) frameworks empower developers to swiftly define and provision their infrastructure with a single click. However, the domain-specific languages (DSLs) utilized for coding the infrastructure often lean towards provider specificity rather than being application-centric. This results in increased developer effort, as they are compelled to duplicate data when deploying serverless functions across diverse regions and providers within federated FaaS environments. To mitigate this challenge, we introduce CODE, a framework engineered to streamline the deployment of functions in federated FaaS settings. CODE facilitates automatic deployment directly from the storage of any provider, eliminating the need for additional development effort to upload or copy deployment packages between disparate providers. Aligned with the guiding principle of “code once, deploy everywhere”, CODE adopts a three-level hierarchy: function → providers → cloud regions. This architectural approach dramatically reduces the lines of code (LoC) in IaC scripts by up to 9.23× when contrasted with prevailing IaC frameworks such as Terraform and Serverless Framework. Additionally, CODE’s unified storage interface slashes LoC by up to 81.8%, both within CODE itself and when coding functions that use storage from providers such as AWS and GCP. In our comprehensive evaluation, we assessed the correlation between deployment package size and deployment time for various functions within a real-world serverless workflow across four regions of AWS and GCP. Our findings indicate that AWS deployment packages are significantly larger, often in the tens of megabytes, compared to GCP. Despite the larger size, AWS deploys these packages up to 6× faster than GCP.

Data pipelines are an integral part of various modern data-driven systems. However, despite their importance, they are often unreliable and deliver poor-quality data. A critical step toward improving this situation is a solid understanding of the aspects contributing to the quality of data pipelines. Therefore, this article first introduces a taxonomy of 41 factors that influence the ability of data pipelines to provide quality data. The taxonomy is based on a multivocal literature review and validated by eight interviews with experts from the data engineering domain. Data, infrastructure, life cycle management, development & deployment, and processing were found to be the main influencing themes. Second, we investigate the root causes of data-related issues, their location in data pipelines, and the main topics of data pipeline processing issues for developers by mining GitHub projects and Stack Overflow posts. We found data-related issues to be primarily caused by incorrect data types (33%), mainly occurring in the data cleaning stage of pipelines (35%). Data integration and ingestion tasks were found to be the most asked topics of developers, accounting for nearly half (47%) of all questions. Compatibility issues were found to be a separate problem area in addition to issues corresponding to the usual data pipeline processing areas (i.e., data loading, ingestion, integration, cleaning, and transformation). These findings suggest that future research efforts should focus on analyzing compatibility and data type issues in more depth and assisting developers in data integration and ingestion tasks. The proposed taxonomy is valuable to practitioners in the context of quality assurance activities and fosters future research into data pipeline quality.

Recent years have seen a surge in cyber-attacks, which can be prevented or mitigated using software security activities. OWASP SAMM is a maturity model providing a versatile way for companies to assess their security posture and plan for improvements. We perform an initial SAMM assessment in collaboration with a company in the financial domain. Our objective is to assess a holistic inventory of the company security-related activities, focusing on how different roles perform the assessment and how they perceive the instrument used in the process. We perform a case study to collect data using SAMM in a lightweight and novel manner through assessment using an online survey with 17 participants and a focus group with seven participants. We show that different roles perceive maturity differently and that the two assessments deviate only for specific practices making the lightweight approach a viable and efficient solution in industrial practice. Our results indicate that the questions included in the SAMM assessment tool are answered easily and confidently across most roles. Our results suggest that companies can productively use a lightweight SAMM assessment. We provide nine lessons learned for guiding industrial practitioners in the evaluation of their current security posture as well as for academics wanting to utilize SAMM as a research tool in industrial settings. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Technical debt is often the result of Short Run decisions made during code development, which can lead to long-term maintenance costs and risks. Hence, evaluating the progression of a project and understanding related code quality aspects is essential.

Building energy efficient software is an increasingly important task for mobile developers. However, a cumulative body of knowledge of techniques that support this goal does not exist. We conduct a systematic literature review to gather information on existing techniques that allow developers to increase energy efficiency in mobile apps. Based on a synthesis of the 91 included primary studies, we propose a taxonomy of techniques for improving the energy efficiency in mobile apps. The taxonomy includes seven main categories of techniques and serves as a collection of available methods for developers and as a reference guide for software testers when performing energy efficiency testing by the means of benchmark tests.

The widespread adoption of AI-enabled systems and their required continuous development and deployment (MLOps) sparks research interest due to the added intricacy of automatically handling data, code, and the model itself. A better understanding of the stages for the continuous development of AI, namely Data Handling, Model Learning, Software Development, and System Operations, and the respective tasks can help to optimize and improve their effectiveness.Thus, this paper explores the degree of automation, development effort, importance, utilization of computing resources, and factors contributing to automation throughout these stages and tasks. We conducted a questionnaire-based global survey to explore these topics by analyzing 150 responses from experienced AI, data, and MLOps engineers.The results determined that the stage System Operations is mainly automated. Whereas several tasks from the other three stages (e.g., data cleaning, data quality assurance, model design, model improvement, and system level quality assurance) are more often partially automated than automated, and documentation-related tasks are mostly not automated or developed. Participants required the highest development effort for the stage Data Handling. Furthermore, the study reveals a negative correlation between automation and the perceived development effort, whereas the importance of the tasks does not seem to affect automation. 93% of participants consider the availability of computing resources, with model training, data transformation, and data cleaning ranked as the most resource-intensive tasks.

Machine Learning (ML)-enabled systems that run in safety-critical settings expose humans to risks. Hence, it is important to build such systems with strong assurances for domain-specific safety requirements. Simulation as well as metaheuristic optimizing search have proven to be valuable tools for online testing of ML-enabled systems for early detection of hazards. However, the efficient generation of effective test cases remains a challenging issue. In particular, the testing process shall produce as many failures as possible but also unveil diverse sets of failure scenarios.To study this phenomenon, we introduce a risk-driven test case generation and diversity analysis method tailored to ML-enabled systems. Our approach uses an online testing technique based on metaheuristic optimizing search to falsify domain-specific safety requirements. All test cases leading to hazards are then analyzed to assess their diversity by using clustering and interpretable ML. We evaluated our approach in a collaborative robotics case study showing that generating tests considering risk metrics represents an effective strategy. Furthermore, we compare alternative optimizing search algorithms and rank them based on the overall diversity of the test cases, ultimately showing that selecting the testing strategy based on the number of failures only may be misleading.

Similar to the concept of "code smells", so-called "test smells" describe situations in test code that possibly indicate a deeper problem and require refactoring measures. The elimination of test smells ensures, among other things, that tests are easy to read, easier to maintain and less prone to errors. To automate the detection of test smells, we created the tool "SniffTest" (STest), which detects five types of test smells in JUnit tests, namely Anonymous Test, Long Test, Conditional Test Logic, Assertion Roulette, and Rotten Green Test. STest applies basic natural language processing methods, namely regular expressions and part-of-speech tagging. These methods are mostly programming language independent and can be easily adapted to other languages and frameworks. To measure the accuracy of the tool, we created a labelled dataset containing 854 JUnit test methods and 106 helper methods from eight large open source GitHub repositories, where we manually annotated the test smells. The tool showed an accuracy of 87 to 100% in detecting the different test smells. Both the tool and the dataset are made available as open source.

Competency models are widely adopted frameworks that are used to improve human resource functions and education. However, the characteristics of competency models related to the information security and cybersecurity domains are not well understood. To bridge this gap, this study investigates the current state of competency models related to the security domain through qualitative content analysis. Additionally, based on the competency model analysis, an evidence-based competency model is proposed. Examining the content of 27 models, we found that the models can benefit target groups in many different ways, ranging from policymaking to performance management. Owing to their many uses, competency models can arguably help to narrow the skills gap from which the profession is suffering. Nonetheless, the models have their shortcomings. First, the models do not cover all of the topics specified by the Cybersecurity Body of Knowledge (i.e., no model is complete). Second, by omitting social, personal, and methodological competencies, many models reduce the competency profile of a security expert to professional competencies. Addressing the limitations of previous work, the proposed competency model provides a holistic view of the competencies required by security professionals for job achievement and can potentially benefit both the education system and the labor market. To conclude, the implications of the competency model analysis and use cases of the proposed model are discussed.

Cyber–physical systems (CPS) are increasingly self-adaptive, i.e. they have the ability to introspect and change their behavior. This self-adaptation process must be considered when modeling the safety and security aspects of the system. This study collects and compares security attacks and safety hazards on self-adaptive systems (SAS) described in the literature. In addition, mitigation and treatment strategies, as well as the modeling and analysis approaches, are investigated. We conducted a systematic literature review on 21 selected papers. The selection process included a database search on four scientific databases using a common search string (1430 papers), forward and backward snowballing (1402 papers), and filtering the results based on predefined inclusion and exclusion criteria. The coding scheme to analyze the content of the papers was obtained through research questions, existing domain-specific taxonomies, and open coding. Safety and security are not jointly modeled in the context of self-adaptive systems. The adaptation process is often not considered in the attack and hazard analysis due to naïve assumptions and modeling. The proposed approaches are mostly verified and validated through simulation often using simple use cases and scenarios. A thorough and joint modeling approach for safety and security in self-adaptive systems is still an open challenge that needs to be addressed. Further work is needed to address the gap between safety and security modeling in self-adaptive systems. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board.

Companies struggle to continuously develop and deploy Artificial Intelligence (AI) models to complex production systems due to AI characteristics while assuring quality. To ease the development process, continuous pipelines for AI have become an active research area where consolidated and in-depth analysis regarding the terminology, triggers, tasks, and challenges is required. This paper includes a Multivocal Literature Review (MLR) where we consolidated 151 relevant formal and informal sources. In addition, nine-semi structured interviews with participants from academia and industry verified and extended the obtained information. Based on these sources, this paper provides and compares terminologies for Development and Operations (DevOps) and Continuous Integration (CI)/Continuous Delivery (CD) for AI, Machine Learning Operations (MLOps), (end-to-end) lifecycle management, and Continuous Delivery for Machine Learning (CD4ML). Furthermore, the paper provides an aggregated list of potential triggers for reiterating the pipeline, such as alert systems or schedules. In addition, this work uses a taxonomy creation strategy to present a consolidated pipeline comprising tasks regarding the continuous development of AI. This pipeline consists of four stages: Data Handling, Model Learning, Software Development and System Operations. Moreover, we map challenges regarding pipeline implementation, adaption, and usage for the continuous development of AI to these four stages.

High data quality is fundamental for today's AI-based systems. However, although data quality has been an object of research for decades, there is a clear lack of research on potential data quality issues (e.g., ambiguous, extraneous values). These kinds of issues are latent in nature and thus often not obvious. Nevertheless, they can be associated with an increased risk of future problems in AI-based systems (e.g., technical debt, data-induced faults). As a counterpart to code smells in software engineering, we refer to such issues as Data Smells. This article conceptualizes data smells and elaborates on their causes, consequences, detection, and use in the context of AI-based systems. In addition, a catalogue of 36 data smells divided into three categories (i.e., Believability Smells, Understandability Smells, Consistency Smells) is presented. Moreover, the article outlines tool support for detecting data smells and presents the result of an initial smell detection on more than 240 real-world datasets.

It is often overseen that AI-enabled systems are also software systems and therefore rely on software quality assurance (SQA). Thus, the goal of this study is to investigate the software quality assurance strategies adopted during the development, integration, and maintenance of AI/ML components and code. We conducted semi-structured interviews with representatives of ten Austrian SMEs that develop AI-enabled systems. A qualitative analysis of the interview data identified 12 issues in the development of AI/ML components. Furthermore, we identified when quality issues arise in AI/ML components and how they are detected. The results of this study should guide future work on software quality assurance processes and techniques for AI/ML components.

A common phenomenon in software development is that as a system is being built and incremented with new features, certain quality aspects of the system begin to deteriorate. Therefore, it is important to be able to accurately analyze and determine the quality implications of each change and increment to a system. To address this topic, the multinational SmartDelta project develops automated solutions for quality assessment of product deltas in a continuous engineering environment. The project will provide smart analytics from development artifacts and system executions, offering insights into quality degradation or improvements across different product versions, and providing recommendations for next builds.

Metamorphic testing has proven to be effective for test case generation and fault detection in many domains. It is a software testing strategy that uses certain relations between input-output pairs of a program, referred to as metamorphic relations. This approach is relevant in the autonomous systems domain since it helps in cases where the outcome of a given test input may be difficult to determine. In this paper therefore, we provide an overview of metamorphic testing as well as an implementation in the autonomous systems domain. We implement an obstacle detection and avoidance task in autonomous drones utilising the GNC API alongside a simulation in Gazebo. Particularly, we describe properties and best practices that are crucial for the development of effective metamorphic relations. We also demonstrate two metamorphic relations for metamorphic testing of single and more than one drones, respectively. Our relations reveal several properties and some weak spots of both the implementation and the avoidance algorithm in the light of metamorphic testing. The results indicate that metamorphic testing has great potential in the autonomous systems domain and should be considered for quality assurance in this field.

Recruiting and onboarding software testing professionals are complex and cost intensive activities. Whether onboarding is successful and sustainable depends on both the employee as well as the organization and is influenced by a number of often highly individual factors. Therefore, we propose the Software Testing Onboarding Model (STORM) for sustainably onboarding software testing professionals based on existing frameworks and models taking into account onboarding processes, sustainability, and test processes. In addition, we provide detailed instructions on how to apply the model to real-world onboarding processes.

As software engineering research becomes more concerned with the psychological, sociological and managerial aspects of software development, relevant theories from reference disciplines are increasingly important for understanding the field's core phenomena of interest. However, the degree to which software engineering research draws on relevant social sciences remains unclear. This study therefore investigates the use of social science theories in five influential software engineering journals over 13 years. It analyzes not only the extent of theory use but also what, how and where these theories are used. While 87 different theories are used, less than two percent of papers use a social science theory, most theories are used in only one paper, most social sciences are ignored, and the theories are rarely tested for applicability to software engineering contexts. Ignoring relevant social science theories may (1) undermine the community's ability to generate, elaborate and maintain a cumulative body of knowledge; and (2) lead to oversimplified models of software engineering phenomena. More attention to theory is needed for software engineering to mature as a scientific discipline.

Capturing customer workloads of database systems to replay these workloads during internal testing can be beneficial for software quality assurance. However, we experienced that such replays can produce a large amount of false positive alerts that make the results unreliable or time consuming to analyze. Therefore, we design a machine learning based approach that attributes root causes to the alerts. This provides several benefits for quality assurance and allows for example to classify whether an alert is true positive or false positive. Our approach considerably reduces manual effort and improves the overall quality assurance for the database system SAP HANA. We discuss the problem, the design and result of our approach, and we present practical limitations that may require further research.

Given the ongoing "arms race" in cybersecurity, the shortage of skilled professionals in this field is one of the strongest in computer science. The currently unmet staffing demand in cybersecurity is estimated at over 3 million jobs worldwide. Furthermore, the qualifications of the existing workforce are largely believed to be insufficient. We attempt to gain deeper insights into the nature of the current skill gap in cybersecurity. To this end, we correlate data from job ads and academic curricula using two kinds of skill characterizations: manual definitions from established skill frameworks as well as "skill topics" automatically derived by text mining tools. Our analysis shows a strong agreement between these two analysis techniques and reveals a substantial undersupply in several crucial skill categories, e.g., software and application security, security management, requirements engineering, compliance and certification. Based on the results of our analysis, we provide recommendations for future curricula development in cybersecurity so as to decrease the identified skill gaps.

Self-adaptive systems offer several attack surfaces due to the communication via different channels and the different sensors required to observe the environment. Often, attacks cause safety to be compromised as well, making it necessary to consider these two aspects together. Furthermore, the approaches currently used for safety and security analysis do not sufficient take into account the intermediate steps of an adaptation. Current work in this area ignores the fact that a self-adaptive system also reveals possible vulnerabilities (even if only temporarily) during the adaptation. To address this issue, we propose a modeling approach that takes into account the different relevant aspects of a system, its adaptation process, as well as safety hazards and security attacks. We present several models that describe different aspects of a self-adaptive system and we outline our idea of how these models can then be combined into an Attack-Fault Tree. This allows modeling aspects of the system on different levels of abstraction and co-evolve the models using transformations according to the adaptation of the system. Finally, analyses can then be performed as usual on the resulting Attack-Fault Tree.

Together with many success stories, promises such as the increase in production speed and the improvement in stakeholders’ collaboration have contributed to making agile a transformation in the software industry in which many companies want to take part. However, driven either by a natural and expected evolution or by contextual factors that challenge the adoption of agile methods as prescribed by their creator(s), software processes in practice mutate into hybrids over time. Are these still agile? In this article, we investigate the question: what makes a software development method agile? We present an empirical study grounded in a large-scale international survey that aims to identify software development methods and practices that improve or tame agility. Based on 556 data points, we analyze the perceived degree of agility in the implementation of standard project disciplines and its relation to used development methods and practices. Our findings suggest that only a small number of participants operate their projects in a purely traditional or agile manner (under 15 percent). That said, most project disciplines and most practices show a clear trend towards increasing degrees of agility. Compared to the methods used to develop software, the selection of practices has a stronger effect on the degree of agility of a given discipline. Finally, there are no methods or practices that explicitly guarantee or prevent agility. We conclude that agility cannot be defined solely at the process level. Additional factors need to be taken into account when trying to implement or improve agility in a software company. Finally, we discuss the field of software process-related research in the light of our findings and present a roadmap for future research.

Background: Recent empirical studies indicate that online controlled experimentation is rarely systematically applied. A structured and complete experiment definition is the basis for systematic and trustworthy experimentation. Aims: As the first step towards guidelines for the definition of experiments, we explore experimentation application types that are conducted in practice. Additionally, we identify experiment definition characteristics that experts regard to considerable contribute to trustworthy experimentation. Method: An expert survey among fifteen industrial experts that published peer-reviewed publications was conducted. Results: In total, we identified fourteen types of applications with 32 concrete applications among the answers of the experts. The most frequently mentioned characteristics regarding the accuracy of an experiment were: success metrics, hypothesis, data quality metrics, guardrail metrics, alerting & shutdown, sizing, and segmentation. Conclusions: There are various applications for experimentation besides the ones mentioned in the literature. Most experts consider only about half of the known characteristics as relevant for the accuracy of an experiment.

Available platforms for online controlled experimentation primarily focus on the technical execution of experiments and are isolated from the remaining software development infrastructure. The platform-independent experimentation infrastructure separates the experiment definition from its execution and focuses on the experimentation process. However, it is still not integrated into the remaining infrastructure. In this paper, we extend the platform-independent experimentation infrastructure about interfaces to ease its integration into the software development infrastructure. The proposed solution is evaluated using a mixed-method research design to assess its usefulness, ease of use, strengths, and weaknesses. The results indicate that the proposed solution represents an adaptable, platform-independent, and cross-domain experimentation infrastructure that is perceived to be easy to use and useful.

This paper presents an envisioned approach to AI-based enhancements of test models in the industrial model-based testing tool TEMPPO Designer. Based on an overview of the tool, we present the required data collector, the AI-based data analyzer and ways to integrate the results into TEMPPO Designer.

Re-architecting monolithic systems with Microservices-based architecture is a common trend. Various companies are migrating to Microservices for different reasons. However, making such an important decision like re-architecting an entire system must be based on real facts and not only on gut feelings. The goal of this work is to propose an evidence-based decision support framework for companies that need to migrate to Microservices, based on the analysis of a set of characteristics and metrics they should collect before re-architecting their monolithic system. We conducted a survey done in the form of interviews with professionals to derive the assessment framework based on Grounded Theory. We identified a set consisting of information and metrics that companies can use to decide whether to migrate to Microservices or not. The proposed assessment framework, based on the aforementioned metrics, could be useful for companies if they need to migrate to Microservices and do not want to run the risk of failing to consider some important information.

Continuous experimentation and A/B testing is an established industry practice that has been researched for more than 10 years. Our aim is to synthesize the conducted research. We wanted to find the core constituents of a framework for continuous experimentation and the solutions that are applied within the field. Finally, we were interested in the challenges and benefits reported of continuous experimentation. We applied forward snowballing on a known set of papers and identified a total of 128 relevant papers. Based on this set of papers we performed two qualitative narrative syntheses and a thematic synthesis to answer the research questions. The framework constituents for continuous experimentation include experimentation processes as well as supportive technical and organizational infrastructure. The solutions found in the literature were synthesized to nine themes, e.g. experiment design, automated experiments, or metric specification. Concerning the challenges of continuous experimentation, the analysis identified cultural, organizational, business, technical, statistical, ethical, and domain-specific challenges. Further, the study concludes that the benefits of experimentation are mostly implicit in the studies. The research on continuous experimentation has yielded a large body of knowledge on experimentation. The synthesis of published research presented within include recommended infrastructure and experimentation process models, guidelines to mitigate the identified challenges, and what problems the various published solutions solve.

The number of connected devices increases continually. The Internet is omnipresent in nearly all areas of daily life, both in private and professional spheres. Humans and institutions are especially concerned about security and usability of the large number of connected objects. This is also a big challenge for professionals in the field of software and hardware testing. They are forced to extend their knowledge to provide quality assurance of connected devices and environments. The qualification requirements are constantly changing. For this reason, the virtual simulation game IoTCityLab will be developed. The target audience are both experienced software testing professionals as also students at universities and vocational trainees who plan a career in the Internet of Things domain. It is a multi-player game in which teams act as in real projects while they will gain experience and knowledge through application. The simulation game is composed of several modules on several approaches of Internet of Things testing. In the first game module the players need to set up a test automation environment for testing an autonomous vehicle in an agile team. In each individual game module the focus is on testing of elements which are part of a smart city infrastructure. Each individual game module focuses on a particular stage of development. By playing the game modules, the players have the opportunity to accompany the proceedings in the implementation of a smart city infrastructure and experience testing and quality assurance in this field interactively.

We want to automate priority assessment of software defects. To do so we provide a tool which uses an explainability-driven framework and classical machine learning algorithms to keep the decisions transparent. Differing from other approaches we only use objective and categorical fields from the bug tracking system as features. This makes our approach lightweight and extremely fast. We perform binary classification with priority labels corresponding to deadlines. Additionally, we evaluate the tool on real data to ensure good performance in the practical use case.

[Background] Recent investigations into the effects of Test-Driven Development (TDD) have been contradictory and inconclusive. This hinders development teams to use research results as the basis for deciding whether and how to apply TDD. [Aim] To support researchers when designing a new study and to increase the applicability of TDD research in the decision-making process in industrial context, we aim at identifying the reasons behind the inconclusive research results in TDD. [Method] We studied the state of the art in TDD research published in top venues in the past decade, and analyzed the way these studies were set up. [Results] We identified five categories of factors that directly impact the outcome of studies on TDD. [Conclusions] This work can help researchers to conduct more reliable studies, and inform practitioners of risks they need to consider when consulting research on TDD.

The definition of an experiment is a fundamental artifact of continuous experiments. It is used to formalize ideas as a plan of action and to document attained knowledge. Although there is a large body of research on continuous experimentation, the definition of experiments is not researched sufficiently. In this paper, we present a taxonomy of characteristics of continuous experiment definitions. The taxonomy is synthesized from a systematic literature review and a tool review of experimentation platforms. Our findings show amongst others that there are 17 characteristics of experiment definitions. Moreover, that platforms primarily focus on the execution of experiments and that experimentation is more than the execution of an experiment.

Retrieving and mining practitioners’ self-reports of their professional experience of software practice could provide valuable evidence for research. The authors are, however, unaware of any existing reviews of research conducted in this area. The authors reviewed and classified previous research, and identified insights into the challenges research confronts when retrieving and mining practitioners’ self-reports of their experience of software practice. They conducted an exploratory review to identify and classify 42 studies. They analysed a selection of those studies for insights on challenges to mining professional experience. They identified only one directly relevant study. Even then this study concerns the software professional's emotional experiences rather than the professional's reporting of behaviour and events occurring during software practice. They discussed the challenges concerning: the prevalence of professional experience; definitions, models and theories; the sparseness of data; units of discourse analysis; annotator agreement; evaluation of the performance of algorithms; and the lack of replications. No directly relevant prior research appears to have been conducted in this area. They discussed the value of reporting negative results in secondary studies. There are a range of research opportunities but also considerable challenges. They formulated a set of guiding questions for further research in this area.

Abstract Context Software testing is an important and costly software engineering activity in the industry. Despite the efforts of the software testing research community in the last several decades, various studies show that still many practitioners in the industry report challenges in their software testing tasks. Objective To shed light on industry's challenges in software testing, we characterize and synthesize the challenges reported by practitioners. Such concrete challenges can then be used for a variety of purposes, eg, research collaborations between industry and academia. Method Our empirical research method is opinion survey. By designing an online survey, we solicited practitioners' opinions about their challenges in different testing activities. Our dataset includes data from 72 practitioners from eight different countries. Results Our results show that test management and test automation are considered the most challenging among all testing activities by practitioners. Our results also include a set of 104 concrete challenges in software testing that may need further investigations by the research community. Conclusion We conclude that the focal points of industrial work and academic research in software testing differ. Furthermore, the paper at hand provides valuable insights concerning practitioners' “pain” points and, thus, provides researchers with a source of important research topics of high practical relevance.

Abstract Predictive analysis in business process monitoring aims at forecasting the future information of a running business process. The prediction is typically made based on the model extracted from historical process execution logs (event logs). In practice, different business domains might require different kinds of predictions. Hence, it is important to have a means for properly specifying the desired prediction tasks, and a mechanism to deal with these various prediction tasks. Although there have been many studies in this area, they mostly focus on a specific prediction task. This work introduces a language for specifying the desired prediction tasks, and this language allows us to express various kinds of prediction tasks. This work also presents a mechanism for automatically creating the corresponding prediction model based on the given specification. Differently from previous studies, instead of focusing on a particular prediction task, we present an approach to deal with various prediction tasks based on the given specification of the desired prediction tasks. We also provide an implementation of the approach which is used to conduct experiments using real-life event logs.

Over the past few decades, the automotive industry was mostly focused on testing the safety aspects of a vehicle. However, this was not the case with security testing as it only began to be addressed recently. As a result, multiple approaches applying various security testing techniques on different software-based vehicle IT components emerged. With that said, the research and practice lack an overview about these techniques. In this paper, we conduct a systematic mapping study. This involved the investigation on the following five dimensions: (1) security testing techniques, (2) AUTOSAR layers, (3) functional interfaces of AUTOSAR, (4) vehicle lifecycle phases and (5) attacks. In total, 39 papers presenting approaches for security testing in automotive engineering were systematically selected and classified. The results identify multiple security testing techniques focusing on early phases of vehicle life cycle through the application and services layer of the AUTOSAR architecture. Finally, there is a need for security regression testing approaches, as well as combined security and safety testing approaches.

The ever-increasing amount, variety as well as generation and processing speed of today's data pose a variety of new challenges for developing Data-Intensive Software Systems (DISS). As with developing other kinds of software systems, developing DISS is often done under severe pressure and strict schedules. Thus, developers of DISS often have to make technical compromises to meet business concerns. This position paper proposes a conceptual model that outlines where Technical Debt (TD) can emerge and proliferate within such data-centric systems by separating a DISS into three parts (Software Systems, Data Storage Systems and Data). Further, the paper illustrates the proliferation of Database Schema Smells as TD items within a relational database-centric software system based on two examples.

Research collaboration between industry and academia supports improvement and innovation in industry and helps ensure the industrial relevance of academic research. However, many researchers and practitioners in the community believe that the level of joint industry-academia collaboration (IAC) projects in Software Engineering (SE) research is relatively low, creating a barrier between research and practice. The goal of the empirical study reported in this paper is to explore and characterize the state of IAC with respect to industrial needs, developed solutions, impacts of the projects and also a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review (SLR) study. To address the above goal, we conducted an opinion survey among researchers and practitioners with respect to their experience in IAC. Our dataset includes 101 data points from IAC projects conducted in 21 different countries. Our findings include: (1) the most popular topics of the IAC projects, in the dataset, are: software testing, quality, process, and project managements; (2) over 90% of IAC projects result in at least one publication; (3) almost 50% of IACs are initiated by industry, busting the myth that industry tends to avoid IACs; and (4) 61% of the IAC projects report having a positive impact on their industrial context, while 31% report no noticeable impacts or were “not sure”. To improve this situation, we present evidence-based recommendations to increase the success of IAC projects, such as the importance of testing pilot solutions before using them in industry. This study aims to contribute to the body of evidence in the area of IAC, and benefit researchers and practitioners. Using the data and evidence presented in this paper, they can conduct more successful IAC projects in SE by being aware of the challenges and how to overcome them, by applying best practices (patterns), and by preventing anti-patterns.

UML activity diagrams and state machines are both used for modeling system behavior from the user perspective and are frequently the basis for deriving system test cases. In practice, system test cases are often derived manually from UML activity diagrams or state machines. For this task, comprehensibility of respective models is essential and a relevant question for practice to support model selection and design, as well as subsequent test derivation. Therefore, the objective of this paper is to compare the comprehensibility of UML activity diagrams and state machines during manual test case derivation. We investigate the comprehensibility of UML activity diagrams and state machines in a controlled student experiment. Three measures for comprehensibility have been investigated: (1) the self-assessed comprehensibility, (2) the actual comprehensibility measured by the correctness of answers to comprehensibility questions, and (3) the number of errors made during test case derivation. The experiment was performed and internally replicated with overall 84 participants divided into three groups at two institutions. Our experiment indicates that activity diagrams are more comprehensible but also more error-prone with regard to manual test case derivation and discusses how these results can improve system modeling and test case design.

As our society becomes more and more dependent on IT systems, failures of these systems can harm more and more people and organizations. Diligently performing risk and hazard analysis helps to minimize the potential harm of IT system failures on the society and increases the probability of their undisturbed operation. Risk and hazard analysis is an important activity for the development and operation of critical software intensive systems, but the increased complexity and size puts additional requirements on the effectiveness of risk and hazard analysis methods. This paper presents a qualitative comparison of two hazard analysis methods, failure mode and effect analysis (FMEA) and system theoretic process analysis (STPA), using case study research methodology. Both methods have been applied on the same forward collision avoidance system to compare the effectiveness of the methods and to investigate what are the main differences between them. Furthermore, this study also evaluates the analysis process of both methods by using a qualitative criteria derived from the technology acceptance model (TAM). The results of the FMEA analysis were compared to the results of the STPA analysis, which were presented in a previous study. Both analyses were conducted on the same forward collision avoidance system. The comparison shows that FMEA and STPA deliver similar analysis results.

This article introduces a new method for knowledge-based security testing by logic programming and the related tool implementation for model-based non-functional security testing of web applications. Our method helps to overcome the current prevalent focus on functional instead of non-functional (or negative) requirements as well as the required high level of security knowledge when performing non-functional security testing. It addresses issues like considering non-functional requirements for testing, managing the virtually infinite amount of negative security test cases, advancing non-functional security testing away from its prevalent penetration testing-like style, and making non-functional security testing feasible for testers that are not experts in security via a security knowledge base. The method and its model-based tool implementation are evaluated in two studies, which show the method’s effectiveness in detecting vulnerabilities in web applications and thus, also its value in making software system more secure.

Requirements Engineering (RE) has established itself as a software engineering discipline over the past decades. While researchers have been investigating the RE discipline with a plethora of empirical studies, attempts to systematically derive an empirical theory in context of the RE discipline have just recently been started. However, such a theory is needed if we are to define and motivate guidance in performing high quality RE research and practice. We aim at providing an empirical and externally valid foundation for a theory of RE practice, which helps software engineers establish effective and efficient RE processes in a problem-driven manner. We designed a survey instrument and an engineer-focused theory that was first piloted in Germany and, after making substantial modifications, has now been replicated in 10 countries worldwide. We have a theory in the form of a set of propositions inferred from our experiences and available studies, as well as the results from our pilot study in Germany. We evaluate the propositions with bootstrapped confidence intervals and derive potential explanations for the propositions. In this article, we report on the design of the family of surveys, its underlying theory, and the full results obtained from the replication studies conducted in 10 countries with participants from 228 organisations. Our results represent a substantial step forward towards developing an empirical theory of RE practice. The results reveal, for example, that there are no strong differences between organisations in different countries and regions, that interviews, facilitated meetings and prototyping are the most used elicitation techniques, that requirements are often documented textually, that traces between requirements and code or design documents are common, that requirements specifications themselves are rarely changed and that requirements engineering (process) improvement endeavours are mostly internally driven. Our study establishes a theory that can be used as starting point for many further studies for more detailed investigations. Practitioners can use the results as theory-supported guidance on selecting suitable RE methods and techniques.

Domain-specific languages (DSLs) are mainly designed ad-hoc and gut feeling resulting in languages that are often not well suited for their users and engineers. In this paper we develop a process for evidence-based language engineering to design domain-specific languages based on empirical evidence to support decision in language engineering. The developed process comprises an iterative execution of the phases DSL engineering, issue identification, data collection and evidence appraisal. We exemplify the concept by designing a DSL for Gherkin, a language test-driven acceptance testing in Xtext. The required evidence is derived by mining and analyzing all GitHub projects until July 1, 2017 that apply Gherkin.

The systematic evaluation of ideas by experiments are the foundation of continuous experimentation. It allows to assess the value of an idea, remove guessing and subjective opinions from the discussion. The enormous interest of it by practitioners and researchers let the body of knowledge consistently grow. New framework, methods and techniques are developed and its application is constantly expanded to new fields like cyber-physical systems or social networks. In this paper we present a systematic mapping study to characterize the current state of research on continuous experimentation. Our study analyzes the following aspects: intensity of research activity and industry-academia collaboration, influential authors and publications, frequent research types and topics, kind of contributions and terms used for continuous experimentation. Our findings show amongst others that the intensity of research activities increases consistently, the collaboration between industry and academia is high and the most cited publications are experience reports from practitioners.

In a previous empirical study by Bavota et al., multiple releases of three open-source systems reported the extent to which refactorings induced defect-fixes. In a much earlier study, van Deursen and Moonen (vD&M) provided a test taxonomy in which Fowler's seventy-two refactorings were categorized according to the post refactoring test burden of each (i.e., the changes required to unit tests after each refactoring had been undertaken). A refactoring was categorized as 'Type B' if it required no change to the original tests and 'Type E' if significant changes were necessary. In this paper, we investigate nine refactorings spread across vD&M's taxonomy and the corresponding defect-fix data provided by Bavota et al., to explore the relationship between defect-fixes due to refactoring and vD&M's taxonomy. Results showed that, in contrast to our intuition, the most defect-fix prone refactorings were of Types C and D and not, as we thought, of Type E. The 'Extract method' refactoring stood out as particularly 'defect-fix' inducing, suggesting that while it may solve one problem (i.e., in decomposing an excessively long method), it may well introduce other problems and required defect-fixes as a by-product.

Risk-based testing is a frequently used testing approach which utilizes identified risks of a software system to provide decision support in all phases of the testing process. Risk assessment, which is a core activity of every risk-based testing process, is often done in an ad hoc manual way. Software quality assessments, based on quality models, already describe the product-related risks of a whole software product and provide objective and automation-supported assessments. But so far, quality models have not been applied for risk assessment and risk-based testing in a systematic way. This article tries to fill this gap and investigates how the information and data of a quality assessment based on the open quality model QuaMoCo can be integrated into risk-based testing. We first present two generic approaches showing how quality assessments based on quality models can be integrated into risk-based testing and then provide the concrete integration on the basis of the open quality model QuaMoCo. Based on five open source products, a case study is performed. Results of the case study show that a risk-based testing strategy outperforms a lines of code-based testing strategy with regard to the number of defects detected. Moreover, a significant positive relationship between the risk coefficient and the associated number of defects was found.

To systematically collect evidence and to structure a given area in software engineering (SE), Systematic Literature Reviews (SLR) and Systematic Mapping (SM) studies have become common. Data extraction is one of the main phases (activities) when conducting an SM or an SLR, whose objective is to extract required data from the primary studies and to accurately record the information researchers need to answer the questions of the SM/SLR study. Based on experience in a large number of SM/SLR studies, we and many other researchers have found the data extraction in SLRs to be time consuming and error-prone, thus raising the real need for heuristics and guidelines for effective and efficient data extraction in these studies, especially to be learnt by junior and young researchers. As a 'guideline' paper, this paper contributes a synthesized list of challenges usually faced during SLRs' data extraction phase and the corresponding solutions (guidelines). For our synthesis, we consider two data sources: (1) the pool of 16 SLR studies in which the authors have been involved in, as well as (2) a review of challenges and guidelines in the existing literature. Our experience in utilizing the presented guidelines in the near past have helped our junior colleagues to conduct data extractions more effectively and efficiently.

Research collaboration between industry and academia supports improvement and innovation in industry and helps to ensure industrial relevance in academic research. However, many researchers and practitioners believe that the level of joint industry-academia collaboration (IAC) in software engineering (SE) research is still relatively low, compared to the amount of activity in each of the two communities. The goal of the empirical study reported in this paper is to exploratory characterize the state of IAC with respect to a set of challenges, patterns and anti-patterns identified by a recent Systematic Literature Review study. To address the above goal, we gathered the opinions of researchers and practitioners w.r.t. their experiences in IAC projects. Our dataset includes 47 opinion data points related to a large set of projects conducted in 10 different countries. We aim to contribute to the body of evidence in the area of IAC, for the benefit of researchers and practitioners in conducting future successful IAC projects in SE. As an output, the study presents a set of empirical findings and evidence-based recommendations to increase the success of IAC projects.

The level of industry-academia collaboration (IAC) in software engineering in general and in software testing in particular is quite low. Many researchers and practitioners are not collaborating with the "other side" to solve industrial problems. To shed light on the above issue and to characterize precisely what industry wants from academia in software testing, we solicited practitioners' opinions on their challenges in different testing activities and also the particularly relevant topics that they want the research community to work on. This short paper aims to draw the community's attention to the important issue of strengthening IAC with the hope of more IAC in software testing in the areas of most importance to the industry.

Testing is a fundamental task to ensure software quality. Regression testing aims to ensure that changes to software do not introduce new failures. As resources are often limited and testing comprises a vast amount of test cases, different regression strategies have been proposed to reduce testing effort by selecting or prioritizing important test cases, e.g., code coverage (to ensure a sufficient testing depth). However, in system testing, source code is often not available creating a black-box system. In this paper, we introduce an automated, multi-objective test case selection technique in black-box systems using genetic algorithms. We define seven different objectives, based on meta-data, allowing a flexible test case selection for a variety of systems. For evaluation, we apply our technique on two different subject systems assessing the feasibility and suitability of our test case selection approach. Results indicate that our approach is applicable based on different data available and is able to outperform random test case selection and retest-all.

Software and system development faces numerous challenges of rapidly changing markets. To address such challenges, companies and projects design and adopt specific development approaches by combining well-structured comprehensive methods and flexible agile practices. Yet, the number of methods and practices is large, and available studies argue that the actual process composition is carried out in a fairly ad-hoc manner. The present paper reports on a survey on hybrid software development approaches. We study which approaches are used in practice, how different approaches are combined, and what contextual factors influence the use and combination of hybrid software development approaches. Our results from 69 study participants show a variety of development approaches used and combined in practice. We show that most combinations follow a pattern in which a traditional process model serves as framework in which several fine-grained (agile) practices are plugged in. We further show that hybrid software development approaches are independent from the company size and external triggers. We conclude that such approaches are the results of a natural process evolution, which is mainly driven by experience, learning, and pragmatism.

Security testing can broadly be described as (1) the testing of security requirements that concerns confidentiality, integrity, availability, authentication, authorization, nonrepudiation and (2) the testing of the software to validate how much it can withstand an attack. Agile testing involves immediately integrating changes into the main system, continuously testing all changes and updating test cases to be able to run a regression test at any time to verify that changes have not broken existing functionality. Software companies have a challenge to systematically apply security testing in their processes nowadays. There is a lack of guidelines in practice as well as empirical studies in real-world projects on agile security testing; industry in general needs a more systematic approach to security. The findings of this research are not surprising, but at the same time are alarming. The lack of knowledge on security by agile teams in general, the large dependency on incidental pen-testers, and the ignorance in static testing for security are indicators that security testing is highly under addressed and that more efforts should be addressed to security testing in agile teams.

Systematic Literature Reviews (SLR) may not provide insight into the "state of the practice" in SE, as they do not typically include the "grey" (non-published) literature. A Multivocal Literature Review (MLR) is a form of a SLR which includes grey literature in addition to the published (formal) literature. Only a few MLRs have been published in SE so far. We aim at raising the awareness for MLRs in SE by addressing two research questions (RQs): (1) What types of knowledge are missed when a SLR does not include the multivocal literature in a SE field? and (2) What do we, as a community, gain when we include the multivocal literature and conduct MLRs? To answer these RQs, we sample a few example SLRs and MLRs and identify the missing and the gained knowledge due to excluding or including the grey literature. We find that (1) grey literature can give substantial benefits in certain areas of SE, and that (2) the inclusion of grey literature brings forward certain challenges as evidence in them is often experience and opinion based. Given these conflicting viewpoints, the authors are planning to prepare systematic guidelines for performing MLRs in SE.

Domain specific languages (DSLs) are widely used in practice and investigated in software engineering research. But so far, language workbenches do not provide sufficient built-in decision support for language design and improvement. Controlled experiments have the potential to provide appropriate, data-driven decision support for language engineers and researchers to compare different language features with evidence-based feedback. This paper provides an integrated end-to-end tool environment to perform controlled experiments in DSL engineering. The experiment environment is built on the basis and integrated into the language workbench Meta Programming System (MPS). The environment not only supports language design but also all steps of experimentation, i.e., planning, operation, analysis & interpretation, as well as presentation & package. The tool environment is presented by means of a running example experiment comparing the time taken to create system acceptance tests for web applications in two different DSLs.

[Context] Problems in Requirements Engineering (RE) can lead to serious consequences during the software development lifecycle. [Goal] The goal of this paper is to propose empirically-based guidelines that can be used by different types of organisations according to their size (small, medium or large) and process model (agile or plan-driven) to help them in preventing such problems. [Method] We analysed data from a survey on RE problems answered by 228 organisations in 10 different countries. [Results] We identified the most critical RE problems, their causes and mitigation actions, organizing this information by clusters of size and process model. Finally, we analysed the causes and mitigation actions of the critical problems of each cluster to get further insights into how to prevent them. [Conclusions] Based on our results, we suggest preliminary guidelines for preventing critical RE problems in response to context characteristics of the companies.

Defect prediction is a powerful method that provides information about the likely defective parts in software system and is applicable to improve effectiveness and efficiency of software quality assurance. This makes defect prediction a perfect candidate to be combined with risk-based testing to optimally guide testing activities towards risky parts of software. As a first step towards a successful combination, this paper presents requirements that have to be fulfilled for enabling the synergies between defect prediction and risk-based testing.

Security testing has become a critical quality assurance technique to provide a sufficient degree of security. However, it is regarded to be too complex to be performed by system testers, who are non-experts in security. This paper provides two approaches to testing security requirements, one based on a Failure Modes, Vulnerabilities and Effect Analysis (FMVEA) and the other based on misuse cases, both suitable for testers who have domain knowledge but are not security experts. We perform a controlled experiment to empirically compare the two testing approaches based on the quality of the derived test cases. The results of the experiment show that the use of attack patterns in the misuse-case-based approach delivers test cases with a better alignment between requirements and security test cases as well as a higher amount of correct test cases.

Model-based security testing relies on models to test whether a software system meets its security requirements. It is an active research field of high relevance for industrial applications, with many approaches and notable results published in recent years. This article provides a taxonomy for model-based security testing approaches. It comprises filter criteria (i.e. model of system security, security model of the environment and explicit test selection criteria) as well as evidence criteria (i.e. maturity of evaluated system, evidence measures and evidence level). The taxonomy is based on a comprehensive analysis of existing classification schemes for model-based testing and security testing. To demonstrate its adequacy, 119 publications on model-based security testing are systematically extracted from the five most relevant digital libraries by three researchers and classified according to the defined filter and evidence criteria. On the basis of the classified publications, the article provides an overview of the state of the art in model-based security testing and discusses promising research directions with regard to security properties, coverage criteria and the feasibility and return on investment of model-based security testing. Copyright © 2015 John Wiley & Sons, Ltd.

The RET (Requirements Engineering and Testing) workshop provides a meeting point for researchers and practitioners from the two separate fields of Requirements Engineering (RE) and Testing. The goal is to improve the connection and alignment of these two areas through an exchange of ideas, challenges, practices, experiences and results. The long term aim is to build a community and a body of knowledge within the intersection of RE and Testing. One of the main outputs of the 1st workshop was a collaboratively constructed map of the area of RET showing the topics relevant to RET for these. The 2nd workshop will continue in the same interactive vein and include a keynote, paper presentations with ample time for discussions, and a group exercise. For true impact and relevance this cross-cutting area requires contribution from both RE and Testing, and from both researchers and practitioners. For that reason we welcome a range of paper contributions from short experience papers to full research papers that both clearly cover connections between the two fields.

In this paper we present the results of an empirical case study performed at the European Space Agency (ESA). In this major project, the various challenges for testing were tackled using a model-based approach for test design and the generation of executable test automation scripts. An evaluation of this approach''s efficiency identified significant cost savings and quality improvements.

Context: In modern systems, like cyber-physical systems, where software and physical services are interacting, safety, security or performance play an important role. In order to guarantee the correct interoperability of such systems, with respect to functional and non-functional requirements, integration testing is an effective measure to achieve this. Model-based testing moreover not only enables early definition and validation, but also test automation. This makes it a good choice to overcome urgent challenges of integration testing. Objective: Many publications on model-based integration testing (MBIT) approaches can be found. Nevertheless, a study giving a systematic overview on the underlying software paradigms, measures for guiding the integration testing process as well as non-functional requirements they are suitable for, is missing. The aim of this paper is to find and synthesize the relevant primary studies to gain a comprehensive understanding of the current state of model-based integration testing. Method: For synthesizing the relevant studies, we conducted a systematic literature review (SLR) according to the guidelines of Kitchenham. Results: The systematic search and selection retrieved 83 relevant studies from which data has been extracted. Our review identified three assessment criteria for guiding the testing process, namely static metrics, dynamic metrics and stochastic &random. In addition it shows that just a small fraction considers non-functional requirements. Most approaches are for component-oriented systems. Conclusion: Results from the SLR show that there are two major research gaps. First, there is an accumulated need for approaches in the MBIT field that support non-functional requirements, as they are gaining importance. Second, means for steering the integration testing process, especially together with automation, need to evolve.

Context: Product release decisions are often made ad hoc or not relying on up-to-date information from systematic and analysis-driven process. Often, too much emphasis solely is put into functionality, thereby neglecting the different quality aspects being important for the success of the product. Objective: For a case study project, we quantitatively measure the impact of the analytical and systematic release planning approach Q-EVOLVE II. Method: As an explorative and retrospective case study, we perform optimized planning scenarios and compare them with the actual baseline scenario. Results: For the case study project in an Austrian public health insurance institution, we demonstrate that the analytical and systematic approach Q-EVOLVE II results in a (1) reduced release time, (2) more features implemented in the same time, and (3) better final quality through increased testing activities and less defects slipping through at the time of release. Conclusion: In the context of the studied institution, analytical and systematic planning of product releases considering quality criteria is superior to ad hoc planning ignoring quality.

Background: Determining the factor probability in risk estimation requires detailed knowledge about the software product and the development process. Basing estimates on expert opinion may be a viable approach if no other data is available. Objective: In this paper we analyze initial results from estimating the risk probability based on expert opinion to answer the questions (1) Are expert opinions consistent? (2) Do expert opinions reflect the actual situation? (3) How can the results be improved? Approach: An industry project serves as case for our study. In this project six members provided initial risk estimates for the components of a software system. The resulting estimates are compared to each other to reveal the agreement between experts and they are compared to the actual risk probabilities derived in an ex-post analysis from the released version. Results: We found a moderate agreement between the rations of the individual experts. We found a significant accuracy when compared to the risk probabilities computed from the actual defects. We identified a number of lessons learned useful for improving the simple initial estimation approach applied in the studied project. Conclusions: Risk estimates have successfully been derived from subjective expert opinions. However, additional measures should be applied to triangulate and improve expert estimates.

Quality of requirements is of great importance for the software development lifecycle as it influences all steps of software development. To ensure various quality attributes, suitable requirements validation techniques such as reviews or testing are essential. In this paper, we show how defect taxonomies can improve requirements reviews and testing. We point out how defect taxonomies can be seamlessly integrated into the requirements engineering process and discuss requirements validation with defect taxonomies as well as its benefits and the lessons learned with reference to industrial projects of a public health insurance institution where this approach has been successfully applied.