The integration of Large Language Models (LLMs), such as ChatGPT and GitHub Copilot, into software engineering workflows has shown potential to enhance productivity, particularly in software testing. This paper investigates whether LLM support improves defect detection effectiveness during unit testing. Building on prior studies comparing manual and tool-supported testing, we replicated and extended an experiment where participants wrote unit tests for a Java-based system with seeded defects within a time-boxed session, supported by LLMs. Comparing LLM supported and manual testing, results show that LLM support significantly increases the number of unit tests generated, defect detection rates, and overall testing efficiency. These findings highlight the potential of LLMs to improve testing and defect detection outcomes, providing empirical insights into their practical application in software testing.

Abstract Context The readability of source code is key for understanding and maintaining software systems and tests. Although several studies investigate the readability of source code, there is limited research specifically on the readability of test code and related influence factors. Objective In this paper, we aim at investigating the factors that influence the readability of test code from an academic perspective based on scientific literature sources and complemented by practical views, as discussed in grey literature. Methods First, we perform a Systematic Mapping Study (SMS) with a focus on scientific literature. Second, we extend this study by reviewing grey literature sources for practical aspects on test code readability and understandability. Finally, we conduct a controlled experiment on the readability of a selected set of test cases to collect additional knowledge on influence factors discussed in practice. Results The result set of the SMS includes 19 primary studies from the scientific literature for further analysis. The grey literature search reveals 62 sources for information on test code readability. Based on an analysis of these sources, we identified a combined set of 14 factors that influence the readability of test code. 7 of these factors were found in scientific and grey literature, while some factors were mainly discussed in academia (2) or industry (5) with only limited overlap. The controlled experiment on practically relevant influence factors showed that the investigated factors have a significant impact on readability for half of the selected test cases. Conclusion Our review of scientific and grey literature showed that test code readability is of interest for academia and industry with a consensus on key influence factors. However, we also found factors only discussed by practitioners. For some of these factors we were able to confirm an impact on readability in a first experiment. Therefore, we see the need to bring together academic and industry viewpoints to achieve a common view on the readability of software test code.

Fast changes in production require flexibility regarding products, production processes, and production resources (PPR), moving towards a flexible production system that consists of independent systems. However, production systems engineering traditionally considers mainly static systems and requires advanced capabilities to effectively and efficiently validate performance indicators specified in test scenarios for a flexible production system. For validating production process and system variants, this vision paper explores the Test-Driven Performance Validation (TPV) approach for knowledge representation of (1) a family of production variants (2) with test scenarios on production effects and conditions, and (3) a Production Asset Network that defines PPR configuration dependencies and production data sources as a foundation for calculating a production variant’s performance. We conclude with a research agenda towards coordinating performance validation of flexible production with human and machine agents.

The IEC 62061:2021 standard requires production owners to ensure both functional safety and information security for their industrial applications. Unfortunately, traditional models of functional safety and information security have been designed in isolation and are difficult to combine. This paper introduces the Safety & Security Combination (SafeSecCombi) approach to combine models for functional safety and security concerns in automating digital production. SafeSecCombi (i) validates causes for desired and undesired effects regarding safety in an industrial production process by linking these causes to products, production processes, and production resources; (ii) identifies Industrial Internet of Things (IIoT) assets that can cause unsafe behavior in case of a successful security attack; and (iii) analyzes risks of security attacks to these IIoT assets. Therefore, SafeSecCombi provides a model for the combined analysis of safety and security concerns regarding a Cyber-Physical Production System (CPPS). In a feasibility study on an industrial work cell for metal processing with a collaborative robot, we evaluated the effectiveness and efficiency of the SafeSecCombi approach. Results indicate that the SafeSecCombi approach is feasible and effective, and provides safety and security experts with actionable, context-specific causes for security-related safety issues and countermeasures that are well grounded in engineering models, as a foundation to address the IEC 62061:2021 requirements.

In Production System Engineering (PSE), domain experts from different disciplines reuse assets such as products, production processes, and resources. Therefore, PSE organizations aim at establishing reuse across engineering disciplines. However, the coordination of multi-disciplinary reuse tasks, e.g., the re-validation of related assets after changes, is hampered by the coarse-grained representation of tasks and by scattered, heterogeneous domain knowledge. This paper introduces the Multi-disciplinary Reuse Coordination (MRC) artifact to improve task management for multi-disciplinary reuse. For assets and their properties, the MRC artifact describes sub-tasks with progress and result states to provide references for detailed reuse task management across engineering disciplines. In a feasibility study on a typical robot cell in automotive manufacturing, we investigate the effectiveness of task management with the MRC artifact compared to traditional approaches. Results indicate that the MRC artifact is feasible and provides effective capabilities for coordinating multi-disciplinary re-validation after changes.

In context of Industry 4.0, the engineering of Cyber-Physical Production Systems (CPPSs) need to incorporate a heterogeneous set of engineering disciplines, data models, and artefacts. The quality of related data models and engineering artefacts is success-critical for the engineering process and the planned CPPS. Software and System tests aim at improving the quality of a CPPS. However, in CPPS, risk cases are often unknown and insufficiently covered by systematic testing methods, especially in heterogeneous environments. In this paper, we describe a multi-view test specification (MVTS) approach based on a risk analysis to systematically derive regular and negative/error test cases in CPPS engineering. We build on the PPR Asset Network (PAN) that provides the structure of a CPPS from product, process, resource perspective and their dependencies, and the Failure Mode and Effect Analysis (FMEA) to efficiently identify risks in CPPS engineering. We conceptually evaluate the MVTS approach with domain experts in a feasibility study to show benefits and limitations in context of traditional software testing. First results showed benefits of the MVTS approach with the help of the PAN and FMEA to systematically capture risks and derive test cases. While the execution of test cases is often limited to the regular systems behavior, negative test are often not executed because of possible physical damages. However, negative test cases can raise the awareness of possible critical risks during CPPS planning and design.

The readability of software code is a key success criterion for understanding and maintaining software systems and tests. In industry practice, a limited number of guidelines aim for improving and assessing the readability of software (test) code. Although several studies focus on investigating the readability of software code, we observed limited research work that focuses on the readability of software test code. In this paper we focus on systematically investigating the characteristics, factors, and assessment criteria that have an impact on the readability of test code. We build on a Systematic Mapping Study (SMS) to identify key characteristics, factors, and assessment criteria that have an impact on test code readability, legibility, and understandability to support and improve maintenance tasks. The result set includes 16 studies for further analysis. The majority of publications focuses on readability investigations of automatically generated test code (88%), often evaluated with surveys to access the readability of test code (44 %). Although several approaches aim at assessing the readability with focus on isolated factors, a combination of different readability aspects within an assessment framework can help to better assess and justify the readability of test code with focus on improving software and system test maintenance.

Together with many success stories, promises such as the increase in production speed and the improvement in stakeholders’ collaboration have contributed to making agile a transformation in the software industry in which many companies want to take part. However, driven either by a natural and expected evolution or by contextual factors that challenge the adoption of agile methods as prescribed by their creator(s), software processes in practice mutate into hybrids over time. Are these still agile? In this article, we investigate the question: what makes a software development method agile? We present an empirical study grounded in a large-scale international survey that aims to identify software development methods and practices that improve or tame agility. Based on 556 data points, we analyze the perceived degree of agility in the implementation of standard project disciplines and its relation to used development methods and practices. Our findings suggest that only a small number of participants operate their projects in a purely traditional or agile manner (under 15 percent). That said, most project disciplines and most practices show a clear trend towards increasing degrees of agility. Compared to the methods used to develop software, the selection of practices has a stronger effect on the degree of agility of a given discipline. Finally, there are no methods or practices that explicitly guarantee or prevent agility. We conclude that agility cannot be defined solely at the process level. Additional factors need to be taken into account when trying to implement or improve agility in a software company. Finally, we discuss the field of software process-related research in the light of our findings and present a roadmap for future research.

Security professionals rely extensively on log data to monitor IT infrastructures and investigate potentially malicious activities. Existing systems support these tasks by collecting log messages in a database, from where log events can be queried and correlated. Such centralized approaches are typically based on a relational model and store log messages as plain text, which offers limited flexibility for the representation of heterogeneous log events and the connections between them. A knowledge graph representation can overcome such limitations and enable graph pattern-based log analysis, leveraging semantic relationships between objects that appear in heterogeneous log streams. In this paper, we present a method to dynamically construct such log knowledge graphs at query time, i.e., without a priori parsing, aggregation, processing, and materialization of log data. Specifically, we propose a method that – for a given query formulated in SPARQL – dynamically constructs a virtual log knowledge graph directly from heterogeneous raw log files across multiple hosts and contextualizes the result with internal and external background knowledge. We evaluate the approach across multiple heterogeneous log sources and machines and see encouraging results that indicate that the approach is viable and facilitates ad-hoc graph-analytic queries in federated settings.

In Cyber-Physical Production System (CPPS) engineering, domain experts design Product-Process-Resource (PPR) assets, defined in shared engineering artifacts, like system plans and tool data. Configuration management of assets should provide a consistent integration of these stakeholder views as a basis for risk and quality management. However, without a suitable data structure, collecting and analyzing the stakeholder views on an asset, scattered over changing engineering artifacts, is often incomplete and time-consuming. In this paper, we introduce the PPR Asset Directory, a conceptual model based on the standards Industry 4.0 Asset Administration Shell and the AutomationML Component, for representing, integrating, and efficiently analyzing (i) multiple stakeholders views in engineering artifacts that define an asset; (ii) dependencies between stakeholder views on an asset; and (iii) dependencies between assets. A PPR Asset Directory for a robot cell in automotive manufacturing shows the feasibility of efficient configuration management queries.

Industry 4.0 envisions adaptive production systems, i.e., Cyber-Physical Production Systems (CPPSs), to manufacture products from a product line. Product-Process-Resource modeling represents the essential aspects of a CPPS. However, due to discipline-specific models, e.g., mechanical, electrical, and automation models, it is often unclear how to integrate the proprietary data into an integrated model due to missing common understanding. This paper investigates (i) how to integrate local engineering views with Common Concepts (CCs) and using them as a defined taxonomy for modeling a network of engineering concepts; (ii) how to build an engineering network graph for visualisation and analysis considering discipline-specific needs. We motivate a method to support CPPS engineering organisations to integrate their heterogeneous data using CCs. This builds the basis for defining multi-domain engineering graphs for visualisation and analysis aspects. In this paper, we present a research agenda discussing open issues and expected results.

The increasing availability of data in Smart Manufacturing opens new challenges and required capabilities in the area of big data in industry and academia. Various organizations have started initiatives to collect and analyse data in their individual contexts with specific goals, e.g., for monitoring, optimization, or decision support in order to reduce risks and costs in their manufacturing systems. However, the variety of available application areas require to focus on most promising activities. Therefore, we see the need for investigating common challenges and priorities in academia and industry from expert and management perspective to identify the state of the practice and promising application areas for driving future research directions. The goal of this paper is to report on an industry-academia survey to capture the current state of the art, required capabilities and priorities in the area of big data applications. Therefore, we conducted a survey in winter 2020/21 in industry and academia. We received 22 responses from different application domains highlighting the need for supporting (a) fault detection and (b) fault classification based on (c) historical and (d) real-time data analysis concepts. Therefore, the survey results reveals current and upcoming challenges in big data applications, such as defect handling based on historical and real-time data.

In the engineering of Cyber-Physical Production Systems (CPPSs), the coordination and data exchange of Product (P), Process (P‘) and Resource (R) assets is success-critical for creating high-quality engineering products. In industry, product, process, and resources are often addressed individually without explicitly expressing their dependencies. Therefore, isolated assets can hinder efficient collaboration within CPPS engineering projects that can lead to risks in case of overseen dependencies. Thus, we see the need for explicitly expressing PPR assets within an PPR Asset Network (PAN) that is (a) capable of handling assets from different viewpoints and (b) can enable efficient added value application such as risk, requirements, and configuration management. The goal of this paper include a process description for the elicitation of the PAN and to illustrate added-value applications based on a selected use case, i.e., the Industry 4.0 Testbed at CTU in Prague. We build on the PPR concept as foundation for the PAN and for added-value applications. PAN added-value applications aim at supporting risk management, requirements engineering, or configuration management by focusing on PPR assets and dependencies in context of CPPS engineering. Although PAN provides a valuable foundation for added-value applications there is the need for initial effort for the creation/generating the PAN.

In Production Systems Engineering (PSE), domain experts aim at reusing production processes implemented as Industry 4.0 assets and software.However, the knowledge on reusable assets is often scattered on multi-disciplinary engineering artifacts and domain experts, making it hard to find suitable reusable assets and map them to requirements.In this paper, we (i) identify challenges and requirements for reuse in PSE based on a domain analysis; (ii) introduce the Industry 4.0 Asset Network (I4AN) that integrates multi-disciplinary dependencies between the assets and exposes recurring patterns; and (iii) present four patterns for reuse in PSE that aim at improving reuse efficiency and risk.We evaluate the I4AN with reuse scenarios in a feasibility study.The study results indicate that the I4AN model satisfies the elicited requirements and enables PSE domain experts to identify patterns for reuse in their contexts.

In Production Systems Engineering (PSE), domain experts aim at reusing partial system designs implemented as Industry 4.0 assets and software. However, the knowledge on assets is often scattered across engineering artifacts from multiple disciplines and domain experts, making it difficult to find reusable assets and map them to requirements. In this paper, we (i) identify challenges and requirements for the representation of reuse knowledge in PSE, based on the results of a domain analysis in automotive manufacturing; (ii) refine the Industry 4.0 Asset Network (I4AN) meta-model that integrates multi-disciplinary dependencies between the assets; (iii) introduce the I4AN reference model that exposes recurring patterns; and (iv) present basic and applied patterns for reuse in PSE that aim at improving reuse efficiency and lowering risks. We evaluate the I4AN reference model and patterns with reuse scenarios in a feasibility study in automotive manufacturing. The study results indicate that the I4AN reference model and patterns satisfy the elicited requirements and enable PSE domain experts to identify patterns for reuse and sufficiently complete sets of reusable assets in their contexts.

Production Systems Engineering (PSE) networks create engineering results represented by interdependent engineering data models. A key aspect of engineering result quality is the consistency achieved in and among the discipline-specific data models. However, the heterogeneity of discipline-specific data models makes it hard to represent and evaluate consistency across engineering disciplines. In this paper, we introduce PSE-MECon - a method for Modelling and Evaluating Consistency rules in PSE environments across disciplines. We describe a meta-model for representing consistency rules in a multi-disciplinary data model and demonstrate the viability of the meta-model by representing dependencies in AutomationML as a foundation for consistency checks for engineering data logistics in multi-disciplinary engineering networks.

In Cyber-Physical Production System (CPPS) engineering for discrete manufacturing, the definition of test cases is vital to ensure correct behavior of production processes and to test risky cases. Unfortunately, the definition of test cases requires know-how both from the CPPS engineering domain and on software test automation, and is time-consuming. In this paper, we investigate how domain experts can efficiently derive test cases for an assembly process step from process preconditions concerning product properties. We introduce the Test Case Derivation for PPR Models (TCD4PPR) method building on the Formalised Process Description and best practices from software testing. We evaluate the TCD4PPR method with an illustrative use case from industry in a feasibility study with domain experts at a large production systems engineering company for discrete manufacturing. The main result was that the domain experts found the TCD4PPR method efficient, usable, and useful. The evaluation results indicate that investing reasonable effort into modeling Product, Process, Resource (PPR) knowledge with preconditions can considerably reduce risks of untested production process behavior.

The verification of Extended Entity Relationship (EER) diagrams and other conceptual models that capture the design of information systems is crucial to ensure reliable systems. To scale up verification processes to larger groups of experts, Human Computation techniques were used focusing primarily on closed tasks, which constrain the number and variety of reported defects in favor of easy aggregation of derived judgements. To address this limitation of closed tasks, in this paper, we investigate EER verification (as instance of a broader family of model verification problems) with open tasks to extend the range of collected results. We also address the challenge of aggregating results of open tasks by proposing a follow-up HC task for defect validation. We evaluate our approach for HC-based EER Verification with open tasks in a set of experiments conducted with junior developers and show that (1) open tasks allow collecting a variety of insights that go beyond a manually built gold standard while still leading to good performance (F1=60%) and (2) HC-based validation can be reliably used for validating the results of open tasks (F1=84% compared to expert validation).

Background: Replication is essential to build knowledge in empirical science. Experiment replications reported in the software engineering context present variabilities on their design elements, e.g., variables, materials. The understanding of these variabilities is required to plan experimental replications within a research program. However, the lack of an explicit representation of experiments' variabilities and commonalities is likely to hamper their understanding and replication planning. Aims: The goal of this paper is to explore the use of Variability Modeling Approaches (VMAs) to represent experiment families (i.e., an original study and its replications) and to investigate the feasibility of using VMAs to support experiment replication planning. Method: We selected two experiment families, analyzed their commonalities and variabilities, and represented them using a set of well-known VMAs: Feature Model, Decision Model, and Orthogonal Variability Model. Based on the resulting models, we conducted a preliminary comparison of using such alternative VMAs to support replication planning. Results: Subjects were able to plan consistent experiment replications with the VMAs as support. Additionally, through a qualitative analysis, we identified and discuss advantages and limitations of using the VMAs. Conclusions: It is feasible to represent experiment families and to plan replications using VMAs. Based on our emerging results, we conclude that the Feature Model VMA provides the most suitable representation. Furthermore, we identified benefits in a potential merge between the Feature Model and Decision Model VMAs to provide more details to support replication planning.

For the design of a Production System (PS), engineers have to select production resources that address the associated product and production process, i.e., product/ion, requirements. The dependencies between product, process, and resource (PPR) provide the foundation for mapping properties of the product and process to skills of production resources, which may be represented as attributes in resource catalogue tables. However, the production resources are represented in resource catalogues by heterogeneous sets of attributes that make it challenging to efficiently find a set of well-fitting resources. In this paper, we present challenges and quality criteria that we identified with domain experts at a large Production Systems Engineering (PSE) company. We focus on use cases that explore and select resources from large resource catalogues. We introduce a data model to organize these resource catalogues in the solution space based on PPR knowledge. We propose a method for efficiently exploring the resource solution space regarding PPR requirements. In a conceptual feasibility study, domain experts rated the quality of the method based on a conceptual prototype. The domain experts found the approach feasible and useful to efficiently document decisions on resource selection.

In discrete manufacturing, basic and detail engineering workgroups collaborate to design a cyber-physical production system. Product/ion-aware modeling recognizes requirements coming from the product and from the production process for designing a production resource. These requirements imply consistency dependencies between product, production process, and resource (PPR) model elements. Unfortunately, there is only limited support for modeling consistency dependencies between PPR model elements and for defining abstract types in addition to concrete instances of PPR model elements. In this paper, we build on the PPR modeling capabilities of the VDI/VDE 3682 guideline, the Formal Process Description (FPD). We propose extensions for representing the refinement of types and instances as well as consistency dependencies between PPR model elements. We evaluate the FPD language extensions in a feasibility study with domain experts at a large production system engineering company for discrete manufacturing. The main result is that the domain experts found the extended FPD useful and usable for representing PPR consistency as a foundation for making design decisions.

In discrete manufacturing, basic and detail engineering workgroups need to collaborate to design highly automated cyber-physical production systems. Product/ion-awareness describes views and requirements coming from product and production process design, which are relevant to engineer production resources. These requirements imply strong dependencies between the product, the production process, and production resources (PPR). The Formal Process Description (FPD) provides basic concepts for modeling PPR knowledge, which support discrete manufacturing to some extent. In this paper, we introduce a meta-model that describes the structure of the FPD including a set of proposed extensions, focusing on expressing consistency dependencies on PPR relations, as a foundation for making design decisions traceable in the engineering process. In addition, the meta-model provides a clear description of how to model parallel or alternative process flows, a common use case in discrete manufacturing. The meta-model provides stakeholders with a clear description of the PPR modeling language (PPR-ML) and a rule set to check the validity of a model.

Context. Production Systems Engineering (PSE) faces a growing complexity of software, i.a., due to increasing capabilities of the hardware, requiring efficient approaches for designing test data and test cases. Apart from this, for long-running legacy systems where often no tests are available tests need to be added belatedly during system maintenance. Equivalence Class Partitioning (ECP) can help to systematically cluster test input and result data as a foundation for test case definition. However, different tools and technologies in PSE often hinder the precise definition and creation of Equivalence Classes (EC) based on the existing source code. Objective. In this paper, we present a model-based approach for deriving ECs based on abstract representations of existing source code and evaluate the approach in a real-world industry use case. Method. We build on best-practices from software testing for developing a model-based approach for deriving ECs as a foundation for test data and test case generation and on the Abstract Syntax Tree (AST) to describe the structure of the underlying source code. Results. First results were promising in the evaluation use case to improve test activities by systematically deriving test data and test cases based on ECs. Conclusions. We observed additional benefits, such as increased test coverage and capabilities for testing error cases.

Industrial production systems are complex automated systems-of-systems. Especially in case of Industry 4.0 production systems, their design phase relies on parallel cooperative work of engineers coming from several disciplines. This paper identifies key engineering roles and their interactions in the design phase. Based on the analysis of shared information among engineers, this paper contributes to the specification of an information model of shared common concepts. Such an information model is useful for designing a central repository in production system engineering projects as well as a basis for standardization processes and engineering tool evaluation. The roles and relevant information are based on lessons learned from the use case of an Industry 4.0 Testbed environment.

Similar to advanced software engineering, multi-disciplinary systems engineering, such as industrial production systems engineering (PSE), has to integrate partial results from workgroups that design in parallel. Due to the heterogeneity of data sources and the divergence of local data models the exchanged engineering artefacts between PSE workgroups are complex, making the automation of the data exchange process in PSE difficult and prone to technical debt (TD). In this paper, we report on a case study at a large PSE company to analyze TD effects, items, and causes in PSE, focusing on the engineering data exchange process. We identified key use cases and TD types, i.e., TD in data models and TD in data instances, which have adverse effects on project effort, cost and duration as well as data quality.

Cyber-Physical Production Systems (CPPSs) engineering relies on the effective and efficient coordination and collaboration of participating engineers from various disciplines implying a proficient knowledge exchange between them. Au-tomationML (AML) provides a standardized, XML-based data format allowing to exchange engineering data, which receives more and more attention as an engineering data model. When using shared data exchange platforms, efficient data storage for AML models is success-critical in CPPS engineering. The purpose of this paper is to draft a novel, flexible evaluation framework in the context of AML model storage, modification, and retrieval and to evaluate two particular data storage paradigms, i.e., XML-(BaseX) and graph-based (Neo4J) databases. Based on a common best practice API, we developed a prototype solution enabling a flexible exchange of underlying data storage paradigms for AML models. We used an academic AML data set for the performance evaluation of two selected database engines for common storage tasks. First results showed that BaseX performs better for creating, updating, and deleting operations while Neo4J performs better for reading operations. While BaseX efficiently supports storing and retrieving AML models, we also observed querying limitations in the AML API. Nevertheless, in the context of AML data storage evaluations, the selected data sets, and the solution approach can be seen as an initial benchmark.

The realization of a cyber-physical production system (CPPS) requires suitable methods and tools for the exchange and integration of engineering data between collaborating disciplines. Unfortunately, the description languages used in a CPPS Engineering (CPPSE) organization to describe discipline-specific views are not necessarily well suited for high-quality data exchange between workgroups. In this paper, we identify technical debt and risks regarding CPPSE description languages for data exchange using the VDI 3695 guideline as best practice. We report on effects and likely causes of the quality risks identified in a case study at a large CPPSE company. Based on data from workshops and semi-structured interviews with 28 domain experts from 12 workgroups, we propose a preliminary model relating causes and effects as foundation for analyzing and managing risks in the CPPSE data exchange process.

In collaborative Production Systems Engineering (PSE), decisions of detail engineers depend on results of previous design decisions regarding Product, Process, and Resource (PPR), coming from other basic or detail engineering disciplines. However, the engineering results exchanged between disciplines often represent the production system view only. PPR design decisions are important to facilitate the product/ion-aware exploration of the system design space, but are hard to express and trace in modeling approaches with only limited support for PPR. In this paper, we report on a survey of modeling approaches, which combine process and product modeling. We investigate the PPR modeling capabilities for product/ion-aware systems design and for tracing design decisions in the engineering process. Major finding is that most of the investigated modeling approaches fulfill general requirements well, but address requirements regarding tracing PPR design decisions at most partially. As an exception, the Formal Process Description (FPD) provides good capabilities for PPR modeling and provides the foundations for tracing design decisions between basic to detail engineering.

In the multi-disciplinary production systems engineering (PSE) process, software engineers depend on requirements and design rationales coming from product and production process planning, summarized as product/ion knowledge.Unfortunately, the engineering artifacts coming from product/ion planning often represent important product/ion knowledge incompletely and not well integrated, leading to risks regarding software engineering quality.In this paper, we report on a case study at a large industrial PSE organization, investigating Technical Debt (TD) effects, items, and causes in PSE process documentation and configuration management according to the VDI guideline 3695 Part 2. We focus on requirements for and issues in the representation of product/ion knowledge in the engineering data provided to software engineers.Based on data elicited from PSE domain experts, we model TD concepts based on the Quality Function Deployment method as foundation for TD analysis and risk management.The initial validation with domain experts revealed how software engineers could benefit from improved product/ion knowledge modeling as foundation for better understanding the rationale of engineering design decisions.

Requirements Engineering (RE) has established itself as a software engineering discipline over the past decades. While researchers have been investigating the RE discipline with a plethora of empirical studies, attempts to systematically derive an empirical theory in context of the RE discipline have just recently been started. However, such a theory is needed if we are to define and motivate guidance in performing high quality RE research and practice. We aim at providing an empirical and externally valid foundation for a theory of RE practice, which helps software engineers establish effective and efficient RE processes in a problem-driven manner. We designed a survey instrument and an engineer-focused theory that was first piloted in Germany and, after making substantial modifications, has now been replicated in 10 countries worldwide. We have a theory in the form of a set of propositions inferred from our experiences and available studies, as well as the results from our pilot study in Germany. We evaluate the propositions with bootstrapped confidence intervals and derive potential explanations for the propositions. In this article, we report on the design of the family of surveys, its underlying theory, and the full results obtained from the replication studies conducted in 10 countries with participants from 228 organisations. Our results represent a substantial step forward towards developing an empirical theory of RE practice. The results reveal, for example, that there are no strong differences between organisations in different countries and regions, that interviews, facilitated meetings and prototyping are the most used elicitation techniques, that requirements are often documented textually, that traces between requirements and code or design documents are common, that requirements specifications themselves are rarely changed and that requirements engineering (process) improvement endeavours are mostly internally driven. Our study establishes a theory that can be used as starting point for many further studies for more detailed investigations. Practitioners can use the results as theory-supported guidance on selecting suitable RE methods and techniques.

Automated and systematic testing of automation systems (AS) and production systems (PS) require an integrated testing tool chain for test case development, execution and reporting. In practice, the test automation tool chain cannot be fully automated because of missing links between different tools used in the test automation process. Closing these gaps typically require (high) human effort. Furthermore, domain and software testing expertise is often bundled by one (expensive) engineer who is responsible for the application domain (reflected in use cases and test cases) and software tests (software test code). This paper presents a flexible Testing Automation Framework (TAF) that enables the configuration of test processes involving different tools and various layers for test automation and enables separated roles for the application domain and software tests. We build on best-practice test automation from Software Engineering and design a test automation process for the automation systems domain. We demonstrate the feasibility with a use case, derived from production systems automation, with selected tools covering all test automation layers. First results showed the feasibility of the framework in the evaluation use case making test processes more flexible and automated. Although the successful implementation of the TAF can support the efficient configuration and execution of test processes, there is additional effort for preparing the flexible and automated tool chain.

Conceptual domain models, such as taxonomies, knowledge graphs or Extended Entity Relationship (EER) diagrams are core to all information systems. The task of verifying the correctness of these models is of high interest to the knowledge and software engineering communities and attracted the first solution approaches using human computation. Yet, since these solutions are published within the boundaries of their communities, there is a lack of concerted work on this topic. As a first step to alleviate this status quo, we formalize the problem of verifying conceptual models and propose a generic approach (VeriCoM) to solve it with human computation techniques. We show how VeriCoM was applied in a software engineering use case focusing on verifying the correctness of an EER diagram against a system specification document. An evaluation of VeriCoM in a series of four workshops within one controlled experiment performed with a crowd of semi-experts lead to the identification of a set of defects with precision of 73% and a recall from a Gold Standard defect set of 63%.

Typical large-scale systems engineering projects depend on seamless cooperation and data exchange of experts from various engineering domains and organizations that work in a heterogeneous engineering environment. Available software tools support individual engineering disciplines quite well, but they only represent a discipline-specific view on the engineering plant. Consequently, a so-called integrated plant model captures and combines all different views into one representation in order to provide an overarching, discipline-independent view on the engineering plant. However, in order to support effective engineering processes, like change management, stakeholders need to be able to (a) define the scope of their changes they want to merge into the integrated plant model rather than the latest status of their view with various fragile adaptations, and (b) extract only engineering information from integrated plant model which is in the scope of the stakeholders's discipline and interest. In this paper, we describe requirements identified in industrial use cases regarding filtering capabilities on (integrated) engineering plant models and model-driven engineering techniques for model-slicing applied on AutomationML models. The approach contributes to quality assurance and fault-prevention in engineering data since it helps to focus on parts of the engineering plant model relevant in certain engineering processes.

[Context] In Multi-Disciplinary Engineering (MDE) environments, the engineering of industrial production systems requires the collaboration of engineers coming from different disciplines. Engineers typically apply discipline specific tools and data models with limited collaboration capabilities. These loosely coupled tools and heterogeneous data models hinder efficient change management and defect detection, which makes MDE projects unnecessarily risky and error prone. [Objective] This paper presents an adapted review approach, AML-Review, for multi-disciplinary engineering (MDE) projects based on best practices for reviews in software engineering. [Method] Software reviews have been successfully used for early defect detection in Software Engineering. However, adaptations are needed for defect detection in MDE environments. We focus on production systems models according to the emerging AutomationML standard. [Results] We evaluated the feasibility of the AML-Review process with requirements and an AutomationML model from a real-world application scenario. The AML-Review process provides the benefits of systematic and traceable review results for MDE projects based on AutomationML. [Conclusion] The prototype results imply that systematic and structured review processes help to improve traceability of requirements and defects and increase defect detection performance.

[Context] Experiment planners need to be aware of relevant Threats to Validity (TTVs), so they can devise effective control actions or accept the risk. [Objective] The aim of this paper is to introduce a TTV knowledge base (KB) that supports experiment planners in identifying relevant TTVs in their research context and actions to control these TTVs. [Method] We identified requirements, designed and populated a TTV KB with data extracted during a systematic review: 63 TTVs and 149 control actions from 206 peer-reviewed published software engineering experiments. We conducted an initial proof of concept on the feasibility of using the TTV KB and analyzed its content. [Results] The proof of concept and content analysis provided indications that experiment planners can benefit from an extensible TTV KB for identifying relevant TTVs and control actions in their specific context. [Conclusions] The TTV KB should be further evaluated and evolved in a variety of software engineering contexts.

In a multi-disciplinary engineering project, such as the parallel engineering of industrial production plants, domain experts want to efficiently monitor project-level constraints that depend on technical parameter values in local engineering models. However, the heterogeneous representations of constraint parameters in these engineering models make the automation of constraint monitoring difficult. In this paper, we introduce the Multi-Model Dashboard (MMD) process providing semantically integrated values of parameters and of constraints to domain experts, as parameter values in various local models change during the project. The tool-supported MMD process guides the definition and monitoring of MMD parameters and constraints. We evaluate the effectiveness and efficiency of the MMD process in a feasibility study with requirements and data from real-world use cases at industry partners. Major results are that the MMD process was effective and efficient in eliciting relevant project constraints and model dependencies and in providing data for change impact analysis.

Large systems engineering projects involve the cooperation of various stakeholders from different engineering disciplines. Individual stakeholders apply various tools and related data storage approaches that (a) might hinder seamless interoperability and (b) include limited capability to support data versioning. Project-level concepts enable the mapping of engineering data coming from different disciplines. However, it remains open how to store data on project level that enable flexible and efficient data access from different disciplines in different environments and enable backtracking to previous versions in case of defects and/or human errors. While semantic data integration provides fundamental solutions for bridging semantic gaps between common project-level concepts and the local tool concepts used by each discipline, semantic storages have been developed to query and reason over gathered data rather than versioning frequent instance changes inherent to such engineering projects in distributed and heterogeneous environments. In this paper we evaluate three software architectures using ontologies in different ways and compare selected quality attributes, i.e., performance and scalability, in the context of an industrial scenarios. Main results suggest that architectures relying on a relational database for versioning individuals still outperforms traditional ontology storages.

Mechatronic design processes in automation systems engineering projects require concurrent engineering of various disciplines operating within technical, commercial, and project management constraints. Unfortunately, early and efficient quality assurance of technical parameters, which affect risk-relevant constraints in the mechatronic design processes, is difficult and error-prone due to the heterogeneous local model representations of shared concepts that the domain experts use to define these constraints. In this paper, we investigate the effectiveness and efficiency of the adapted multi-model dashboard (MMD) approach and a traditional approach for monitoring technical parameters and constraints in mechatronic design processes. We evaluate these approaches in the context of a real-world use case, the welding line process for automobile parts. Major results are: the MMD process was at least as effective and efficient in eliciting relevant project constraints and model dependencies as traditional approaches in the evaluation context.

Context. Software engineering researchers conduct systematic literature reviews (SLRs) to build bodies of knowledge (BoKs). Unfortunately, relevant knowledge collected in the SLR process is not publicly available, which considerably slows down building BoKs incrementally. Objective. We present and evaluate the Systematic Knowledge Engineering (SKE) process to support efficiently building BoKs from published research. Method. SKE is based on the SLR process and on Knowledge Engineering practices to build a Knowledge Base (KB) by reusing intermediate data extraction results from SLRs. We evaluated the feasibility of applying SKE by building a Software Inspection BoK KB from published experiments and a Software Product Line BoK KB from published experience reports. We compared the effort, benefits, and risks of building BoK KBs regarding the SKE and the traditional SLR processes. Results. The application of SKE for incrementally collecting and organizing knowledge in the context of a BoK was feasible for different domains and different types of evidence. While the efforts for conducting the SKE and traditional SLR processes are comparable, SKE provides significant benefits for building BoKs. Conclusions. SKE enables researchers in a scientific community to reuse and incrementally build knowledge in a BoK. SKE is ready to be evaluated in other software engineering domains.

[Context] Replication Data Management (RDM) aims at enabling the use of data collections from several itera-tions of an experiment. However, there are several major chal-lenges to RDM from integrating data models and data from em-pirical study infrastructures that were not designed to cooperate, e.g., data model variation of local data sources. [Objective] In this paper we analyze RDM needs and evaluate conceptual RDM approaches to support replication researchers. [Method] We adapted the ATAM evaluation process to (a) analyze RDM use cases and needs of empirical replication study research groups and (b) compare three conceptual approaches to address these RDM needs: central data repositories with a fixed data model, heterogeneous local repositories, and an empirical ecosystem. [Results] While the central and local approaches have major issues that are hard to resolve in practice, the empirical ecosys-tem allows bridging current gaps in RDM from heterogeneous data sources. [Conclusions] The empirical ecosystem approach should be explored in diverse empirical environments.

Interoperability of engineering tools, which is important for engineering efficiency, requires the openness of the tools for the import and/or export of engineering data. A metric to assess the openness of an engineering tool has been presented recently at IEEE ETFA 2012. This contribution reflects the methodology and focuses on the initial empirical assessment of the method with data from industrial practice, i.e., the comparison of openness evaluation of well-established engineering tools. The authors report evaluation results and discuss lessons learnt to guide improving the openness of automation tools for supporting more efficient engineering processes. Major results are: the measurement method has been found useful and usable in practice and pointed out strengths and weaknesses in 15 well-established engineering tools from 4 tool categories.

Automation systems engineering projects typically rely on the collaboration of experts from different engineering disciplines. Multi-disciplinary interaction requires from project participants to exchange (intermediate) artefacts represented and accessed differently in each of the tools. However, local data representations are only fully understood by the tool's expert and therefore not suitable for all possible (human) readers. In order to share engineering knowledge, the process relies on time-consuming human search and translation activities between corresponding tool data representations which causes overhead in communication and is prone to errors. In this paper, the XLink navigation concept is introduced which enables semi-automated translation and navigation between related data elements of different local representations. Using common concepts shared between various engineering tools, XLink enables project participant to define references to local data elements which may be shared with any other project participant. The concept enables direct navigation between related data elements of different local data models without the need for human search and translation activities. Based on a real-world use case and an implemented prototype evaluated in a controlled experiment, we show that the concept is more effective and efficient than the standard human-based approach.

Automation systems development projects typically require the integration of heterogeneous local tool data models that come from various disciplines and sources. Semantic data integration provides solutions for bridging semantic gaps between common project-level concepts and the local tool concepts used by each discipline. The following use cases represent the foundation for efficient data integration: (a) data insertion in the local tool models, (b) transformation of data between the local models and a common model, and (c) querying across concepts from different local models by using the common model. The selection of a proper semantic data storage for storing the data has a strong impact on efficiently executing these use cases. Three different important types of semantic storages have been identified: ontology file storages, triple storages, and relational databases storages. In this paper, we evaluate them, and identify their drawbacks and advantages in the context of the presented integration use cases' requirements.

Mechatronics is a multidisciplinary field of engineering combining disciplines like mechanical, electronic or software engineering, in order to design and manufacture useful products. Nowadays, mechatronic engineering is well-supported either by using integrated tool suites providing a homogeneous approach to engineering, or by relying on established tool chains consisting of a set of engineering tools connected using a common data exchange format. However, in practice neither tool suites nor tool chains have become a de facto standard in engineering, leading to tedious and often manual integration efforts required to combine specific engineering tools or tool suites. This paper presents an engineering tool integration framework that allows the definition and usage of mechatronic objects originating from heterogeneous engineering tools, so-called “engineering objects”. These engineering objects can additionally include project and organizational information, thus enabling exhaustive engineering process management and monitoring. The presented approach is evaluated in an industrial case study from the hydro power plant engineering domain. Major results are engineering objects that can include heterogeneous data, such as project or organization-specific information, thus enabling automated and therefore more efficient synchronization between the involved engineering disciplines, as well as added-value applications, like project monitoring or quality assured data import and export.

In the field of industrial automation systems software becomes an important factor because engineers tend to move the realization of functional requirements from hardware to software components. The main reason for this is that software components allow increasing product flexibility. As a consequence software complexity increases rapidly and requires systematic, automation-supported and agile testing approaches. Thus, systematic and agile testing are key challenges in industrial control software development to ensure and improve systems quality. Further different implementation standards, i.e., IEC 61131-3 and IEC 61499, arise additional challenges in constructing and testing industrial automation systems software. This paper presents an agile and keyword-driven test approach with focus on testing implementations based on both important industrial standards and illustrates the applicability of the purposed approach in a sample implementation, i.e., a High Speed Pick and Place unit. Main results show the applicability of keyword-driven testing based on a defined subset of keywords (common for IEC 61131-3 and IEC 61499) and thus enable agile and automation-supported testing more effective and efficient.

Automation Systems Engineering projects typically depend on the collaboration of several engineering disciplines. While available software tools are supporting individual engineering disciplines quite well, there is very little work on tool collaboration and engineering process automation across discipline boundaries. Following a sequential process structure with distributed and parallel activities, an efficient integration of loosely linked heterogeneous engineering tool models for information retrieval or model consistency checking for quality assurance (QA) requires high human effort. Mechatronic objects provide a logical view on integrated data from mechanical, electrical, and software engineering but there is limited support of heterogeneous and dynamic engineering tool models. Based on so-called engineering objects this paper introduces the capability of single-click navigation between heterogeneous engineering tools taking into account the different views on common engineering concepts. The proposed approach is evaluated by using an example from the process automation domain. Major results are that navigation increases the understanding and traceability of complex relations and thus enables the efficient diagnosis of deviations between models.

Modern industrial automation systems engineering (ASE) environments have to accommodate for heterogeneity coming from the engineering disciplines involved, the software tools and their data models, and run-time data collection. In many ASE environments domain experts have to invest considerable effort to bridge the semantic gaps between common project-level engineering concepts and the diverse local data representations. In this paper we discuss the needs for semantic integration and applications of machine-understandable knowledge engineering in three real-world ASE use cases from our industry partners. We provide an evaluation concept with empirical studies to measure the benefits and limitations of the proposed approach compared to the traditional expert-intensive approach. Major result of the initial evaluation is that semantic integration has good potential to make engineering processes more efficient and robust if supported well with user interfaces that end users find usable and useful.

Software systems in safety-critical industrial automation systems, such as power plants and steel mills, become increasingly large, complex, and distributed. For assessing risks, like low product quality and project cost and duration overruns, to trustworthy services provided by software as part of automation systems there are established risk analysis approaches based on data collection from project participants and data models. However, in multi-disciplinary engineering projects there are often semantic gaps between the software tools and data models of the participating engineering disciplines, e.g., mechanic, electrical, and software engineering. In this paper we discuss current limitations to risk assessment in (software+) engineering projects and introduce the SEMRISK approach for risk assessment in projects with semantically heterogeneous software tools and data models. The SEMRISK approach provides the knowledge engineering foundation to allow an end-to-end view for service-relevant data elements such as signals, by providing a project domain ontology and mappings to the tool data models of the involved engineering disciplines. We empirically evaluate the effectiveness and efficiency of the approach based on a real-world industrial use case from the safety-critical power plant domain. Major results are that the approach was effective and considerably more efficient than the current approach at the industry partner.

The engineering of complex production automation systems involves experts from several backgrounds, such as mechanical, electrical, and software engineering. The production automation expert knowledge is embedded in their tools and data models, which are, unfortunately, insufficiently integrated across the expert disciplines, due to semantically heterogeneous data structures and terminologies. Traditional integration approaches to data integration using a common repository are limited as they require an agreement on a common data schema by all project stakeholders. In this paper we introduce the Engineering Knowledge Base (EKB), a semantic-web-based framework, which supports the efficient integration of information originating from different expert domains without a complete common data schema. We evaluate the proposed approach with data from real-world use cases from the production automation domain on data exchange between tools and model checking across tools. Major results are that the EKB framework supports stronger semantic mapping mechanisms than a common repository and is more efficient if data definitions evolve frequently.

We propose a set of quantitative metrics to empirically evaluate security quality levels on an Open (Software) Engineering Service Bus (EngSB) platform.

The engineering of complex software-intensive systems, like industrial production plants, requires software engineering to coordinate and interact with other engineering disciplines. Project and quality managers need empirical study results to improve system quality, e.g., from process analysis of engineering process event sequences. In this paper, we propose a framework adapted from business process analysis to empirically analyzing engineering process event information. Initial results support the suitability of the approach for (software+) engineering environments. 2. RESEARCH APPROACH Based on the need for monitoring and improving crossdisciplinary engineering projects, we derive two research questions: 1. What adaptations are necessary to use the “process mining” methodology for analyzing (software+) engineering processes? 2. What kinds of events need to be integrated? We propose a framework for empirical event-based process analysis, as illustrated in Figure 1. The framework consists of three steps: (1) Heterogeneous event data is collected from a variety of tools used by multiple engineering disciplines. (2) Semantic integration integrates the collected heterogeneous event sequences and stores the data in an event log. (3) The integrated event data is used for process mining.

Test-driven development (TDD) - an established approach in business IT software development - enables test case generation based on models early in the development process. Applying TDD and models in automation systems engineering (ASE) can increase testing effectiveness and efficiency. A key question is which models are suitable for ASE application. UML models support software and systems engineering development in (a) systematically capturing requirements, (b) describing the static system architecture, and (c) specifying dynamic systems behavior. In this paper we discuss selection criteria for UML model selection in ASE and evaluate strengths and limitations of selected models.

Increasing complexity of software components in automation systems require systematic and frequent testing approaches. Test-First Development (TFD) - an established approach in business IT software development - promises to support test automation in automation systems development. Nevertheless, linking test case generation, execution, and reporting requires a sound framework to support testing processes more efficiently. In this paper we present a framework for automating test processes based on UML models and TFD. Applying this framework in prototype applications in industry environment identified the framework as promising candidate to improve automation systems development and product quality.

The development of video games comprises engineering teams within various disciplines, e.g., software engineering, game production, and creative arts. Game jams are a promising approach for (software+) development projects to foster on new product development. This paper evaluates the concept of game jam, a community design/development activity, and its positive effects on new software product development with tight schedules in time-oriented, competitive environments. Game jams have received more public attention in recent times, but the concept itself has not been formally discussed so far. A game jam is a composition of design and development strategies: new product development, participatory design, lightweight construction, rapid experience prototyping, product-value focusing, aesthetics and technology, concurrent development and multidisciplinarity. Although game jams are normally used for rapid prototyping of small computer games, the constellation of the mentioned elements provides a powerful technique for rapidly prototyping new product ideas and disruptive innovations.

Automation systems software must provide sufficient diagnosis information for testing to enable early defect detection and quality measurement. However, in many automation systems the aspects of automation, testing, and diagnosis are intertwined in the code. This makes the code harder to read, modify, and test. In this paper we introduce the design of a test-driven automation (TDA) component with separate aspects for automation, diagnosis, and testing to improve testability and test efficiency. We illustrate with a prototype, how automation component aspects allow flexible configuration of a ¿system under test¿ for test automation. Major result of the pilot application is that the TDA concept was found usable and useful to improve testing efficiency.

Studying the effectiveness of scenario development meetings in the software architecture process is important to improve meeting effectiveness. This paper reports initial findings from analyzing the data collected in a controlled experiment aimed at studying the effectiveness of meetings in terms of gained and lost scenarios of individuals, real and nominal (non-communicating) teams. Our findings question the effectiveness of holding meetings since more important scenarios were lost than gained in these meetings. In the study nominal teams performed better than individuals and real teams.

We have developed a framework for groupware tool support for the software architecture evaluation process in the context of global software development. We have empirically assessed the effectiveness of the groupware-supported software architecture evaluation process in a set of controlled experiments. While we found that groupware-supported distributed meetings can be very effective, we saw the need to investigate users' acceptance of the tool used in these empirical studies. In this paper we report on the "perceived usefulness" and "ease of use" of the groupware tool based on the adapted Davis' technology acceptance model (TAM), a widely used general-purpose instrument for measuring users' attitude towards a particular technology. Main results from analyzing the TAM data are: a majority of the participants found the tool quite useful and easy to use for supporting collaborative tasks like architecture evaluation; a majority of the respondents was also very positive about the regular use of the tool for collaborative tasks in the future. However, there was considerably less support for preferring a distributed tool-based meeting to a face-to-face meeting.

The planning phase of empirical studies is a success-critical key activity in empirical research to achieve best benefits for contributing stakeholders, e.g., researchers and industry partners, and to reduce study risks, e.g., insufficient validity and unaddressed stakeholder win conditions. The design of empirical studies typically covers issues of empirical methodology, but seldom explicitly discusses tradeoffs between conflicting study goals. This work proposes a value-based empirical research planning framework for eliciting and reconciling stakeholder win conditions in order to compare the benefits and risks of empirical study variants and reports on findings from an initial feasibility study in a ISERN meeting of empirical research experts.