Metamorphic Testing (MT) addresses the test oracle problem by defining how program outputs should change in response to specific input changes. The relations between input changes and their corresponding output changes are called Metamorphic Relations (MRs). Generating suitable MRs is complex and often requires deep domain knowledge. Our previous work introduced MetaTrimmer, a test-data-driven approach for selecting and constraining MRs, involving three steps: Test Data (TD) Generation, MT Process, and MR Analysis. MR Analysis is done to decide whether the violation of an MR for a specific input data pair (original and changed) indicates a failure or simply means that the MR does not apply for the chosen inputs. In this paper, we present an association-rule-based approach that semi-automatically extracts constraints dividing the input space into valid/invalid data during the MR Analysis step of MetaTrimmer. We validate our approach using 44 methods to which six predefined MRs are applied. Our results indicate that the proposed method efficiently identifies correct input data space constraints. More studies are needed to provide additional evidence that MetaTrimmer with the enhanced MR Analysis step is scalable and generalisable.

Regression testing is crucial in ensuring that pure code refactoring does not adversely affect existing software functionality, but it can be expensive, accounting for half the cost of software maintenance. Automated test case generation reduces effort but may generate weak test suites. Test amplification is a promising solution that enhances tests by generating additional or improving existing ones, increasing test coverage, but it faces the test oracle problem. To address this, we propose a test oracle derivation approach that uses object state data produced during System Under Test (SUT) test execution to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT’s intended behaviour. Our preliminary evaluation shows that the proposed approach can enhance the detection of behaviour changes substantially, providing initial evidence of its effectiveness.

Identifying and selecting high-quality Metamorphic Relations (MRs) is a challenge in Metamorphic Testing (MT). While some techniques for automatically selecting MRs have been proposed, they are either domain-specific or rely on strict assumptions about the applicability of a pre-defined MRs. This paper presents a preliminary evaluation of MetaTrimmer, a method for selecting and constraining MRs based on test data. MetaTrimmer comprises three steps: generating random test data inputs for the SUT (Step 1), performing test data transformations and logging MR violations (Step 2), and conducting manual inspections to derive constraints (Step 3). The novelty of MetaTrimmer is its avoidance of complex prediction models that require labeled datasets regarding the applicability of MRs. Moreover, MetaTrimmer facilitates the seamless integration of MT with advanced fuzzing for test data generation. In a preliminary evaluation, MetaTrimmer shows the potential to overcome existing limitations and enhance MR effectiveness.

The evolution and re-engineering of software is strongly supported by comprehensive test suites for checking the correctness and robustness of software systems. However, this requires comprehensive test suites to be in place or developed prior to re-engineering of systems. Many techniques and strategies have been proposed to automate the creation of test suites. Specification-based testing aims to use all available behavioral specifications of software to drive black-box testing of software. In this paper, we present an approach, which applies specification-based testing to component software in the engineering domain, supporting an ongoing re-engineering process aiming to transform monolithic C++ engineering software into reusable fine-grained computational services. The developed test generator (TxTea) uses source code annotations added to component interface definitions and drives test data generation by a randomized testing approach, which integrates a satisfiability modulo theories (SMT) solver to check satisfiability and generate test data for comprehensive preconditions. As we show, TxTea can be successfully applied to a given component technology, and as we discuss in this paper, the customized specification language and generic implementation make reuse in more general use cases feasible.

Metamorphic Testing (MT) is a testing technique that can effectively alleviate the oracle problem. MT uses Metamorphic Relations (MRs) to determine if a test case passes or fails. MRs specify how the outputs should vary in response to specific input changes when executing the System Under Test (SUT). If a particular MR is violated for at least one test input (and its change), there is a high probability that the SUT has a fault. On the other hand, if a particular MR is not violated, it does not guarantee that the SUT is fault free. However, deciding if the MR is being violated due to a bug or because the MR does not hold/fit for particular conditions generated by specific inputs remains a manual task and unexplored. In this paper, we develop a method for refining MRs to offer hints as to whether a violation results from a bug or arises from the MR not being matched to certain test data under specific circumstances. In our initial proof-of-concept, we derive the relevant information from rules using the Association Rule Mining (ARM) technique. In our initial proof-of-concept, we validate our method on a toy example and discuss the lessons learned from our experiments. Our proof-of-concept demonstrates that our method is applicable and that we can provide suggestions that help strengthen the test suite for regression testing purposes.

Open Platform Communications Unified Architecture (OPC UA) is an increasingly important standard for industrial communication, which brings up the need for testing. Still, functional or formal verification of the implementation relies on conformance testing or custom solutions. Model-based testing has been studied and applied for decades, targeting many different technologies. OPC UA already provides a defined serialization format for its interfaces, representing a simple model of the application that could be utilized for testing.In this paper, we present our work on applying model-based testing to OPC UA server applications using the serialized interface description as the model. We introduce a possible toolchain for that approach and, based on that, discuss the possibilities and shortcomings of the interface description as a model for testing.

Metamorphic testing (TM) examines the relations between inputs and outputs of test runs. These relations are known as metamorphic relations (MR). Currently, MRs are handpicked and require in-depth knowledge of the System Under Test (SUT), as well as its problem domain. As a result, the identification and selection of high-quality MRs is a challenge. Kanewala et al. suggested the Predicting Metamorphic Relations (PMR) approach for automatic prediction of applicable MRs picked from a predefined list. PMR is based on a Support Vector Machine (SVM) model using features derived from the Control Flow Graphs (CFGs) of 100 Java methods. The original study of Kanewala et al. showed encouraging results, but developing classification models from CFG-related features is costly. In this paper, we aim at developing a PMR approach that is less costly without losing performance. We complement the original PMR approach by considering other than CFG-related features. We define 21 features that can be directly extracted from source code and build several classifiers, including SVM models. Our results indicate that using the original CFG-based method-level features, in particular for a SVM with random walk kernel (RWK), achieve better predictions in terms of AUC-ROC for most of the candidate MRs than our models. However, for one of the candidate MRs, using source code features achieved the best AUC-ROC result (greater than 0.8).

Metamorphic Testing (MT) addresses the test oracle problem by examining the relations between inputs and outputs of test executions. Such relations are known as Metamorphic Relations (MRs). In current practice, identifying and selecting suitable MRs is usually a challenging manual task, requiring a thorough grasp of the SUT and its application domain. Thus, Kanewala et al. proposed the Predicting Metamorphic Relations (PMR) approach to automatically suggest MRs from a list of six pre-defined MRs for testing newly developed methods. PMR is based on a classification model trained on features extracted from the control-flow graph (CFG) of 100 Java methods. In our replication study, we explore the generalizability of PMR. First, since not all details necessary for a replication are provided, we rebuild the entire preprocessing and training pipeline and repeat the original study in a close replication to verify the reported results and establish the basis for further experiments. Second, we perform a conceptual replication to explore the reusability of the PMR model trained on CFGs from Java methods in the first step for functionally identical methods implemented in Python and C++. Finally, we retrain the model on the CFGs from the Python and C++ methods to investigate the dependence on programming language and implementation details. We were able to successfully replicate the original study achieving comparable results for the Java methods set. However, the prediction performance of the Java-based classifiers significantly decreases when applied to functionally equivalent Python and C++ methods despite using only CFG features to abstract from language details. Since the performance improved again when the classifiers were retrained on the CFGs of the methods written in Python and C++, we conclude that the PMR approach can be generalized, but only when classifiers are developed starting from code artefacts in the used programming language.

To visualize the progress of the software development and to foster the awareness of software quality, visualizations of the software build chain have been established in recent years and can be applied in the software service domain easily. Other domains like mechanical engineering are lacking a comprehensive view of the software build chain, since existing of-the-shelf solutions do not support the integration of software quality assurance activities outside the responsibility of the used CI tool adequately. In this paper we present a prototype of a tool that supports the requirement of an extended build chain visualization. In close cooperation with an industrial partner company, we implemented a flexible, adaptable, web based visualization system called DevOpsView and we discuss our findings about the future of customized build chain visualization systems.

Technical software documentation is an essential source of knowledge supporting many different software development activities and stakeholders. In this paper we report on use cases, existing documentation practices, and open challenges for technical software documentation in a company from the manufacturing domain. Further, we present an approach for automatically providing technical software documentation as a service. The approach has been validated in an industrial setting, i.e., we report on our experiences and lessons learned. Validation shows that while up-to-date and dynamically explorable documentation could be provided, the proposed solution failed in sufficiently supporting different stakeholders and their concerns i.e., it failed in providing sustainable documentation and knowledge preservation over time.

Modern graphical user interfaces (GUIs) are highly dynamic and support multi-touch interactions and screen gestures besides conventional inputs via mouse and keyboard. Hence, the flexibility of modern GUIs enables countless usage scenarios and combinations including all kind of interactions. From the viewpoint of testing, this flexibility results in a combinatorial explosion of possible interaction sequences. It dramatically raises the required time and effort involved in GUI testing, which brings manual exploration as well as conventional regression testing approaches to its limits. Automated test generation (ATG) has been proposed as a solution to reduce the effort for manually designing test cases and to speed-up test execution cycles. In this paper we describe how we successfully harnessed a state-of-the-art ATG tool (Randoop) developed for code-based API testing to generate GUI test cases. The key is an adapter that transforms API calls to GUI events. The approach is the result of a research transfer project with the goal to apply ATG for testing of human machine interfaces used to control industrial machinery. In this project the ATG tool was used to generate unit test cases for custom GUI controls and system tests for exploring navigation scenarios. It helped to increase the test coverage and was able reveal new defects in the implementation of the GUI controls as well as in the GUI application.

User interfaces of modern industrial automation systems are required to provide similar user experience as known from mass market consumer devices like smartphones. They need to support an increasing amount of dynamic visualizations as well as multi-touch interactions and screen gestures. However, the rich visualization frameworks that are applied and the elaborated user interfaces built on top have demanding hardware requirements, which can result in critical performance issues when run on resource-limited target platforms. In this paper we share our experiences and findings from analyzing the performance issues of a state-of-the-art user interface based on JavaFX. We describe the performance analysis approach, the tools we used for measurement and analysis, as well as insights gained into the JavaFX platform. We started with measuring and optimizing the performance of a concrete implementation of a user interface. Our results triggered immediate improvement actions, but they also led to the plan to develop of a performance model and corresponding tool support for predicting potential performance issues at design time, before the user interface is completed and deployed to the target hardware.

High resource demand for system testing is a major obstacle for continuous delivery. This resource demand can be reduced by prioritizing test cases, e.g., by focusing on tests that cover a lot of functionality. For large-scale systems, like an enterprise SOA, defining such test cases can be difficult for the tester because of the lack of relevant knowledge about the system. We propose an approach for test case prioritization and selection that is based on architectural viewpoint that provides software testers with the required architectural information. We outline how architectural information is used for defining and selecting prioritized test cases. The approach has been developed in close cooperation with the provider of an enterprise SOA in the banking domain in Austria following an action research approach. In addition, the approach has been validated in an industrial case study. Validation showed that there is no further need for manual architectural analysis to be able to prioritize and select test cases. We also show the limitations of our approach as it is based on static code analysis.

Retrieving and analyzing information from software repositories and detecting dependencies are important tasks supporting software evolution. Dependency information is used for change impact analysis, defect prediction as well as cohesion and coupling measurement. In this paper we report our experience from extracting dependency information from the change history of a commercial software system. We analyzed the software system's evolution of about six years, from the start of development to the transition to product releases and maintenance. Analyzing the co-evolution of software artifacts allows detecting logical dependencies between system parts implemented with heterogeneous technologies as well as between different types of development artifacts such as source code, data models or documentation. However, the quality of the extracted dependencies relies on established development practices and conformance to a defined change process. In this paper we indicate resulting limitations and recommend further processing and filtering steps to prepare the dependency data for subsequent analysis and measurement activities.

Individual differences of developers in performance and introduced defects have been reported by many research studies and are frequently observed in software development practice. Thus, when the source of defects in the final product is discussed, developers are usually the first under suspicion. However, defects residing in a released software product are the result of defects introduced throughout the sequence of development activities (e.g., specification, design, implementation, testing and stabilization) less the defects detected and removed in these activities. This case study explores and describes the difference between developers in terms of associated post-release (i.e., delivered) defects. The results are put in relation to the intensity with which a developer's changes and enhancements have been tested to identify a latent influence by pre-release quality assurance measures.

Software testing is essential and takes a large part of resources during software development. This motivates automating software testing as far as possible. Frameworks for automating unit testing are approved and applied for a plethora of programming languages to write tests for small units in the same programming language. Both constraints, unit size and programming language, inhibit automation of software testing in domain of mobile software frameworks. This circumstance has motivated the development of a new testbed for a framework in the domain of mobile systems. In this paper, we describe requirements and challenges in testing mobile software frameworks in general and present a novel testbed for the APOXI framework that addresses these requirements. The main ideas behind this testbed are the usage of a scripting language to specify test cases and to incorporate domain-specific aspects on the language level. The testbed facilitates component and system testing but can be used for unit testing as well.