Microservice-based systems have emerged as an effective architecture for countless industry applications. They provide applications as small, independent, and modular services. With the increasing interest in such systems, it is important to tackle challenges related to their quality assurance. However, to advance research in this area, systems are required to evaluate new approaches and tools. In this paper, we perform a systematic literature search for systems used in research for testing and monitoring microservice-based systems to aid future research. We provide an overview of the found studies and the systems used in their evaluation. We compose a list of publicly available systems and their characteristics, like size, available tests, and technologies used. Finally, we investigated the context in which these systems were used to provide insights in their usage and additional data that is available for them.

A machine control is essential for every production machine. Traditional machine controllers are housed in a single box with numerous connectors or a bus for attaching sensors and actuators. Programming is carried out using one of the IEC 61131–3 languages (usually adapted by the vendor). Switching to a different vendor becomes difficult. In this paper, we introduce a novel architecture to mitigate complexity and portability issues and address other programming challenges for machine control systems commonly used in mechanical engineering.

Metamorphic Testing (MT) addresses the test oracle problem by defining how program outputs should change in response to specific input changes. The relations between input changes and their corresponding output changes are called Metamorphic Relations (MRs). Generating suitable MRs is complex and often requires deep domain knowledge. Our previous work introduced MetaTrimmer, a test-data-driven approach for selecting and constraining MRs, involving three steps: Test Data (TD) Generation, MT Process, and MR Analysis. MR Analysis is done to decide whether the violation of an MR for a specific input data pair (original and changed) indicates a failure or simply means that the MR does not apply for the chosen inputs. In this paper, we present an association-rule-based approach that semi-automatically extracts constraints dividing the input space into valid/invalid data during the MR Analysis step of MetaTrimmer. We validate our approach using 44 methods to which six predefined MRs are applied. Our results indicate that the proposed method efficiently identifies correct input data space constraints. More studies are needed to provide additional evidence that MetaTrimmer with the enhanced MR Analysis step is scalable and generalisable.

Regression testing is crucial in ensuring that pure code refactoring does not adversely affect existing software functionality, but it can be expensive, accounting for half the cost of software maintenance. Automated test case generation reduces effort but may generate weak test suites. Test amplification is a promising solution that enhances tests by generating additional or improving existing ones, increasing test coverage, but it faces the test oracle problem. To address this, we propose a test oracle derivation approach that uses object state data produced during System Under Test (SUT) test execution to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT’s intended behaviour. Our preliminary evaluation shows that the proposed approach can enhance the detection of behaviour changes substantially, providing initial evidence of its effectiveness.

Identifying and selecting high-quality Metamorphic Relations (MRs) is a challenge in Metamorphic Testing (MT). While some techniques for automatically selecting MRs have been proposed, they are either domain-specific or rely on strict assumptions about the applicability of a pre-defined MRs. This paper presents a preliminary evaluation of MetaTrimmer, a method for selecting and constraining MRs based on test data. MetaTrimmer comprises three steps: generating random test data inputs for the SUT (Step 1), performing test data transformations and logging MR violations (Step 2), and conducting manual inspections to derive constraints (Step 3). The novelty of MetaTrimmer is its avoidance of complex prediction models that require labeled datasets regarding the applicability of MRs. Moreover, MetaTrimmer facilitates the seamless integration of MT with advanced fuzzing for test data generation. In a preliminary evaluation, MetaTrimmer shows the potential to overcome existing limitations and enhance MR effectiveness.

Metamorphic Testing (MT) is a testing technique that can effectively alleviate the oracle problem. MT uses Metamorphic Relations (MRs) to determine if a test case passes or fails. MRs specify how the outputs should vary in response to specific input changes when executing the System Under Test (SUT). If a particular MR is violated for at least one test input (and its change), there is a high probability that the SUT has a fault. On the other hand, if a particular MR is not violated, it does not guarantee that the SUT is fault free. However, deciding if the MR is being violated due to a bug or because the MR does not hold/fit for particular conditions generated by specific inputs remains a manual task and unexplored. In this paper, we develop a method for refining MRs to offer hints as to whether a violation results from a bug or arises from the MR not being matched to certain test data under specific circumstances. In our initial proof-of-concept, we derive the relevant information from rules using the Association Rule Mining (ARM) technique. In our initial proof-of-concept, we validate our method on a toy example and discuss the lessons learned from our experiments. Our proof-of-concept demonstrates that our method is applicable and that we can provide suggestions that help strengthen the test suite for regression testing purposes.

Testing the graphical user interface (GUI) of a system is an important part of quality assurance and to ensure end-user satisfaction. Despite numerous techniques for automated GUI testing having been proposed, there are still many open challenges for applying them in industry. In this paper, we present a GUI testing tool that we applied in industry to automate the testing of the GUI of manufacturing machinery. Our framework provides an interface to interact with Java AWT GUI components and to determine the current state of the GUI. This interface can be used by automated test scripts that are executed in nightly build at our industry partner. Moreover, we developed a GUI tool that communicates with the interface to run automated random robustness tests. We discuss the application of our tool at our industry partner and reflect on the lessons learned from it.

Open Platform Communications Unified Architecture (OPC UA) is an increasingly important standard for industrial communication, which brings up the need for testing. Still, functional or formal verification of the implementation relies on conformance testing or custom solutions. Model-based testing has been studied and applied for decades, targeting many different technologies. OPC UA already provides a defined serialization format for its interfaces, representing a simple model of the application that could be utilized for testing.In this paper, we present our work on applying model-based testing to OPC UA server applications using the serialized interface description as the model. We introduce a possible toolchain for that approach and, based on that, discuss the possibilities and shortcomings of the interface description as a model for testing.

Metamorphic testing (TM) examines the relations between inputs and outputs of test runs. These relations are known as metamorphic relations (MR). Currently, MRs are handpicked and require in-depth knowledge of the System Under Test (SUT), as well as its problem domain. As a result, the identification and selection of high-quality MRs is a challenge. Kanewala et al. suggested the Predicting Metamorphic Relations (PMR) approach for automatic prediction of applicable MRs picked from a predefined list. PMR is based on a Support Vector Machine (SVM) model using features derived from the Control Flow Graphs (CFGs) of 100 Java methods. The original study of Kanewala et al. showed encouraging results, but developing classification models from CFG-related features is costly. In this paper, we aim at developing a PMR approach that is less costly without losing performance. We complement the original PMR approach by considering other than CFG-related features. We define 21 features that can be directly extracted from source code and build several classifiers, including SVM models. Our results indicate that using the original CFG-based method-level features, in particular for a SVM with random walk kernel (RWK), achieve better predictions in terms of AUC-ROC for most of the candidate MRs than our models. However, for one of the candidate MRs, using source code features achieved the best AUC-ROC result (greater than 0.8).

Many testing tasks in practice are still done manually. Which test cases are developed, automated and selected for execution is done by experience and instinct, rather than facts and data from the current system. To change this automated tool support requires lots of different data from the system under test. One very important information is what parts of the system are reached by a given test. This coverage information can be used to select, prioritize, or remove tests for the tested system revision. In this paper, we discuss our work on recording code coverage for individual system level tests. The discussed approach has shown to work in industry and is already in use. Nonetheless, we still have several open questions that require answers to improve coverage recordings. We highlight the lessons learned from our ongoing work and discuss the open questions to encourage further research on these problems.

Abstract Software companies commonly develop and maintain variants of systems, with different feature combinations for different customers. Thus, they must cope with variability in space. Software companies further must cope with variability in time, when updating system variants by revising existing software features. Inevitably, variants evolve orthogonally along these two dimensions, resulting in challenges for software maintenance. Our work addresses this challenge with ECSEST (Extraction and Composition for Systems Evolving in Space and Time), an approach for locating feature revisions and composing variants with different feature revisions. We evaluated ECSEST using feature revisions and variants from six highly configurable open source systems. To assess the correctness of our approach, we compared the artifacts of input variants with the artifacts from the corresponding composed variants based on the implementation of the extracted features. The extracted traces allowed composing variants with 99-100% precision, as well as with 97-99% average recall. Regarding the composition of variants with new configurations, our approach can combine different feature revisions with 99% precision and recall on average. Additionally, our approach retrieves hints when composing new configurations, which are useful to find artifacts that may have to be added or removed for completing a product. The hints help to understand possible feature interactions or dependencies. The average time to locate feature revisions ranged from 25 to 250 seconds, whereas the average time for composing a variant was 18 seconds. Therefore, our experiments demonstrate that ECSEST is feasible and effective.

Highly configurable software gives developers more flexibility to meet different customer requirements and enables users to better tailor software to their needs. However, variability causes higher complexity in software and complicates many development processes, such as testing. One major challenge for testing of configurable software is adjusting tests to fit different configurations, which often has to be done manually. In our previous work, we evaluated the use of an automated reuse technique to support the reuse of existing tests for new configurations. Research on automated reuse of model variants and on applying model-based testing to configurable software encouraged us to also evaluate the automated reuse of model-based test variants. The goal is to investigate differences in applying automated reuse to the different testing paradigms. Our evaluation provides evidence for the usefulness of automated reuse for both testing paradigms. Nonetheless we found some differences in the robustness of tests to small inaccuracies of the reuse approach.

Evolving configurable software poses unique challenges, due to dependencies between the configuration knowledge and the assets that implement the system. Despite a lot of research in the area of configurable software evolution many open challenges subsist, which hamper industrial adoption of different techniques. For instance, there is not much research on the co-evolution of tests and the configuration space. A challenge for this is that the configuration space is often not clearly defined and reasoning over its evolutions and the implication from it is difficult. In this paper, we study the configuration space of the web-based bug tracker Bugzilla, a highly configurable system, replicating the analysis of previous research. In addition, we analyze the evolution of the configuration space over several versions of the system to investigate the changes that were made over the revisions spanning the development of several years and their implications and open challenges.

Most engineering tools do not provide much support for collaborating teams and today's engineering knowledge repositories lack flexibility and are limited. Engineering teams have different needs and their team members have different preferences on how and when to collaborate. These needs may depend on the individual work style, the role an engineer has, and the tasks they have to perform within the collaborating group. However, individual collaboration is insufficient and engineers need to collaborate in groups. This work presents a collaboration framework for collaborating groups capable of providing synchronous and asynchronous mode of collaboration. Additionally, our approach enables engineers to mix these collaboration modes to meet the preferences of individual group members. We evaluate the scalability of this framework using four real life large collaboration projects. These projects were found from GitHub and they were under active development by the time of evaluation. We have tested our approach creating groups of different sizes for each project. The results showed that our approach scales to support every case for the groups created. Additionally, we scouted the literature and discovered studies that support the usefulness of different groups with collaboration styles.

Robust and effective support for the detection and management of software features and their interactions is crucial for many development tasks but has proven to be an elusive goal despite extensive research on the subject. This is especially challenging for variable systems where multiple variants of a system and their features must be collectively considered. Here an important issue is the typically large number of feature interactions that can occur in variable systems. We propose a method that computes, from a set of known source code level interactions of n features, the relevant interactions involving n+1 features. Our method is based on the insight that, if a set of features interact, it is much more likely that these features also interact with additional features, as opposed to completely different features interacting. This key insight enables us to drastically prune the space of potential feature interactions to those that will have a true impact at source code level. This substantial space reduction can be leveraged by analysis techniques that are based on feature interactions (e.g Combinatorial Interaction Testing). Our observation is based on eight variable systems, implemented in Java and C, totaling over nine million LoC, with over seven thousand feature interactions.

Robust and effective support for the detection and management of features and their interactions is crucial for many software development tasks but has proven to be an elusive goal despite the extensive research and practice on the subject. Providing the required support becomes even more challenging with variable software whereby multiple variants of a system and their features must be collectively considered. An important premise to provide better support for feature interactions in variable systems is the need of a deeper understanding on how features interact at different levels starting from the source level. In this context, recent work has looked at feature interactions from different angles and for different purposes, for instance for developing performance models, extracting interfaces for maintenance or describing feature evolution patterns. However, there is a gap in understanding how features interact in fact at the source level in contrast with how features ought to interact according to variability models that describe the valid combinations of features in variable software systems. In this paper we perform an empirical study to explore this gap. We use seven case studies, implemented in Java and C, totalling over nine million LoC, and analysed over seven thousand feature interactions. Our study revealed important inconsistencies between how feature interactions occur at source level and how they are modeled, and corroborated that the majority of source level interactions involve less than three features. We discuss the implications of our findings and avenues for further research.

Software reuse has become mandatory for companies to compete and a wide range of reuse techniques are available today. However, ad hoc practices such as copying existing systems and customizing them to meet customer-specific needs are still pervasive, and are generically called clone-and-own. We have developed a conceptual framework to support this practice named ECCO that stands for Extraction and Composition for Clone-and-Own. In this paper we present our Eclipse-based tool to support this approach. Our tool can automatically locate reusable parts from previously developed products and subsequently compose a new product from a selection of desired features. The tools demonstration video can be found here: http://youtu.be/N6gPekuxU6o.

To keep pace with the increasing demand for custom-tailored software systems, companies often apply a practice called clone-and-own, whereby a new variant of a software system is built by coping and adapting existing variants. Instead of a single and configurable system, clone-and-own leads to ad hoc product portfolios of multiple yet similar variants that soon become impossible to maintain effectively. Clone-and-own has widespread industrial use because it requires no major upfront investments and is intuitive, but it lacks a methodology for systematic reuse. In this work we propose ECCO (Extraction and Composition for Clone-and-Own), a novel approach to enhance clone and-own that actively supports the development and maintenance of software product variants. A software engineer selects the desired features and ECCO finds the proper software artifacts to reuse and then provides guidance during the manual completion by hinting which software artifacts may need adaptation. We evaluated our approach on 6 case studies, covering 402 variants having up to 344KLOC, and found that precision and recall of composed products quickly reach a near optimum (>95% reuse).