Metamorphic Testing (MT) addresses the test oracle problem by defining how program outputs should change in response to specific input changes. The relations between input changes and their corresponding output changes are called Metamorphic Relations (MRs). Generating suitable MRs is complex and often requires deep domain knowledge. Our previous work introduced MetaTrimmer, a test-data-driven approach for selecting and constraining MRs, involving three steps: Test Data (TD) Generation, MT Process, and MR Analysis. MR Analysis is done to decide whether the violation of an MR for a specific input data pair (original and changed) indicates a failure or simply means that the MR does not apply for the chosen inputs. In this paper, we present an association-rule-based approach that semi-automatically extracts constraints dividing the input space into valid/invalid data during the MR Analysis step of MetaTrimmer. We validate our approach using 44 methods to which six predefined MRs are applied. Our results indicate that the proposed method efficiently identifies correct input data space constraints. More studies are needed to provide additional evidence that MetaTrimmer with the enhanced MR Analysis step is scalable and generalisable.

Regression testing is crucial in ensuring that pure code refactoring does not adversely affect existing software functionality, but it can be expensive, accounting for half the cost of software maintenance. Automated test case generation reduces effort but may generate weak test suites. Test amplification is a promising solution that enhances tests by generating additional or improving existing ones, increasing test coverage, but it faces the test oracle problem. To address this, we propose a test oracle derivation approach that uses object state data produced during System Under Test (SUT) test execution to amplify regression test oracles. The approach monitors the object state during test execution and compares it to the previous version to detect any changes in relation to the SUT’s intended behaviour. Our preliminary evaluation shows that the proposed approach can enhance the detection of behaviour changes substantially, providing initial evidence of its effectiveness.

Identifying and selecting high-quality Metamorphic Relations (MRs) is a challenge in Metamorphic Testing (MT). While some techniques for automatically selecting MRs have been proposed, they are either domain-specific or rely on strict assumptions about the applicability of a pre-defined MRs. This paper presents a preliminary evaluation of MetaTrimmer, a method for selecting and constraining MRs based on test data. MetaTrimmer comprises three steps: generating random test data inputs for the SUT (Step 1), performing test data transformations and logging MR violations (Step 2), and conducting manual inspections to derive constraints (Step 3). The novelty of MetaTrimmer is its avoidance of complex prediction models that require labeled datasets regarding the applicability of MRs. Moreover, MetaTrimmer facilitates the seamless integration of MT with advanced fuzzing for test data generation. In a preliminary evaluation, MetaTrimmer shows the potential to overcome existing limitations and enhance MR effectiveness.

Metamorphic Testing (MT) addresses the test oracle problem by examining the relationships between input-output pairs in consecutive executions of the System Under Test (SUT). These relations, known as Metamorphic Relations (MRs), specify the expected output changes resulting from specific input changes. However, achieving full automation in generating, selecting, and understanding MR violations poses challenges. Our research aims to develop methods and tools that assist testers in generating MRs, defining constraints, and providing explainability for MR outcomes. In the MR generation phase, we explore automated techniques that utilise a domain-specific language to generate and describe MRs. The MR constraint definition focuses on capturing the nuances of MR applicability by defining constraints. These constraints help identify the specific conditions under which MRs are expected to hold. The evaluation and validation involve conducting empirical studies to assess the effectiveness of the developed methods and validate their applicability in real-world regression testing scenarios. Through this research, we aim to advance the automation of MR generation, enhance the understanding of MR violations, and facilitate their effective application in regression testing.

Metamorphic Testing (MT) is a testing technique that can effectively alleviate the oracle problem. MT uses Metamorphic Relations (MRs) to determine if a test case passes or fails. MRs specify how the outputs should vary in response to specific input changes when executing the System Under Test (SUT). If a particular MR is violated for at least one test input (and its change), there is a high probability that the SUT has a fault. On the other hand, if a particular MR is not violated, it does not guarantee that the SUT is fault free. However, deciding if the MR is being violated due to a bug or because the MR does not hold/fit for particular conditions generated by specific inputs remains a manual task and unexplored. In this paper, we develop a method for refining MRs to offer hints as to whether a violation results from a bug or arises from the MR not being matched to certain test data under specific circumstances. In our initial proof-of-concept, we derive the relevant information from rules using the Association Rule Mining (ARM) technique. In our initial proof-of-concept, we validate our method on a toy example and discuss the lessons learned from our experiments. Our proof-of-concept demonstrates that our method is applicable and that we can provide suggestions that help strengthen the test suite for regression testing purposes.

Metamorphic testing (TM) examines the relations between inputs and outputs of test runs. These relations are known as metamorphic relations (MR). Currently, MRs are handpicked and require in-depth knowledge of the System Under Test (SUT), as well as its problem domain. As a result, the identification and selection of high-quality MRs is a challenge. Kanewala et al. suggested the Predicting Metamorphic Relations (PMR) approach for automatic prediction of applicable MRs picked from a predefined list. PMR is based on a Support Vector Machine (SVM) model using features derived from the Control Flow Graphs (CFGs) of 100 Java methods. The original study of Kanewala et al. showed encouraging results, but developing classification models from CFG-related features is costly. In this paper, we aim at developing a PMR approach that is less costly without losing performance. We complement the original PMR approach by considering other than CFG-related features. We define 21 features that can be directly extracted from source code and build several classifiers, including SVM models. Our results indicate that using the original CFG-based method-level features, in particular for a SVM with random walk kernel (RWK), achieve better predictions in terms of AUC-ROC for most of the candidate MRs than our models. However, for one of the candidate MRs, using source code features achieved the best AUC-ROC result (greater than 0.8).

Metamorphic Testing (MT) addresses the test oracle problem by examining the relations between inputs and outputs of test executions. Such relations are known as Metamorphic Relations (MRs). In current practice, identifying and selecting suitable MRs is usually a challenging manual task, requiring a thorough grasp of the SUT and its application domain. Thus, Kanewala et al. proposed the Predicting Metamorphic Relations (PMR) approach to automatically suggest MRs from a list of six pre-defined MRs for testing newly developed methods. PMR is based on a classification model trained on features extracted from the control-flow graph (CFG) of 100 Java methods. In our replication study, we explore the generalizability of PMR. First, since not all details necessary for a replication are provided, we rebuild the entire preprocessing and training pipeline and repeat the original study in a close replication to verify the reported results and establish the basis for further experiments. Second, we perform a conceptual replication to explore the reusability of the PMR model trained on CFGs from Java methods in the first step for functionally identical methods implemented in Python and C++. Finally, we retrain the model on the CFGs from the Python and C++ methods to investigate the dependence on programming language and implementation details. We were able to successfully replicate the original study achieving comparable results for the Java methods set. However, the prediction performance of the Java-based classifiers significantly decreases when applied to functionally equivalent Python and C++ methods despite using only CFG features to abstract from language details. Since the performance improved again when the classifiers were retrained on the CFGs of the methods written in Python and C++, we conclude that the PMR approach can be generalized, but only when classifiers are developed starting from code artefacts in the used programming language.

Data Centre Networks (DCNs) handle large volumes of data transmission that can consume a lot of bandwidth in short bursts or over prolonged periods of time. One class of traffic that constantly poses a challenge is Heavy-Hitter (HH) flows - large-volume flows that consume considerably more network resources than other flows combined. The identification of such flows is critical to prevent network congestion and overall network performance degradation. Most of the existing methods to identify HHs are based on thresholds, i.e., if the flow exceeds a predefined threshold, it will be marked as a HH; otherwise, it will be classified as a non-HH. However, these approaches present two significant issues. First, there is no consistent and accepted threshold that would reliably classify flows. Second, the existing threshold approaches use counters (duration, packets, and bytes); thus their accuracy depends on how complete the flow information is. In this paper, we address those issues using per-flow packet size distribution which can capture the behaviour and dynamics of network traffic flow more accurately than the counters in the early stage of the flow. We then propose the use of the template matching technique to identify HHs and achieved a classification accuracy of 96% using only the first 14 packets of a flow.