Abstract Metamorphic testing (MT) is an effective testing technique having a broad range of applications. One key task for MT is the identification of metamorphic relations (MRs), which is a fundamental mechanism in MT and is critical to the automation of MT. Prior studies have proposed approaches for predicting MRs (PMR). One major idea behind these PMR approaches is to represent program source code information via manually designed code features and then to apply machine‐learning–based classifiers to automatically predict whether a specific MR can be applied on the target program. Nevertheless, the human‐involved procedure of selecting and extracting code features is costly, and it may not be easy to obtain sufficiently comprehensive features for representing source code. To overcome this limitation, in this study, we explore and evaluate the effectiveness of code representation learning techniques for PMR. By applying neural code representation models for automatically mapping program source code to code vectors, the PMR procedure can be boosted with learned code representations. We develop 32 PMR instances by, respectively, combining 8 code representation models with 4 typical classification models and conduct an extensive empirical study to investigate the effectiveness of code representation learning techniques in the context of MR prediction. Our findings reveal that code representation learning can positively contribute to the prediction of MRs and provide insights into the practical usage of code representation models in the context of MR prediction. Our findings could help researchers and practitioners to gain a deeper understanding of the strength of code representation learning for PMR and, hence, pave the way for future research in deriving or extracting MRs from program source code.

The Multi-Agent Path Finding (MAPF) problem, i.e., the scheduling of multiple agents to reach their destinations, has been widely investigated. Testing MAPF systems is challenging, due to the complexity and variety of scenarios and the agents’ distribution and interaction. Moreover, MAPF testing suffers from the oracle problem, i.e., it is not always clear whether a test shows a failure or not. Indeed, only considering whether the agents reach their destinations without collision is not sufficient. Other properties related to the ‘quality’ of the generated paths should be assessed, e.g., an agent should not follow an unnecessarily long path. To tackle this issue, this article proposes MET-MAPF, a Metamorphic Testing approach for MAPF systems. We identified 10 Metamorphic Relations (MRs) that a MAPF system should guarantee, designed over the environment in which agents operate, the behaviour of the single agents and the interactions among agents. Starting from the different MRs, MET-MAPF automatically generates test cases addressing them, so possibly exposing different types of failures. Experimental results show that MET-MAPF can indeed find MR violations not exposed by approaches that only consider the completion of the mission as test oracle. Moreover, experiments show that different MRs expose different types of violations.

Due to the prevalent application of machine learning (ML) techniques and the intrinsic black-box nature of ML models, the need for good explanations that are sufficient and necessary towards a model’s prediction has been well recognized and emphasized. Existing explanation approaches, however, favor either the sufficiency or necessity. To fill this gap, we present DDImage, a technique and tool that automatically produces explanations preserving dual properties for ML-based image classifiers. The core idea behind DDImage is to discover an appropriate explanation by debugging the given input image via a series of image reductions, with respect to the sufficiency and necessity properties. We conduct comprehensive experiments to compare our approach against two state-of-the-art approaches, BayLIME and SEDC, on widely-used models and datasets. The results show that our approach outperforms the other methods in producing minimal explanations preserving both sufficiency and necessity, and it matches or exceeds the other methods in terms of stability.

The crossover operation is an important component of genetic algorithms for test case generation. The usefulness of crossover, however, is not really clear, especially for industrial settings. This research applies EvoSuite, a well-known searchbased test case generator, to both open-source and industrial Java programs, and investigates the impact of the use of crossover on the code coverage achieved by the generated test cases. Our empirical study shows that the usefulness of the crossover operation varies for open-source and industrial code, and that the crossover operation may not necessarily be good, especially for some industrial programs. We further analyze (un)favorable conditions/code features for the use of crossover operations, providing hints for effective test case generation.

Deep neural networks (DNNs) have achieved a major success in solving challenging tasks such as social networks analysis and image classification. Despite the prosperous development of DNNs, recent research has demonstrated the feasibility of exploiting DNNs using adversarial examples, in which a small distortion is added into the input data to largely mislead prediction of DNNs.Determining the similarity of two binary codes is the foundation for many reverse engineering, re-engineering, and security applications. Currently, the majority of binary code matching tools are based on DNNs, the dependability of which has not been completely studied. In this research, we present an attack that perturbs software in executable format to deceive DNN-based binary code matching. Unlike prior attacks which mostly change non-functional code components to generate adversarial programs, our approach proposes the design of several semantics-preserving transformations directly toward the control flow graph of binary code, making it particularly effective to deceive DNNs. To speedup the process, we design a framework that leverages gradient- or hill climbing-based optimizations to generate adversarial examples in both white-box and black-box settings. We evaluated our attack against two popular DNN-based binary code matching tools, asm2vec and ncc, and achieve reasonably high success rates. Our attack toward an industrial-strength DNN-based binary code matching service, BinaryAI, shows that the proposed attack can fool remote APIs in challenging black-box settings with a success rate of over 16.2% (on average). Furthermore, we show that the generated adversarial programs can be used to augment robustness of two white-box models, asm2vec and ncc, reducing the attack success rates by 17.3% and 6.8% while preserving stable, if not better, standard accuracy.

Abstract Named entity recognition (NER) is a widely used natural language processing technique; it plays a key role in information extraction from sentences. To be able to test the correctness of NER systems is important, but it is expensive because an automated test oracle is normally unavailable. To address the oracle problem, this study proposes to apply metamorphic testing (MT). The authors conduct a case study with Litigant, an industrial NER system of the Ant Group, and show that MT can effectively detect real‐life bugs in the absence of an ideal oracle. The authors further investigate the causes for a series of entity recognition failures detected. Outcomes of this research further justify the application of MT to the natural language processing domain as well as provide hints for practitioners to improve the quality process of their NER systems.

Visual question answering (VQA) takes an image and a natural-language question as input and returns a natural-language answer. To date, VQA models are primarily assessed by their accuracy on high-level reasoning questions. Nevertheless, Given that perception tasks (e.g., recognizing objects) are the building blocks in the compositional process required by high-level reasoning, there is a demanding need to gain insights into how much of a problem low-level perception is. Inspired by the principles of software metamorphic testing, we introduce MetaVQA, a model-agnostic framework for benchmarking perception capability of VQA models. Given an image i, MetaVQA is able to synthesize a low-level perception question q. It then jointly transforms (i, q) to one or a set of sub-questions and sub-images. MetaVQA checks whether the answer to (i, q) satisfies metamorphic relationships (MRs), denoting perception consistency, with the composed answers of transformed questions and images. Violating MRs denotes a failure of answering perception questions. MetaVQA successfully detects over 4.9 million perception failures made by popular VQA models with metamorphic testing. The state-of-the-art VQA models (e.g., the champion of VQA 2020 Challenge) suffer from perception consistency problems. In contrast, the Oscar VQA models, by using anchor points to align questions and images, show generally better consistency in perception tasks. We hope MetaVQA will revitalize interest in enhancing the low-level perceptual abilities of VQA models, a cornerstone of high-level reasoning.

Spectrum-Based Fault Localisation (SBFL) is a well-known technique to find faulty statements in a program. To date, various techniques aiming to improve SBFL from different aspects have been proposed, following their own theories and assumptions. Therefore, it is challenging to make a fair assessment of their rationale and practicability. In this paper, we propose the SPectra Illustration for Comprehensive Analysis (SPICA), a methodology for reviewing and analysing existing SBFL works using spectrum visualisation. Specifically, taking as input a specific SBFL technique (e.g., a suspiciousness metric), SPICA illustrates the relevant artefacts within the spectrum space and then analyse the visualised spectra distribution following the steps of 1) examining the Geometric Characteristics (GCs) and 2) knowledge mining. In this way, we can do overall reviews for various SBFL techniques and get analysis results. As examples, we use SPICA to analyse five representative SBFL techniques, which provide fundamental theories or experimental results. Finally, we provide an overall assessment of the rationale for each technique, attached with suggestions that could be useful for future validation and extension.

Natural language inference (NLI) is a fundamental NLP task that forms the cornerstone of deep natural language understanding. Unfortunately, evaluation of NLI models is challenging. On one hand, due to the lack of test oracles, it is difficult to automatically judge the correctness of NLI's prediction results. On the other hand, apart from knowing how well a model performs, there is a further need for understanding the capabilities and characteristics of different NLI models. To mitigate these issues, we propose to apply the technique of metamorphic testing (MT) to NLI. We identify six categories of metamorphic relations, covering a wide range of properties that are expected to be possessed by NLI task. Based on this, MT can be conducted on NLI models without using test oracles, and MT results are able to interpret NLI models' capabilities from varying aspects. We further demonstrate the validity and effectiveness of our approach by conducting experiments on five NLI models. Our experiments expose a large number of prediction failures from subject NLI models, and also yield interpretations for common characteristics of NLI models.

Test-suite-based automated program repair (APR) techniques acquire information from an input test suite to guide the repair process, aiming to produce a repair that can pass all test cases of the input test suite. Obviously, the input test suite has a critical impact on the repair effectiveness of APR techniques. This article reports on a study of the APR input test suites from a new perspective. We first propose a novel method of constructing the APR input test suites, using information derived from violated metamorphic relations. We then empirically evaluate our construction method using three APR techniques (Angelix, CETI, and GenProg), comparing it with random and code-coverage-based construction methods that are used as the experimental control. The results show that our approach is complementary to these two input test suite construction methods. This article illustrates a new use of metamorphic relations for program repair.

Code search engines are widely used by software developers. However, due to the huge amount of data being processed as well as the lack of complete specifications, the testing of code search engines faces the oracle problem. Metamorphic testing (MT) is a well-known testing technique that is effective for alleviating the oracle problem. In this paper, we propose to apply MT to test code search engines. We first identify five metamorphic relations (MRs) by considering the characteristics of code search engines. Four MRs are defined based on the partial specifications of code search engines and thus are used for the purpose of verification, and the one MR identified from the users' perspective can be used to conduct validation. The approach is evaluated by conducting a series of experiments involving four popular code search engines (namely, Krugle, searchcode, sourcegraph and Zoekt), and one well-known code repository (namely, GitHub) that provides the code search services. The experimental results show that the abnormal behaviors of Krugle, searchcode, GitHub, and Zoekt have been successfully detected. By further inspecting the root causes of these abnormal behaviors, three critical issues related with the code search engines under investigation are identified and reported. These results demonstrate the effectiveness of our approach, and are also helpful for the users and developers to gain a deeper understanding about the code search engines.

A software system's requirements are often specified by textual use cases due to the latter's concrete and narrative style of expressions. However, they have limitation in the synthesis of the system behavior since they have a poor basis for the formal interpretation. Existing synthesis techniques are either largely manual or focus on the use case interactions. We present a framework from a model-based point of view to automatically synthesize system behavior from textual use cases to a Petri net model. The generated net model can well describe component module interactions and thus can be used to check the requirement properties. The function of Send-Railway-Emergency-Call of European Integrated Railway Radio Enhanced Network is used to show the proposed method. Moreover, the experimental results on a set of examples demonstrate the effectiveness of the proposed method.

Switched stochastic systems (SSS) can be used to describe hybrid systems with randomness. However, the languages to describe their discrete switching logic and stochastic dynamic processes are different, and this difference makes their design and analysis hard. This paper proposes a new Petri net model, namely stochastic-differential Petri net (S-DPN), to describe both discrete switching logic, represented by a Markov chain, and stochastic dynamic processes, represented by a set of stochastic differential equations. We then apply a model checking technique to S-DPN to check the correctness of the requirements of SSS. A temperature control system is used to demonstrate the effectiveness of our method.

Web service composition is a distributed model to construct new web service on top of existing primitive or other composite web services. However, current service technologies, including proposed composition languages, do not address the reliability of web service composition. Thus, it is hard to predict the system reliability. In this paper, we propose a method to compute system reliability based on Service Component Architecture (SCA), a standard that provides a language-independent way to define and compose service components in the system. We first present a formal service component signature model with respect to the specification of the SCA assembly model, and then propose a language-independent dynamic behavior model for specifying the interface behavior of the service component by port activities. Then, the failure behaviors of ports are defined through the Nonhomogeneous Poisson Process (NHPP). Based on the semantics of ports, several rules have been generated to compute reliability of port expressions, thus the overall system reliability can be automatically computed. An Online Shop example from IBM web site is given to illustrate our method, together with a testing bed to calculate port reliability.