Abstract In the discovery of modern software vulnerability, mutation-based fuzzing techniques are widely applied, with their performance highly dependent on the effectiveness of mutation scheduling strategies. Most current research primarily focuses on optimizing seed scheduling. However, mutation scheduling plays an equally critical role in fuzzing, as it determines how mutation operators are selected and applied to generate new test cases. Existing mutation scheduling schemes are faced with several issues, such as the need to input manual parameters from users and improper overhead management. To address these challenges, this paper proposes an innovative mutation scheduling model, HavocEDA, utilizing Estimation of Distribution Algorithm to optimize the selection of mutation operators in fuzzing, thereby enhancing the efficiency of vulnerability detection. The HavocEDA model can dynamically adjust the probability distribution of mutation operators based on fuzzing feedback, enabling a more efficient exploration of potential vulnerabilities within the software. We implemented a prototype based on the popular general-purpose fuzzer AFL. We evaluated HavocEDA on 9 open-source Linux programs in FuzzBench. Experimental results indicate that HavocEDA shows a performance improvement in edge coverage and crash discovery compared with the state-of-the-art fuzzers AFL, DARWIN, MOpt, and HavocMAB.

Neural machine translation (NMT) systems have been widely deployed in real-world applications. However, despite the great translation performance, these systems inevitably face robustness issues and sometimes produce erroneous outputs, particularly in complex and ambiguous scenarios. In this work, we propose a Fuzzing framework with Differential Testing for NMT systems, namely, DiFuzzNMT. DiFuzzNMT employs a heuristic strategy to continuously search inputs that produce greater output differences across various NMT systems for error detection. Specifically, DiFuzzNMT establishes token mappings between outputs of different NMT systems for the same input through word alignment. To guide the test input generation process of fuzzing, we design specific testing guidance that takes into account the differences between outputs, including word alignment differences and token semantic differences. All the test inputs are generated based on “seed” inputs (inputs to generate new inputs) by applying a mutation operator. Test inputs exhibiting higher testing guidance values are selected as new seeds, while the others are discarded. A potential translation error is reported when the same test input exhibits significant differences across different NMT systems. By iteratively retaining seeds and generating test inputs, DiFuzzNMT can effectively detect translation errors. To evaluate the effectiveness of DiFuzzNMT, we conduct experiments on two widely used NMT APIs (Baidu Translate and Tencent Translate), using a publicly available dataset of 800 original sentences across 8 thematic categories. The experimental results show that DiFuzzNMT detects more translation errors than baselines and exhibits greater diversity. Furthermore, the results show that the proposed testing guidance improves the method's ability to detect translation errors.

ABSTRACT With the increasingly frequent attacks on software systems, software security is an issue that must be addressed. Within software security, automated detection of software vulnerabilities is an important subject. Most existing vulnerability detectors rely on the features of a single code type (e.g., source code or intermediate representation [IR]), which may lead to both the global features of the code slices and the memory operation information not being captured or considered. In particular, vulnerability detection based on source‐code features cannot usually include some macro or type definition content. In this paper, we propose a vulnerability‐detection method that combines the semantic features of source code and the low level virtual machine (LLVM) IR. Our proposed approach starts by slicing (C/C++) source files using improved slicing techniques to cover more comprehensive code information. It then extracts semantic information from the LLVM IR based on the executable source code. This can enrich the features fed to the artificial neural network (ANN) model for learning. We conducted an experimental evaluation using a publicly‐available dataset of 11,381 C/C++ programs. The experimental results show the vulnerability‐detection accuracy of our proposed method to reach over 96% for code slices generated according to four different slicing criteria. This outperforms most other compared detection methods.

ABSTRACT Deep neural networks (DNNs) possess potent feature learning capability, enabling them to comprehend natural language, which strongly support developing dialogue systems. However, dialogue systems usually perform incorrect behaviours in some corner cases, which may cause misunderstanding or economic loss. To test and debug dialogue systems, a popular fuzzing framework by metamorphic testing with Gini impurity guidance is proposed, namely, DialTest. However, DialTest treats all seeds (the initial test inputs to generate the mutated test inputs) equally during the fuzzing process and does not differentiate seeds, resulting in a certain limitation to its incorrect behaviour detection capability. In this paper, we propose to enhance the DialTest by applying a lightweight energy adjustment strategy called DialTest with Energy Adjustment (DialTest‐EA). DialTest‐EA employs the ant colony optimization algorithm (ACO) to adjust the mutation energy of each seed adaptively, ensuring that potential seeds have more opportunities to generate subsequent test inputs. To evaluate the effectiveness of the proposed DialTest‐EA, we conduct a series of comparisons with the original DialTest and random mutation strategy. The experimental results show that the proposed DialTest‐EA outperforms the compared methods both in the intent detection and slot filling tasks. Compared with the original DialTest, the intent detection accuracy of generated test cases by the proposed method is reduced by more than 14%, and the slot filling accuracy is reduced by more than 8%.

Abstract Regression testing is an essential and expensive process in software testing. However, there may be insufficient resources for the execution of all test cases during regression testing. Test case prioritization (TCP) techniques improve the efficiency of regression testing by adjusting the test case execution sequence. Traditional TCP techniques usually rely on the historical execution information of the software under test for more efficient results. String distance‐based TCP (SD‐TCP) avoids these limitations; it uses only the textual difference information of the test cases themselves for prioritization. However, the time overhead on the sorting process of this method is not ideal, and the extreme test case inputs have an impact on the stability of the method. To address these problems, we propose a novel test case prioritization strategy, it first classifies the test cases more finely using the K‐medoids algorithm and then transforms the set into subsequences and improves the early diversity by greedy sorting within clusters. Finally, the test cases are selected through a polling strategy to compose the execution sequence. Extensive experimental results demonstrate that the proposed approach outperforms SD‐TCP in better time efficiency on test case prioritization; it also has a higher average percentage of fault detected (APFD) value than random prioritization (RP) and SD‐TCP.

There is a severe class imbalance problem in defect datasets, with nondefective data dominating the distribution, making it easy to generate inaccurate software defect prediction models. Ensemble learning has been proven to be one of the best methods to solve class imbalance problem. Traditional ensemble prediction models usually ensemble the results of several base classifiers simply, and most of them only ensemble once, rarely consider the diversity of ensemble or the combination of ensemble learning and neural network. In order to explore whether the secondary ensemble of classifiers based on a diversity ensemble combined with neural network can improve the performance of defect prediction model, in this article, we propose a novel dual ensemble software defect prediction (DE-SDP) approach based on a diversity ensemble combined with neural network. In the first ensemble, we use cross-validation to build different subclassifiers, then, these subclassifiers are used to establish base ensemble classifiers with weighted average method. Through seven classification algorithms, seven base ensemble classifiers can be established. In the second ensemble, a neural network model and stacking are used to ensemble the base ensemble classifiers again. We have evaluated DE-SDP against other ensemble defect prediction methods on eight datasets of NASA MDP. The results show that our approach is superior to other ensemble approaches and effectively improves the performance of defect prediction model.

Adaptive Random Testing(ART) enhances the testing effectiveness (including fault-detection capability) ofRandom Testing(RT) by increasing the diversity of the random test cases throughout the input domain. Many ART algorithms have been investigated such asFixed-Size-Candidate-Set ART(FSCS) andRestricted Random Testing(RRT), and have been widely used in many practical applications. Despite its popularity, ART suffers from the problem of high computational costs during test-case generation, especially as the number of test cases increases. Although several strategies have been proposed to enhance the ART testing efficiency, such as theforgetting strategyand thek-dimensional tree strategy, these algorithms still face some challenges, including: (1) Although these algorithms can reduce the computation time, their execution costs are still very high, especially when the number of test cases is large; and (2) To achieve low computational costs, they may sacrifice some fault-detection capability. In this paper, we propose an approach based onApproximate Nearest Neighbors(ANNs), calledLocality-Sensitive Hashing ART(LSH-ART). When calculating distances among different test inputs, LSH-ART identifies the approximate (not necessarily exact) nearest neighbors for candidates in an efficient way. LSH-ART attempts to balance ART testing effectiveness and efficiency.

This document1is an extended abstract of a Science of Computer Programming paper, "Candidate Test Set Reduction for Adaptive Random Testing: An Overheads Reduction Technique," presented as a J1C2 (Journal publication first, Conference presentation following) at the 30th IEEE International Conference on Software Analysis, Evolution and Reengineering (Saner 2023).The paper presents a candidate set reduction strategy to enhance the Fixed-Sized-Candidate-Set version of Adaptive Random Testing (FSCS-ART). The proposed method reduces the number of randomly-generated candidate test cases by retaining valuable, unused candidates from previous iterations. As the computational costs associated with a stored/retained candidate are less than the costs associated with a randomly-generating one, the overall computational overheads of FSCS-ART are reduced. The reported experimental studies show that the proposed method has a comparable failure-detection effectiveness to FSCS-ART, but less computational overheads.

With the rapid growth of Artificial Intelligence, dialogue systems have become increasingly powerful. Though Recurrent Neural Network power the dialogue systems, it also bring challenges to the systems’ testing. In order to ensure the safety of these systems, which we must pay attention to, DialTest showed up. DialTest broke the traditional test methods, it made innovation at many levels. We have to acknowledge this great contribution. However, DialTest has a smattering of shortcomings. It treats all seeds as equal, implying that it cannot adjust the energy assignment quickly, resulting in energy waste. Moreover, DialTest’s mutant sentences generated by a few original seed sentences in the late stage of variation. This paper presents an improved DialTest with an adaptive mutation schedule, we called it EcoDialTest. EcoDialTest divides all the seed into three states, different states have different energy distribution strategies. We devise a new mutation strategy to improve the effectiveness and dependability of the seeds in the transformed seed set. All of these were implemented based on DialTest, we still adopt DeepGini impurity as the main guidance to guide the test generation process and utilize the three mutation operators as it does. Through ATIS, Snips and Facebook datasets, EcoDialTest was evaluated by two state-of-the-art models in the experiment. According to the result, we found that EcoDialTest attained lower values in both intent accuracy and slot accuracy than DialTest.

Abstract Existing association-based outlier detection approaches were proposed to seek for potential outliers from huge full set of uncertain data streams ($UDS$), but could not effectively process the small scale of $UDS$ that satisfies preset constraints; thus, they were time consuming. To solve this problem, this paper proposes a novel minimal rare pattern-based outlier detection approach, namely Constrained Minimal Rare Pattern-based Outlier Detection (CMRP-OD), to discover outliers from small sets of $UDS$ that satisfy the user-preset succinct or convertible monotonic constraints. First, two concepts of ‘maximal probability’ and ‘support cap’ are proposed to compress the scale of extensible patterns, and then the matrix is designed to store the information of each valid pattern to reduce the scanning times of $UDS$, thus decreasing the time consumption. Second, more factors that can influence the determination of outlier are considered in the design of deviation indices, thus increasing the detection accuracy. Extensive experiments show that compared with the state-of-the-art approaches, CMRP-OD approach has at least 10% improvement on detection accuracy, and its time cost is also almost reduced half.

Summary Combinatorial testing (CT) is considered as a practical approach to detect software faults, which has arisen from the interaction between factors affecting the software behavior. However, most of the traditional algorithms on CT generation did not take advantage of the execution results of the earlier test cases, as well as neglect the impact of the nonequilibrium input parameter model (NE‐IPM) effect on redundant test cases, which bring a deleterious effect to the detection accuracy of the software faults. To solve these problems, we propose a novel CT approach with fuzzing strategy called CTAF. Based on the idea that fuzzing is performed during execution, CTAF exploits the execution results of earlier tests to provide guidance for subsequent test generation thereby reducing the redundant test cases without compromising the diversity of test cases. And then, we designed three experiments on real subjects of six open source software systems, and the experimental results show that the proposed CTAF approach can effectively improve the NE‐IPM effect and enhance the detection accuracy of software faults.

Coverage-guided Greybox fuzzing is regarded as a practical approach to detect software vulnerabilities, which targets to expand code coverage as much as possible. A common implementation is to assign more energy to such seeds which find new edges with less execution time. However, solely considering new edges may be less effective because some hard-to-find branches often exist in the complex code of program. Code complexity is one of the key indicators to measure the code security. Compared to the code with simple structure, the program with higher code complexity is more likely to find more branches and cause security problems. In this paper, we propose a novel fuzzing method which further uses code complexity to optimize power schedule process in AFL (American Fuzzy Lop) and AFLFAST (American Fuzzy Lop Fast). The goal of our method is to generate inputs which are more biased toward the code with higher complexity of the program under test. In addition, we conduct a preliminary empirical study under three widely used real-world programs, and the experimental results show that the proposed approach can trigger more crashes as well as improve the coverage discovery.

C has been regarded as a dominant programming language for system software implementation. Meanwhile, it often suffers from various memory vulnerabilities due to its low-level memory control. Quite massive approaches are proposed to enhance memory security, among which Coverage-based Greybox Fuzzing (CGF) is very popular because of its practicality and satisfactory effectiveness. However, CGF identifies vulnerabilities based on the catched crashes, thus cannot detect vulnerabilities with non-crash. In this paper, we consider to trace pointer metadata (status, bounds and referents) to detect more various vulnerabilities. Additionally, since pointers in C are often directly related to memory operations, we design two standards to further use pointer metadata as the guidance of CGF, making fuzzing process target to the vulnerable part of programs.

Cross-project software defect prediction solves the problem of insufficient training data for traditional defect prediction, and overcomes the challenge of applying models learned from multiple different source projects to target project. At the same time, two new problems emerge: (1) too many irrelevant and redundant features in the model training process will affect the training efficiency and thus decrease the prediction accuracy of the model; (2) the distribution of metric values will vary greatly from project to project due to the development environment and other factors, resulting in lower prediction accuracy when the model achieves cross-project prediction. In the proposed method, the Pearson feature selection method is introduced to address data redundancy, and the metric compensation based transfer learning technique is used to address the problem of large differences in data distribution between the source project and target project. In this paper, we propose a software defect prediction method with metric compensation based on feature selection and transfer learning. The experimental results show that the model constructed with this method achieves better results on area under the receiver operating characteristic curve (AUC) value and F1-measure metric.

Adaptive random testing (ART) improves the failure-detection effectiveness of Random testing (RT) by making test cases more evenly distributed in the input domain. The Fixed-size-Candidate-set ART (FSCS-ART) is one of the most classical algorithms, which selects the candidate test case furthest from the previously executed test case as the next test case. However, when the number of executed test cases is large, the computational overhead will be very high. In this paper, we propose an enhanced version of FSCS-ART based on a modified Metric-Memory tree (MM-tree), namely Fixed-size-Candidate-set ART based on the modified MM-tree (MMFC-ART). Simulations and empirical studies are conducted to verify the effectiveness and efficiency of MMFC-ART. The experimental results indicate that MMFC-ART significantly reduces the computational overhead while ensuring comparable or better failure-detection effectiveness than FSCS-ART. Meanwhile, compared with KD-tree-enhanced Fixed-size-Candidate-set ART (KDFC-ART), MMFC-ART has better performance in high dimensions in terms of efficiency. In terms of effectiveness, MMFC-ART has better failure-detection effectiveness in some scenarios. Overall, MMFC-ART is cost-effective compared to FSCS-ART and KDFC-ART.

Abstract Combinatorial interaction testing (CIT) aims at constructing a covering array (CA) of all value combinations at a specific interaction strength, to detect faults that are caused by the interaction of parameters. CIT has been widely used in different applications, with many algorithms and tools having been proposed to support CA construction. To date, however, there appears to have been no studies comparing different CA constructors when only some of the CA test cases are executed. In this paper, we present an investigation of five popular CA constructors: ACTS, Jenny, PICT, CASA and TCA. We conducted empirical studies examining the five programs, focusing on interaction coverage and fault detection. The experimental results show that when there is no preference or special justification for using other CA constructors, then Jenny is recommended—because it achieves better interaction coverage and fault detection than the other four constructors in many cases. Our results also show that when using ACTS or CASA, their CAs must be prioritized before testing. The main reason for this is that these CAs can result in considerable interaction coverage or fault detection capabilities when executing a large number of test cases; however, they may also produce the lowest rates of fault detection and interaction coverage.

Random testing (RT) is a well-studied testing method that has been widely applied to the testing of many applications, including embedded software systems, SQL database systems, and Android applications. Adaptive random testing (ART) aims to enhance RT's failure-detection ability by more evenly spreading the test cases over the input domain. Since its introduction in 2001, there have been many contributions to the development of ART, including various approaches, implementations, assessment and evaluation methods, and applications. This paper provides a comprehensive survey on ART, classifying techniques, summarizing application areas, and analyzing experimental evaluations. This paper also addresses some misconceptions about ART, and identifies open research challenges to be further investigated in the future work.

With the progressive development of web applications and the urgent requirement of web security, vulnerability scanner has been particularly emphasized, which is regarded as a fundamental component for web security assurance. Various scanners are developed with the intention of that discovering the possible vulnerabilities in advance to avoid malicious attacks. However, most of them only focus on the vulnerability detection with single target, which fail in satisfying the efficiency demand of users. In this paper, an effective web vulnerability scanner that integrates the information collection with the vulnerability detection is proposed to verify whether the target web application is vulnerable or not. The experimental results show that, by guiding the detection process with the useful collected information, our tool achieves great web vulnerability detection capability with a large scanning scope.

Internet of Things(IoT) have been widely implemented in diverse domains of real-life, and become one of the most popular applications of the internet. Nevertheless, the development of IoT has suffered from its security issues so far. Various IoT vulnerabilities bring serious risks to the privacy and the property security of users. To study the security vulnerabilities in depth, the classification for IoT vulnerabilities becomes a basic requirement. However, manual classification relies on human experience and is very laborious. In this paper, an IoT vulnerabilities classification system based on Support Vector Machines(SVM) and Particle Swarm optimization(PSO) is developed to identify and classify IoT vulnerabilities automatically. The experimental results prove that our system presents great potential of vulnerability classification.