Deep Neural Network (DNN) testing has emerged as an effective way of uncovering erroneous behaviors in DNN models and further enhancing their performance. Research on test input generation has gained much attention from both researchers and practitioners, aiming to expose faults in models. The newly generated inputs subsequently serve as additional training instances for model refinement through retraining. Existing approaches generate test inputs by optimizing an objective function based on testing metrics such as neuron coverage and property-related metrics, and the gradient of the objective is used to perturb seed inputs. However, these approaches pay limited attention to the model’s decision logic, particularly the erroneous decision patterns learned during training. Furthermore, they primarily focus on detecting faults without considering the diversity of detected misbehaviors, which limits the models’ ability to learn diverse features through retraining. To address these limitations, this article introduces SUNTest, a novel test input generation approach designed to detect diverse faults and enhance the robustness of DNN models. SUNTest focuses on erroneous decision-making by localizing suspicious neurons responsible for misbehaviors through the execution spectrum analysis of neurons. To guide input mutations toward inducing diverse faults, SUNTest designs a hybrid fitness function that incorporates two types of feedback derived from neuron behaviors, including the fault-revealing capability of test inputs guided by suspicious neurons and the diversity of test inputs. Additionally, SUNTest adopts an adaptive selection strategy for mutation operators to prioritize operators likely to induce new fault types and improve the fitness value in each iteration. Experiments conducted on eight DNN models demonstrate the effectiveness of SUNTest in fault localization and test input generation. It outperforms existing test input generators in the number of detected faults, uncovering up to 80.9 more distinct fault types. In terms of model enhancement, SUNTest increases the average accuracy improvement by up to 8.04% compared to baseline approaches.

Deep Learning has been applied to many applications across different domains. However, the distribution shift between the test data and training data is a major factor impacting the quality of deep neural networks (DNNs). To address this issue, existing research mainly focuses on enhancing DNN models by retraining them using labeled test data. However, labeling test data is costly, which seriously reduces the efficiency of DNN testing. To solve this problem, test selection strategically selected a small set of tests to label. Unfortunately, existing test selection methods seldom focus on the data distribution shift. To address the issue, this paper proposes an approach for test selection named Feature Distribution Analysis-Based Test Selection (FATS). FATS analyzes the distributions of test data and training data and then adopts learning to rank (a kind of supervised machine learning to solve ranking tasks) to intelligently combine the results of analysis for test selection. We conduct an empirical study on popular datasets and DNN models, and then compare FATS with seven test selection methods. Experiment results show that FATS effectively alleviates the impact of distribution shifts and outperforms the compared methods with the average accuracy improvement of 19.6%$\sim$69.7% for DNN model enhancement.

To reveal incorrect behaviors and improve the quality of DNN models through testing, a commonly used approach is to collect massive test inputs and manually label them for model optimization. However, it is labor-intensive and time-consuming to manually label the whole collected test inputs. To reduce the labeling cost, recent studies propose test input selection approaches to select a subset of test inputs for model retraining, thereby improving the performance of DNN models. Existing approaches primarily rely on output probabilities to select error-revealing test inputs. Nevertheless, such approaches select test inputs without the knowledge of underlying mechanisms that lead to incorrect decisions. Furthermore, they fail to consider the diversity of test inputs, limiting the capability of DNN models to learn more diverse features. To address these limitations, this paper proposes MOON, a white-box test input selection approach based on multi-objective optimization. The neuron spectrum is proposed to localize suspicious neurons that contribute to erroneous decisions made by DNN models. MOON formulates the test input selection method into a search-based testing problem. By tailoring a multi-objective optimization algorithm, it guides the search process towards maximizing the outputs of suspicious neurons while promoting diversity in neuron behaviors. An empirical evaluation on three datasets and six DNN models was conducted. The experimental results demonstrate the effectiveness of MOON in localizing suspicious neurons. In addition, MOON significantly outperforms state-of-the-art test selection approaches, with an accuracy improvement rate of up to 236.1%.

Deep Neural Networks (DNNs) have been applied in many domains, such as autonomous driving and image recognition. However, due to the lower-than-expected performance of the DNN models in the real application, researchers are committed to sampling a test subset from test data with the limited labeling effort to retrain the DNN models for enhancement. Existing test input selection methods aim at selecting the test inputs according to the probability that is classified incorrectly by the DNN model. However, the test inputs selected by using existing methods might have similar features, making the DNN model unable to learn more diverse features when retraining. To address this limitation, this paper proposes Multiple-Objective Optimization-Based Test Input Selection (MOTS) to select more effective test subset to retrain the DNN model for enhancement. Different from existing works, this work not only considers the uncertainty of the test input but also takes the diversity of the test subset into account. Then MOTS uses a multiple-objective optimization algorithm NSGA-II to solve this problem, which ensures the test subset has more diverse features and is more helpful for retraining DNN models. This paper conducts the experiment on two popular DNN models and three widely-used datasets. The experiment results indicate that MOTS achieves 114%, 72%, 55%, 41% average accuracy improvement under four different sampling ratios 1%, 3%, 5%, 10% compared with five baseline methods. Therefore, MOTS is very effective in improving the quality of the DNN models compared with the state-of-the-art methods and the diversity of the test subset contributes greatly to the effectiveness of retraining DNN models.

Abstract Despite the tremendous development of artificial intelligence (AI)‐based mobile apps, they suffer from quality issues. Data‐driven AI software poses challenges for maintenance and quality assurance. Metamorphic testing has been successfully adopted to AI software. However, most previous studies require testers to manually identify metamorphic relations in an ad hoc and arbitrary manner, thereby encountering difficulties in reflecting real‐world usage scenarios. Previous work showed that information available in user reviews is effective for maintenance and testing tasks. Yet, there is a lack of studies leveraging reviews to facilitate AI function maintenance and testing activities. This paper proposes METUR, a novel approach to supporting maintenance and testing for AI functions based on reviews. Firstly, METUR automatically classifies reviews that can be exploited for supporting AI function maintenance and evolution activities. Then, it identifies test contexts from reviews in the usage scenario category. METUR instantiates the metamorphic relation pattern for deriving concrete metamorphic relations based on test contexts. The follow‐up test dataset is constructed for conducting metamorphic testing. Empirical studies on plant identification apps indicate that METUR effectively categorizes reviews that are related to AI functions. METUR is feasible and effective in detecting inconsistent behaviors by using the metamorphic relations constructed based on reviews.

Software defect prediction(SDP), which predicts defective code areas, including files, code blocks, code lines, etc. It can help developers or testers in allocating test resources before the testing phase. Software defect number prediction(SDNP) is an important research direction of SDP. Previous studies mostly used regression-based methods or different neural networks to mine the semantic features contained in AST, but the way to represent code was relatively simple. In this article, we propose a framework for representing the semantic features in terms of sequences of nodes with a double sequence structure, by analyzing the ASTs and the changes in the code blocks between adjacent version. In addition, to combine statistical metric information, we also propose a model that dynamically determines the ratio of semantic features to traditional metric features during model training by using the gated fusion mechanism to perform SDNP. In the experimental part, we select 10 open source Java projects as training and test sets, and conduct a lot of comparative experiments. The experimental results demonstrate the superiority of our proposed method compared to the baseline approach.

Software defect prediction (SDP), which predicts defective code regions, can help developers reasonably allocate limited resources for locating bugs and prioritizing their testing efforts. Previous work on defect prediction has used machine learning and artificial software metrics. However, traditional defect prediction features extracted from artificial software metrics often fail to capture the syntactic and semantic information of defective modules. This work on defect prediction mostly focuses on abstract syntax tree (AST). Moreover, because current research on AST technology is relatively mature, it is difficult to further improve the accuracy of defect prediction when only using AST to characterize codes. In this paper, in order to capture more semantic features, we extract semantic information both from the sequences of AST tokens and code change tokens. In addition, to leverage the traditional features extracted from statistical metrics, we also combine the semantic features with traditional defect prediction features to perform SDP, and use the gated fusion mechanism to determine the combination ratio of the two kinds of features. In our empirical studies, 10 open-source Java projects from the PROMISE repository are chosen as our empirical subjects. Experimental results show that our proposed approach can perform better than several state-of-the-art baseline SDP methods.

Requirement traceability is essential for software development and maintenance, thereby effectively recovering the requirements traceability has become an important issue for requirement engineering. With the development of software systems, it is always unrealistic to maintain traceability links between requirements and source code manually. Therefore, researchers have proposed information retrieval-based approaches to recover the links automatically. Although these methods reduce human labor, they do not fully extract the specific features, resulting in poor traceability accuracy. In this paper, we propose an approach to recovering traceability between requirements and source code, which combines word embedding and self-attention model to extract features and generate text vectors. These technologies make full use of the semantic information of the context and feature representation. In addition, the paper discusses the impact of code content and comments on the results and improves the results on weight. Finally, the proposed approach is compared with the commonly-used baselines, and the study results show that the proposed approach outperforms others.

Although data-driven image recognition software has widely emerged in various fields, they suffer from quality issues.Metamorphic testing has been successfully applied to AI software for alleviating test oracle problems.Nevertheless, metamorphic testing still relies on manual methods in most cases, which is timeconsuming.To improve test efficiency, a testing tool called MT4ImgRec is designed to automatically perform metamorphic testing for image recognition software.

High-quality Artificial intelligence (AI) software in different domains, like image recognition, has been widely emerged in our lives.They are built on machine learning models to implement intelligent features.However, the current research on image recognition software rarely discusses test questions, clear quality requirements, and verification methods.This paper presents a case study of a realistic image recognition application called Calorie Mama using manual and automation testing with a 3D decision table.The study results indicate the proposed method is feasible and effective in quality evaluation.

Software bug issues are unavoidable in software development and maintenance. In order to manage bugs effectively, bug tracking systems are developed to help to record, manage and track the bugs of each project. The rich information in the bug repository provides the possibility of establishment of entity-centric knowledge bases to help understand and fix the bugs. However, existing named entity recognition (NER) systems deal with text that is structured, formal, well written, with a good grammatical structure and few spelling errors, which cannot be directly used for bug-specific named entity recognition. For bug data, they are free-form texts, which include a mixed language studded with code, abbreviations and software-specific vocabularies. In this paper, we summarize the characteristics of bug entities, propose a classification method for bug entities, and build a baseline corpus on two open source projects (Mozilla and Eclipse). On this basis, we propose an approach for bug-specific entity recognition called BNER with the Conditional Random Fields (CRF) model and word embedding technique. An empirical study is conducted to evaluate the accuracy of our BNER technique, and the results show that the two designed baseline corpus are suitable for bug-specific named entity recognition, and our BNER approach is effective on cross-projects NER.