The integration of combinatorial testing (CT) with safety requirement analysis remains challenging for complex software systems, particularly when evolving requirements necessitate dynamic updates to test constraints. Traditional CT methods often fail to systematically incorporate safety constraints derived from early-phase requirements, leading to test suites that become obsolete as system specifications change. In this paper, we propose a novel method that embeds system-theoretic process analysis (STPA) into the CT workflow, enabling the automatic derivation and maintenance of safety-aware combinatorial test. Our approach First formalizes STPA-identified unsafe control actions and hazard mitigation rules as refined constraints, then integrated the refined constraints into constraint model to guide incremental test sequence generation. Experimental results demonstrate that our method achieves an effective improvement in test generation efficiency across three representative case studies (robotic control, secure vault system and civil aircraft flight mode transitions), while maintaining compliance with dynamically evolving safety requirements.

Deep Neural Network (DNN) testing has emerged as an effective way of uncovering erroneous behaviors in DNN models and further enhancing their performance. Research on test input generation has gained much attention from both researchers and practitioners, aiming to expose faults in models. The newly generated inputs subsequently serve as additional training instances for model refinement through retraining. Existing approaches generate test inputs by optimizing an objective function based on testing metrics such as neuron coverage and property-related metrics, and the gradient of the objective is used to perturb seed inputs. However, these approaches pay limited attention to the model’s decision logic, particularly the erroneous decision patterns learned during training. Furthermore, they primarily focus on detecting faults without considering the diversity of detected misbehaviors, which limits the models’ ability to learn diverse features through retraining. To address these limitations, this article introduces SUNTest, a novel test input generation approach designed to detect diverse faults and enhance the robustness of DNN models. SUNTest focuses on erroneous decision-making by localizing suspicious neurons responsible for misbehaviors through the execution spectrum analysis of neurons. To guide input mutations toward inducing diverse faults, SUNTest designs a hybrid fitness function that incorporates two types of feedback derived from neuron behaviors, including the fault-revealing capability of test inputs guided by suspicious neurons and the diversity of test inputs. Additionally, SUNTest adopts an adaptive selection strategy for mutation operators to prioritize operators likely to induce new fault types and improve the fitness value in each iteration. Experiments conducted on eight DNN models demonstrate the effectiveness of SUNTest in fault localization and test input generation. It outperforms existing test input generators in the number of detected faults, uncovering up to 80.9 more distinct fault types. In terms of model enhancement, SUNTest increases the average accuracy improvement by up to 8.04% compared to baseline approaches.

Multimodal sentiment analysis aims to integrate diverse modalities for precise emotional interpretation. However, external factors such as sensor malfunctions or network issues may disrupt certain modalities. This may lead to missing data, which poses challenges in real-world deployment. Most existing approaches focus on designing feature reconstruction strategies, overlooking the collaborative integration of reconstruction and fusion strategies. Moreover, they fail to capture the relationships between features in the global dimension and those in the local dimension. These limitations hinder the full capture of the complex nature of multimodal data, especially in scenarios involving missing modalities. To address the above issues, this paper proposes a robust model named MFMB-Net with multiple branches for feature multi-focus fusion and reconstruction. We design a two-stream fusion branch where macro-fusion focuses on the fusion of features in the global dimension and micro-fusion targets local dimension features. This dual-stream fusion branch distributes multi-focus across both pathways, simultaneously capturing global coarse-grained and local fine-grained features. Additionally, the reconstruction branch interacts collaboratively with the fusion branch to reconstruct and enhance the missing data. It integrates the reconstructed feature information with the fused information thus refining the representation fidelity of the missing information. Experiments performed on two benchmarks show that our approach obtains results superior to state-of-the-art models.

Deep neural networks have substantially improved machine translation quality; however, terminology remains prone to mistranslation, leading to misinterpretation and communication barriers. Such terminology-related errors persist in MT, often causing biased understanding and hindering communication. Existing terminology evaluations frequently rely on local substitutions or deletions in the source sentence, an approach that is both noise-prone and cumbersome. To address this challenge, guided by the principle of perturbation invariance, this paper proposes a terminology-oriented, multidimensional metamorphic testing method. We introduce two terminology mutation operations: (i) inserting hyphens and (ii) repeating the term at the sentence-initial or sentence-final position, to assess systems’ robustness to terminology. Experimental results show that MDT-TM achieves detection accuracies of 81.55%, 80.18%, and 76.65% on Google, Microsoft Bing, and Baidu, respectively.

The wide range of applications of deep neural networks in image recognition, medical diagnosis, and safetycritical areas makes it essential to test them adequately. However, traditional software testing methodologies proves challenging due to the inherent complex topology and blackbox characteristics of DNNs. Existing testing techniques (e.g., coverage-guided testing, fuzzy testing, etc.) generally lack guidance mechanisms for generating samples and similar fault characteristics. To address the lack of guidance and constraints on the generated samples, a conditional metric, the Gini impurity, is introduced to generate test inputs with high fault detection rates. In addition, many randomly generated similar and redundant test case samples ignore the problem of generating image diversity, and to solve this problem, the geometric diversity (GD) score is introduced to generate image test cases with more different fault feature types. In this paper, we propose DeepMCC, a multi-conditional constraintbased test case generation method based on the Conditional Generative Adversarial Networks (CGANs) framework, which innovatively introduces geometric diversity (GD) scores and Gini impurity. To validate the effectiveness of our method, we conduct experiments on four publicly available image datasets with five widely used DNN models, and the results show that (1) our method is more effective in terms of fault detection rate compared to black-box test methods, which generally improves the fault detection rate by 1% to 2%; (2) with the preset constraints on the geometric diversity scores, we can precisely control the generated test cases with higher diversity; (3) using the images generated by this method for retraining can effectively improve the accuracy of the model, which improves the accuracy by 0.1% to 0.4% on the original test set compared to other methods.

With the advancement of deep neural networks, machine translation has seen rapid progress in recent years. Individuals often rely on machine translation software to facil-itate various tasks. However, the intricacies of neural networks can lead to translation errors, resulting in misunderstandings or conflicts. The most common method for testing machine translation is metamorphic testing. However, metamorphic testing at the phrase or sentence hierarchy may result in some test cases being incorrectly identified as failures. To mitigate this issue, we added the word hierarchy. We proposed a multi-hierarchy metamorphic testing method, MHT, to test machine translation. Hyphenated words as a specific format prone to translation errors, which are chosen as our research object. Based on the common notion that translations of words within the same sentence should be similar, we extract contents from different hierarchies within sentences containing hyphenated words and compare the similarity of their corresponding translations for these specific words. We conducted the experiments on 881 sentences leveraging Google Translate, Microsoft Bing Translator, and Baidu Translate, which detected 111, 91, and 111 suspicious errors with high precision (78.4 %, 82.4 %, and 81.1 %). Translation errors mainly include mis-translation, under-translation, over-translation, and non-translation.

Deep Learning has been applied to many applications across different domains. However, the distribution shift between the test data and training data is a major factor impacting the quality of deep neural networks (DNNs). To address this issue, existing research mainly focuses on enhancing DNN models by retraining them using labeled test data. However, labeling test data is costly, which seriously reduces the efficiency of DNN testing. To solve this problem, test selection strategically selected a small set of tests to label. Unfortunately, existing test selection methods seldom focus on the data distribution shift. To address the issue, this paper proposes an approach for test selection named Feature Distribution Analysis-Based Test Selection (FATS). FATS analyzes the distributions of test data and training data and then adopts learning to rank (a kind of supervised machine learning to solve ranking tasks) to intelligently combine the results of analysis for test selection. We conduct an empirical study on popular datasets and DNN models, and then compare FATS with seven test selection methods. Experiment results show that FATS effectively alleviates the impact of distribution shifts and outperforms the compared methods with the average accuracy improvement of 19.6%$\sim$69.7% for DNN model enhancement.

Memory-related vulnerabilities can result in performance degradation or even program crashes, constituting severe threats to the security of modern software. Despite the promising results of deep learning (DL)-based vulnerability detectors, there exist three main limitations: (1) rich contextual program semantics related to vulnerabilities have not yet been fully modeled; (2) multi-granularity vulnerability features in hierarchical code structure are still hard to be captured; and (3) heterogeneous flow information is not well utilized. To address these limitations, in this article, we propose a novel DL-based approach, called MVD+ , to detect memory-related vulnerabilities at the statement-level. Specifically, it conducts both intraprocedural and interprocedural analysis to model vulnerability features, and adopts a hierarchical representation learning strategy, which performs syntax-aware neural embedding within statements and captures structured context information across statements based on a novel Flow-Sensitive Graph Neural Networks, to learn both syntactic and semantic features of vulnerable code. To demonstrate the performance, we conducted extensive experiments against eight state-of-the-art DL-based approaches as well as five well-known static analyzers on our constructed dataset with 6,879 vulnerabilities in 12 popular C/C++ applications. The experimental results confirmed that MVD+ can significantly outperform current state-of-the-art baselines and make a great trade-off between effectiveness and efficiency.

To reveal incorrect behaviors and improve the quality of DNN models through testing, a commonly used approach is to collect massive test inputs and manually label them for model optimization. However, it is labor-intensive and time-consuming to manually label the whole collected test inputs. To reduce the labeling cost, recent studies propose test input selection approaches to select a subset of test inputs for model retraining, thereby improving the performance of DNN models. Existing approaches primarily rely on output probabilities to select error-revealing test inputs. Nevertheless, such approaches select test inputs without the knowledge of underlying mechanisms that lead to incorrect decisions. Furthermore, they fail to consider the diversity of test inputs, limiting the capability of DNN models to learn more diverse features. To address these limitations, this paper proposes MOON, a white-box test input selection approach based on multi-objective optimization. The neuron spectrum is proposed to localize suspicious neurons that contribute to erroneous decisions made by DNN models. MOON formulates the test input selection method into a search-based testing problem. By tailoring a multi-objective optimization algorithm, it guides the search process towards maximizing the outputs of suspicious neurons while promoting diversity in neuron behaviors. An empirical evaluation on three datasets and six DNN models was conducted. The experimental results demonstrate the effectiveness of MOON in localizing suspicious neurons. In addition, MOON significantly outperforms state-of-the-art test selection approaches, with an accuracy improvement rate of up to 236.1%.

Abstract Despite the tremendous development of artificial intelligence (AI)‐based mobile apps, they suffer from quality issues. Data‐driven AI software poses challenges for maintenance and quality assurance. Metamorphic testing has been successfully adopted to AI software. However, most previous studies require testers to manually identify metamorphic relations in an ad hoc and arbitrary manner, thereby encountering difficulties in reflecting real‐world usage scenarios. Previous work showed that information available in user reviews is effective for maintenance and testing tasks. Yet, there is a lack of studies leveraging reviews to facilitate AI function maintenance and testing activities. This paper proposes METUR, a novel approach to supporting maintenance and testing for AI functions based on reviews. Firstly, METUR automatically classifies reviews that can be exploited for supporting AI function maintenance and evolution activities. Then, it identifies test contexts from reviews in the usage scenario category. METUR instantiates the metamorphic relation pattern for deriving concrete metamorphic relations based on test contexts. The follow‐up test dataset is constructed for conducting metamorphic testing. Empirical studies on plant identification apps indicate that METUR effectively categorizes reviews that are related to AI functions. METUR is feasible and effective in detecting inconsistent behaviors by using the metamorphic relations constructed based on reviews.

Code recommendation with programming context is to use the contextual code surrounding the missing code to automatically find that which of the code snippets would be useful to assist in the completion of the program. In this way, the developers need not take time to formulate explicit queries or write descriptions. Existing work only treats code as textual documents and use information retrieval techniques to retrieve relevant code snippets, which is difficult to capture the semantics of code adequately. The self-attention mechanism have achieved promising progress in various natural language processing tasks, especially for extracting deep semantic information from long sequences. Inspired from this, we propose a novel code recommendation with programming context based on self-attention mechanism (CRAM). The proposed approach first builds a small-scale candidate set from codebase. Then, it utilizes self-attention networks in the abstract syntax tree to capture the deep semantics of code, and finally recommend the relevant code to developers. We conduct several experiments to evaluate our approach in a large-scale codebase containing 741 148 code snippets. The experimental results show that CRAM can effectively recommend code and outperforms related work in recall, precision, and NDCG.

Memory-related vulnerabilities constitute severe threats to the security of modern software. Despite the success of deep learning-based approaches to generic vulnerability detection, they are still limited by the underutilization of flow information when applied for detecting memory-related vulnerabilities, leading to high false positives.

Software defect prediction(SDP), which predicts defective code areas, including files, code blocks, code lines, etc. It can help developers or testers in allocating test resources before the testing phase. Software defect number prediction(SDNP) is an important research direction of SDP. Previous studies mostly used regression-based methods or different neural networks to mine the semantic features contained in AST, but the way to represent code was relatively simple. In this article, we propose a framework for representing the semantic features in terms of sequences of nodes with a double sequence structure, by analyzing the ASTs and the changes in the code blocks between adjacent version. In addition, to combine statistical metric information, we also propose a model that dynamically determines the ratio of semantic features to traditional metric features during model training by using the gated fusion mechanism to perform SDNP. In the experimental part, we select 10 open source Java projects as training and test sets, and conduct a lot of comparative experiments. The experimental results demonstrate the superiority of our proposed method compared to the baseline approach.

Software defect prediction (SDP), which predicts defective code regions, can help developers reasonably allocate limited resources for locating bugs and prioritizing their testing efforts. Previous work on defect prediction has used machine learning and artificial software metrics. However, traditional defect prediction features extracted from artificial software metrics often fail to capture the syntactic and semantic information of defective modules. This work on defect prediction mostly focuses on abstract syntax tree (AST). Moreover, because current research on AST technology is relatively mature, it is difficult to further improve the accuracy of defect prediction when only using AST to characterize codes. In this paper, in order to capture more semantic features, we extract semantic information both from the sequences of AST tokens and code change tokens. In addition, to leverage the traditional features extracted from statistical metrics, we also combine the semantic features with traditional defect prediction features to perform SDP, and use the gated fusion mechanism to determine the combination ratio of the two kinds of features. In our empirical studies, 10 open-source Java projects from the PROMISE repository are chosen as our empirical subjects. Experimental results show that our proposed approach can perform better than several state-of-the-art baseline SDP methods.

Requirement traceability is essential for software development and maintenance, thereby effectively recovering the requirements traceability has become an important issue for requirement engineering. With the development of software systems, it is always unrealistic to maintain traceability links between requirements and source code manually. Therefore, researchers have proposed information retrieval-based approaches to recover the links automatically. Although these methods reduce human labor, they do not fully extract the specific features, resulting in poor traceability accuracy. In this paper, we propose an approach to recovering traceability between requirements and source code, which combines word embedding and self-attention model to extract features and generate text vectors. These technologies make full use of the semantic information of the context and feature representation. In addition, the paper discusses the impact of code content and comments on the results and improves the results on weight. Finally, the proposed approach is compared with the commonly-used baselines, and the study results show that the proposed approach outperforms others.

Although data-driven image recognition software has widely emerged in various fields, they suffer from quality issues.Metamorphic testing has been successfully applied to AI software for alleviating test oracle problems.Nevertheless, metamorphic testing still relies on manual methods in most cases, which is timeconsuming.To improve test efficiency, a testing tool called MT4ImgRec is designed to automatically perform metamorphic testing for image recognition software.

High-quality Artificial intelligence (AI) software in different domains, like image recognition, has been widely emerged in our lives.They are built on machine learning models to implement intelligent features.However, the current research on image recognition software rarely discusses test questions, clear quality requirements, and verification methods.This paper presents a case study of a realistic image recognition application called Calorie Mama using manual and automation testing with a 3D decision table.The study results indicate the proposed method is feasible and effective in quality evaluation.

Regression testing is a very time-consuming and expensive testing activity. Many test case prioritization techniques have been proposed to speed up regression testing. Previous studies show that no one technique is always best. Random strategy, as the simplest strategy, is not always so bad. Particularly, when a test suite has higher fault detection capability, the strategy can generate a better result. Nevertheless, due to the randomness, the strategy is not always as satisfactory as expected. In this context, we present a test case prioritization approach using fixed size candidate set adaptive random testing algorithm to reduce the effect of randomness and improve fault detection effectiveness. The distance between pair-wise test cases is assessed by exclusive OR. We designed and conducted empirical studies on eight C programs to validate the effectiveness of the proposed approach. The experimental results, confirmed by a statistical analysis, indicate that the approach we proposed is more effective than random and the total greedy prioritization techniques in terms of fault detection effectiveness. Although the presented approach has comparable fault detection effectiveness to ART-based and the additional greedy techniques, the time cost is much lower. Consequently, the proposed approach is much more cost-effective.

International audience

With the rapid advance of mobile computing technology and wireless networking, there is a significant increase of mobile subscriptions.This brings new business requirements and demands in mobile software testing, and causes new issues and challenges in mobile testing and automation.As there are multiple platforms for diverse devices, engineers suffer from the different scripting languages to write platform-specific test scripts.In addition, a unified automation infrastructure is not offered with the existing test platform.This paper proposes a novel approach to mobile application testing based on natural language scripting.A Java-based test script generation approach is developed to support executable test script generation based on the given natural language-based mobile app test operation scripts.A prototype tool is implemented based on some open sources.Finally, the paper reports empirical studies to indicate the feasibility and effectiveness of the proposed approach.

Face recognition system is a widely-used intelligent application nowadays.Existing recognition system evaluation methods primarily focus on recognition rate, i.e., the correct result.However, current research seldom focuses on the quality evaluation of face recognition systems.They seldom consider accuracy or the quality of recognition.To address this issue, this paper proposes several quality factors for evaluation.In addition, corresponding metrics for diverse quality factors are illustrated.Moreover, the paper presents an experimental study on a realistic non-trial face age recognition system using the proposed quality evaluation method.The study result shows the proposed method is feasible and effectiveness in quality evaluation.

With the fast advance of big data technology and analytics solutions, building high-quality big data computing services in different application domains is becoming a very hot research and application topic among academic and industry communities, and government agencies.Therefore, big data based applications are widely-used currently, such as recommendation, predication, and decision system.Nevertheless, there are increasing quality problems resulting in erroneous testing costs in enterprises and businesses.Current research work seldom discusses how to effectively validate big data applications to assure system quality.This paper focuses on big data system validation and quality assurance, and includes informative discussions about essential quality parameters, primary focuses, and validation process.Moreover, the paper discusses potential testing methods for big data application systems.Furthermore, the primary issues, challenges, and needs in testing big data application are presented.

With the rapid advance of mobile computing technology and wireless networking, there is a significant increase of mobile subscriptions.This brings new business requirements and demands in mobile software testing, and causes new issues and challenges in mobile testing and automation.Current existing mobile application testing tools mostly concentrate on GUI, load and performance testing which seldom consider large-scale concurrent automation, coverage analysis, fault tolerance and usage of well-defined models.This paper introduces an implemented system that provides an automation solution across platforms on diverse devices using GUI ripping test scripting technique.Through incorporating open source technologies such as Appium and Selenium Grid, this paper addresses the scalable test automation control with the capability of fault tolerant.Additionally, maximum test coverage can also be obtained by executing parallel test scripts within the model.Finally, the paper reports case studies to indicate the feasibility and effectiveness of the proposed approach.

With the rapid advance of mobile computing, cloud computing and wireless network, there is a significant increase of mobile subscriptions. This brings new business requirements and demands in mobile testing service, and causes new issues and challenges. In this paper, informative discussions about cloud-based mobile testing-as-a-service (mobile TaaS) are offered, including the essential concepts, focuses, test process, and the expected testing infrastructures. Moreover, the paper presents a comparison among cloud-based mobile TaaS approaches and several best practices in industry are discussed. Futhermore, the primary issues, challenges, and needs are analyzed.

With the rapid advance of mobile computing technology and wireless networking, there is a significant increase of mobile applications (apps). This brings new business requirements and demands in mobile software testing, and causes new issues and challenges in mobile test automation. Existing mobile application testing approaches mostly concentrate on GUI-based testing, load and performance testing without considering large-scale concurrent mobile app test automation, and model-based test coverage analysis. In this paper, a mobile hierarchical GUI model is proposed to present mobile operation scenario flows and gesture features in a hierarchical manner, in order to facilitate test dependency analysis in test automation. Mobile app test coverage analysis is performed based on GUI ripping models. The paper also presents a developed system that provides a test automation solution using GUI models. Finally, the paper reports a case study to indicate the feasibility and effectiveness of the proposed approach.

Regression testing is an important but expensive stage of software maintenance. Regression test selection addresses the problem of reducing testing cost through selecting a subset of the existing test cases or rerun. Cost-effectiveness is an indispensable factor to consider when developing a regression testing technique. Cost models are created for the purpose of assessing cost-effectiveness of these techniques. The current regression test selection strategies or cost analysis seldom consider hierarchy, which is an inherent characteristic of object-oriented program. In addition, selecting test cases at different levels influences the precision and efficiency of selection. This paper presents a hierarchical regression test selection technique for Java programs to stepwise select test cases from high level of program to low level of program. To effectively evaluate the correlative overall cost, this paper also proposes a hierarchical cost model for analyzing the cost-effectiveness of selection level according to hierarchy step by step. The empirical studies show that our approach can reduce the total cost and achieve more cost-effective results.

Software change impact analysis (CIA) is a key technique for identifying unpredicted and potential effects caused by changes made to software. Different change types often have different impact mechanisms, even some changes do not impact other entities in programs in spite of some dependences existed between these entities and the modified entity. In this paper, we propose a static CIA technique, which considers different impact mechanisms and rules of different change types, to calculate the impact sets. Precision improvement of the impact sets relies on 3 aspects: change types of a modified entity, dependences between the modified entity and other entities, and the intuition that to win at the start -- if the initial impact set is estimated more accurately, then the final impact set depending on this initial impact set will be more precise. Experimental case study demonstrates the effectiveness of our technique, and its potential applications in software maintenance.