ABSTRACT A code summarization engine based on large language models (LLMs) can describe code functionality from different perspectives according to programmers' needs. However, these engines are at risk of black‐box backdoor attacks. We propose a simple yet practical method called Bad Prompt Attack (BPA), specifically designed to investigate such black‐box backdoor attacks. This innovative attack method aims to induce the code summarization engine to generate summarizations that conceal security vulnerabilities in source code. Consistent with most commercial code summarization engines, BPA only assumes black‐box query access to the target engine without requiring knowledge of its internal structure. This attack targets in‐context learning by injecting adversarial demonstrations into user input prompts. We validated our method on the SOTA black‐box commercial service, OpenAI API. In security‐critical test cases covering seven types of CWE, BPA significantly increased the likelihood that the code summarization engine would generate the attacker‐desired code summarization targets, achieving an average attack success rate (ASR) of 91.4%. This result underscores the potential threat of backdoor attacks on code summarization tasks while providing essential reference points for future defense research.

As deep neural network (DNN) has made significant advancements across various fields, systematically testing DNN has become increasingly crucial. To uncover potential faults within DNN, a vast number of test cases and their correct labels are required, but the process of labeling is time-consuming and labor-intensive. To alleviate the burden on developers, test case selection techniques for DNN models have been proposed, assisting in the selection of test cases from large datasets that are more likely to reveal model faults. In this study, we introduce an efficient adaptive test case selection method based on the principle of uniform distribution of test cases, named EATS. In addition, we propose a test case optimization method and an image similarity calculation method. The optimization method can save time during the test case selection process, while the image similarity calculation method computes the degree of difference between images based on model uncertainty. EATS leverages the model’s uncertainty to achieve a more uniform distribution of selected test cases, aiming to select test cases that can induce a diversity of model fault predictions, thereby optimizing model performance. We conducted comparative experiments of EATS and other test case selection strategies on four common datasets and their corresponding DNN models. The experimental results show that EATS outperforms other methods in terms of the uniformity of test case distribution, diversity of errors discovered, and model optimization. It also demonstrates excellent time efficiency.

Abstract Estimating the population of defects is a key reference for completion evaluation in crowdsourced testing. Current studies commonly use the biological capture–recapture model (CRC model) to estimate the population of defects in crowdsourced tests with good results. However, the lack of consideration of false negatives for defects in existing studies leads to imprecise estimation of completion in crowdsourced testing, which in turn influences decision support during crowdsourced testing. This study analyzes the basic assumptions of the CRC model in biology and maps them to the field of crowdsourced testing, improves the Lincoln–Peterson estimator and sample coverage estimator utilizing false negative assumptions, and compares them with original estimators in 29 real crowdsourced testing projects. The experimental results show that our improved estimators can effectively reduce the estimation relative error of the population of defects caused by false negatives.

Abstract The development of artificial intelligence and information communication technology has significantly propelled advancements in autonomous driving. The advent of autonomous driving has a profound impact on societal development and transportation methods. However, as intelligent systems, autonomous driving systems (ADSs) often make wrong judgements in specific scenarios, resulting in accidents. There is an urgent need for comprehensive testing and validation of ADSs. Metamorphic testing (MT) techniques have demonstrated effectiveness in testing ADSs. Nevertheless, existing testing methods primarily encompass relatively simple metamorphic relations (MRs) that only verify ADSs from a single perspective. To ensure the safety of ADSs, it is essential to consider the various elements of driving scenarios during the testing process. Therefore, this paper proposes MetaSem, a novel metamorphic testing method based on semantic information of autonomous driving scenes. Based on semantic information of the autonomous driving scenes and traffic regulations, we design 11 MRs targeting different scenario elements. Three transformation modules are developed to execute addition, deletion and replacement operations on various scene elements within the images. Finally, corresponding evaluation metrics are defined based on MRs. MetaSem automatically discovers inconsistent behaviours according to the evaluation metrics. Our empirical study on three advanced and popular autonomous driving models demonstrates that MetaSem not only efficiently generates visually natural and realistic scene images but also detects 11,787 inconsistent behaviours on three driving models.

The widespread adoption of deep learning (DL) has made it critical to ensure its reliability. Mutation testing has been employed in DL testing to assess test data quality, but it can be costly of a large number of generated mutants. Cost reduction can be achieved by selecting a sufficient subset of mutation operators. However, it remains unclear to what extent the DL mutation operators contribute to test effectiveness, making it challenging to determine which are useful mutation operators in a selective mutation approach.