The widespread use of deep learning has made ensuring the robustness of image classification models a critical challenge for system security. Research has demonstrated that adversarial examples can serve as test cases to find errors in models and evaluate their robustness. Existing ensemblebased methods are designed to improve the effectiveness of adversarial test case generation by integrating multiple models. However, they still face key challenges, including limited perturbation diversity, inaccurate gradient directions, and high gradient variance across models, which reduce the effectiveness of surrogate-based test cases on other models and increase testing costs. To address these issues, we propose a dynamic variance reduction-based reusable test case generation method (DVRTM). This method uses a two-level gradient-based update mechanism for test case generation: an outer loop and an inner loop. The outer loop ensembles gradients from multiple models to increase perturbation diversity. In the inner loop, one model is randomly selected in each iteration. The input example is then adjusted dynamically based on the model’s gradient and loss. The gradients of the example before and after adjustment are combined with historical gradient information to reduce gradient variance and optimize the gradient direction, thereby avoiding overfitting to a single model. Experimental results show that DVRTM outperforms comparison methods on CIFAR-10, CIFAR-100, and ImageNet datasets. On the CIFAR-10 dataset, the test cases generated by DVRTM achieve the highest average error detection rate of 92.69% across models with different architectures. Even when target models adopt defense methods, DVRTM maintains stable and superior detection performance.

Deep neural network (DNN) based text intelligence softwares have proliferated and have been widely used. However, they face challenges such as robustness. Testing is essential to DNNs in real-world applications. Researchers have proposed various testing techniques for generating test cases. With the black-box hard-label setup where only the output label of the DNN is accessible, related studies mainly concentrated on generating the new text test case by iteratively perturbing single initial test case, which limits the effectiveness. To address this issue, this paper proposes a new text test case generation method MVGText (Momentum and Variance Guided hard-label Text test case generation) for text-oriented DNN, which generates high-quality text test cases by searching based on multiple initial test cases. Firstly, multiple initial text test cases are generated randomly, in which global fixed word replacement is employed to enhance the success rate. Then, momentum and variance are applied to guide the search for the more imperceptible test case. Finally, greedy optimization is used to obtain a higher quality test case. Experimental evaluation shows that MVGText achieves an average post-test success rate of 1.4%. Furthermore, the test cases generated by MVGText exhibit the highest similarity to the original text compared to other comparative methods.

Deep learning (DL) components have been broadly applied in diverse applications. Similar to traditional software engineering, effective test case generation methods are needed by industry to enhance the quality and robustness of these deep learning components. To this end, we propose a novel automatic software testing technique, TAEFuzz (Automatic Fuzz -Testing via T ransferable A dversarial E xamples), which aims to automatically assess and enhance the robustness of image-based deep learning (DL) systems based on test cases generated by transferable adversarial examples. TAEFuzz alleviates the over-fitting problem during optimized test case generation and prevents test cases from prematurely falling into local optima. In addition, TAEFuzz enhances the visual quality of test cases through constraining perturbations inserted into sensitive areas of the images. For a system with low robustness, TAEFuzz trains a low-cost denoising module to reduce the impact of perturbations in transferable adversarial examples on the system. Experimental results demonstrate that the test cases generated by TAEFuzz can discover up to 46.1% more errors in the targeted systems, and ensure the visual quality of test cases. Compared to existing techniques, TAEFuzz also enhances the robustness of the target systems against transferable adversarial examples with the perturbation denoising module.