The outstanding performance of machine learning models across various fields enables them to be used in sensitive and high-risk activities such as healthcare, automated driving, and security services. However, the explainability of their outputs and the safety of their operation are major concerns. Although Explainable AI (XAI) can enhance the interpretability of AI models, further research is necessary to evaluate its effectiveness in explaining adversarial attacks. The use of XAI techniques and the rise of adversarial attacks are important issues related to the explainability and security of deep learning, respectively. Furthermore, the relation between the explainability and safety of deep learning, such as the vulnerability of the XAI explanation to an adversarial attack, is key to unlocking these concerns. In this paper, we use the Saliency Map as an XAI technique to explain the behavior of the Fast Gradient Sign Method (FGSM) adversarial attack on the ResNet model, and show the vulnerability related to such an explanation with respect to the attack. Extensive experiments show that as the severity of FGSM attack on the ResNet model increases, the Saliency Map gradually fails, exposing its potential vulnerability.

Backdoor attacks significantly threaten machine learning-based vision systems. Existing detection methods typically require clean data from a similar distribution as the dataset under inspection, limiting practical deployment. This work proposes a Mutation-Based Method (MBM) for detecting and filtering backdoored samples in image training dataset, without referencing any external clean data. MBM aims at distinguishing backdoored and benign samples distribution via their distinct stability in feature space under certain data augmentations. Firstly, MBM applies multiple data augmentation techniques, generating mutated versions of each sample to ‘deactivate’ potential triggers while maintaining natural semantics not heavily distorted. Secondly, MBM measures how sample features diverge after mutating from its origin as poison score, which we call ‘Feature Stability’. Thirdly, by analyzing extreme scores within each class, MBM effectively identifies the backdoored class, and isolates samples not from backdoored class as clean data. Finally, a benign distribution is fit to benchmark against backdoored samples from backdoored class. We validated MBM on the CIFAR-10 dataset, achieving a true positive rate above 95% and a false positive rate below 0.2% for all defense settings. Our results confirm MBM’s efficacy without reliance on external clean data.