Adversarial patches pose a significant threat to computer vision models' integrity, decreasing the accuracy of various tasks, including object detection (OD). Most existing OD defenses exhibit a trade-off between enhancing the model's adversarial robustness and maintaining its performance on benign images. We propose KDAT (knowledge distillation with adversarial tuning), a novel mechanism that enhances the robustness of an OD model without compromising its performance on benign images or its inference time. Our method combines the knowledge distillation (KD) technique with the adversarial tuning concept to teach the model to match the predictions of adversarial images with those of their corresponding benign ones. To match these predictions, we designed four unique loss components, allowing the student model to effectively distill the knowledge of different features from various parts of the teacher model. Our extensive evaluation on the COCO and INRIA datasets demonstrates KDAT's ability to improve the performance of Faster R-CNN and DETR on benign images by 2-4 mAP% and adversarial examples by 10-15 mAP%, outperforming other state-of-the-art (SOTA) defenses. Furthermore, our additional physical evaluation on the Superstore dataset demonstrates KDAT's SOTA adversarial robustness against printed patches (improvement of 22 mAP% compared to the undefended model).

Amit Giloni, Chiara Picardi, Roy Betser, Shamik Bose, Aishvariya Priya Rathina Sabapathy, Roman Vainshtein. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

Distinguishing between real and AI-generated images, commonly referred to as 'image detection', presents a timely and significant challenge. Despite extensive research in the (semi-)supervised regime, zero-shot and few-shot solutions have only recently emerged as promising alternatives. Their main advantage is in alleviating the ongoing data maintenance, which quickly becomes outdated due to advances in generative technologies. We identify two main gaps: (1) a lack of theoretical grounding for the methods, and (2) significant room for performance improvements in zero-shot and few-shot regimes. Our approach is founded on understanding and quantifying the biases inherent in generated content, where we use these quantities as criteria for characterizing generated images. Specifically, we explore the biases of the implicit probability manifold, captured by a pre-trained diffusion model. Through score-function analysis, we approximate the curvature, gradient, and bias towards points on the probability manifold, establishing criteria for detection in the zero-shot regime. We further extend our contribution to the few-shot setting by employing a mixture-of-experts methodology. Empirical results across 20 generative models demonstrate that our method outperforms current approaches in both zero-shot and few-shot settings. This work advances the theoretical understanding and practical usage of generated content biases through the lens of manifold analysis.

Although object detection models are widely used, their predictive performance has been shown to deteriorate when faced with abnormal scenes. Such abnormalities can occur naturally (by partially occluded or out-of-distribution objects) or deliberately (in the case of an adversarial attack). Existing uncertainty quantification methods, such as object detection evaluation metrics and label-uncertainty quantification techniques, do not consider the abnormalities' effect on the model's internal decision-making process. Furthermore, practical methods that consider the effects of abnormalities (such as abnormality detection and mitigation) are designed to deal with one type of abnormality. We present distinctive localization (DiL), an unsupervised, practical and explainable metric that quantitatively interprets any type of abnormality and can be leveraged for preventive purposes. By utilizing XAI techniques (saliency maps), DiL maps the objectness of a given scene and captures the model's inner uncertainty regarding the identified (and missed) objects. DiL was evaluated across nine use cases, including partially occluded and out-of-distribution objects, as well as adversarial patches, in both physical and digital spaces, on benchmark datasets, and our newly E-PO dataset (generated with DALL-E 2). Our results show that DiL: i) successfully interprets and quantifies an abnormality's effect on the model's decision-making process, regardless of the abnormality type; and ii) can be leveraged to detect and mitigate this effect.

Recent studies have demonstrated external information leakage (EIL) attacks which allow an attacker to infer various sensitive implicit properties related to a machine learning (ML) model’s training data. Most of those attacks assumed 1) a white-box scenario in which the attacker has complete access to the ML model, its structure, and its parameters, or 2) a black-box (alternatively gray-box) scenario with non-realistic requirements such as a high query budget or high computational resources for the attacker. In this paper, we propose two practical query-based (i.e., black-box) EIL attacks that target face recognition ML models and allow an attacker to infer sensitive implicit properties, such as the facial characteristics, gender, ethnicity, income level, and average age of the individuals in the training data, with a limited number of queries. The first proposed attack, referred to as the random noise injection (RNI) attack, exploits the effect of injecting random noise into input samples on the target model’s predictions. The second proposed attack, referred to as the property substitute model (PSM) attack, creates a substitute model for each property value examined, whose predictions are compared to the target model’s predictions. Our comprehensive evaluation (a total of 730 experiments) performed on the CelebA dataset shows that the proposed attacks outperform existing EIL attacks and successfully infer private information, posing a threat to the privacy and security of the face recognition models.

Abstract Object detection models, which are widely used in various domains (such as retail), have been shown to be vulnerable to adversarial attacks. Existing methods for detecting adversarial attacks on object detectors have had difficulty detecting new real-life attacks. We present X-Detect, a novel adversarial patch detector that can: (1) detect adversarial samples in real time, allowing the defender to take preventive action; (2) provide explanations for the alerts raised to support the defender’s decision-making process, and (3) handle unfamiliar threats in the form of new attacks. Given a new scene, X-Detect uses an ensemble of explainable-by-design detectors that utilize object extraction, scene manipulation, and feature transformation techniques to determine whether an alert needs to be raised. X-Detect was evaluated in both the physical and digital space using five different attack scenarios (including adaptive attacks) and the benchmark COCO dataset and our new Superstore dataset. The physical evaluation was performed using a smart shopping cart setup in real-world settings and included 17 adversarial patch attacks recorded in 1700 adversarial videos. The results showed that X-Detect outperforms the state-of-the-art methods in distinguishing between benign and adversarial scenes for all attack scenarios while maintaining a 0% FPR (no false alarms) and providing actionable explanations for the alerts raised. A demo is available.

Utilizing existing methods for bias detection in machine learning (ML) models is challenging since each method: 1) explores a different ethical aspect of bias, which may result in contradictory output among the different methods; 2) provides output in a different range/scale and therefore cannot be compared with other methods; and 3) requires different input, thereby requiring a human expert's involvement to adjust each method according to the model examined. In this article, we present BENN, a novel bias estimation method that uses a pretrained unsupervised deep neural network. Given an ML model and data samples, BENN provides a bias estimation for every feature based on the examined model's predictions. We evaluated BENN using three benchmark datasets, one proprietary churn prediction model used by a European telecommunications company, and a synthetic dataset that includes both a biased feature and a fair one. BENN's results were compared with an ensemble of 21 existing bias estimation methods. The evaluation results show that BENN provides bias estimations that are aligned with those of the ensemble while offering significant advantages, including the fact that it is a generic approach (i.e., can be applied to any ML model) and does not require a domain expert.

A machine learning (ML) fairness estimator, which is used to assess an ML model’s fairness, should satisfy several conditions when used in real-life settings. Specifically, it should: i) support a comprehensive fairness evaluation that explores all ethical aspects; ii) be flexible and support different ML model settings; iii) enable comparison between different evaluations and ML models; and iv) provide reasoning and explanations for the fairness assessments produced. Existing methods do not sufficiently satisfy all of the above conditions. In this paper, we present FEPC (Fairness Estimation using Prototypes and Critics for tabular data), a novel method for fairness assessment that provides explanations and reasoning for its assessments by using an adversarial attack and customized fairness measurement. Given an ML model and data records, FEPC performs a comprehensive fairness evaluation and produces a fairness assessment for each examined feature. FEPC was evaluated using two benchmark datasets (ProPublica COMPAS and Statlog datasets) and a synthetic dataset containing two features, one of which is biased and one of which is fair, and compared to existing fairness assessment methods. The evaluation demonstrates that FEPC satisfies all of the conditions, making it suitable for real-life settings, and outperforms existing methods.