Visual Prompt (VP) learning has rapidly emerged as a popular paradigm for parameter-efficient task adaptation in CLIP-based models. However, while VP optimizes pixel-space vectors without altering CLIP’s internal weights, this decoupled design inadvertently introduces critical security vulnerabilities. Attackers can exploit VP to implant covert backdoors using imperceptible trigger patterns that bypass traditional anomaly detection mechanisms, posing significant risks to real-world applications. Existing backdoor techniques, however, rely on visually noticeable patterns and exhibit inconsistencies in representation alignment, which make them prone to detection and ineffective against robust defenses. In response to these limitations, we propose Stealthy Backdoor Carriers (SBC), a novel attack framework that leverages CLIP’s inherent vulnerabilities to covertly and persistently inject backdoors.SBCadopts a dual-constrained optimization strategy that balances imperceptibility—minimizing trigger perturbations for visual stealth—and cross-modal embedding alignment—ensuring poisoned and target samples share consistent representations within CLIP’s multimodal space. Experimental results across five benchmark datasets demonstrateSBC’s exceptional effectiveness, achieving a +49.63% improvement in attack success rate relative to existing methods while maintaining robustness against advanced defenses like Neural Cleanse. Our work highlights the need for reevaluating the security implications of VP learning frameworks and provides valuable insights for mitigating prompt-based vulnerabilities in AI systems. Our code is available at https://github.com/Maozhen-Zhang/sbc.git.

This work presents a weakly supervised referring image segmentation method, namedCollabLearn, that segments objects described by free-form referring expression utilizing solely image-text pairs. Existing methods suffer from incorrect localization of referring expressions due to the lack of high-level semantics in cross-modal alignment or rough segmentation of referenced objects stemming from the absence of low-level details. To address these issues, we propose an innovative framework for generating cross-modal features encompassing both high-level semantics and low-level details via two fusion modules: a semantic awareness module and a detail cognition module. Each of these modules generates an activation map, and they mutually correct each other through a collaborative learning strategy. Specifically, the semantic awareness module performs in-depth cross-modal interaction and achieves accurate localization in a top-down manner. The detail cognition module facilitates the segmentation of entire objects in a bottom-up manner. A collaborative learning strategy is designed to enable interaction between these two modules, enforcing sufficient vision-language alignment. Experiments on three benchmarks demonstrate that CollabLearn consistently outperforms state-of-the-art weakly supervised methods.

Scene graph generation (SGG) effectively improves semantic understanding of the visual world. However, the recent interest of researchers focuses on enhancing SGG in non-adversarial settings, which raises our curiosity about the adversarial robustness of SGG models. To bridge this gap, we perform adversarial attacks on two typical SGG tasks, Scene Graph Detection (SGDet) and Scene Graph Classification (SGCls). Specifically, we initially propose a bounding box relabeling method to reconstruct reasonable attack targets for SGCls. It solves the inconsistency between the specified bounding boxes and the scene graphs selected as attack targets. Subsequently, we introduce a two-step weighted attack by removing the predicted objects and relational triples that affect attack performance, which significantly increases the success rate of adversarial attacks on two SGG tasks. Extensive experiments demonstrate the effectiveness of our methods on five popular SGG models and four adversarial attacks. The Pytorch® implementation can be downloaded from an open-source Github project https://github.com/Dlut-lab-zmn/SGG_Attack.

Fast adversarial training (FAT) is beneficial for improving the adversarial robustness of neural networks. However, previous FAT work has encountered a significant issue known as catastrophic overfitting when dealing with large perturbation budgets, i.e. the adversarial robustness of models declines to near zero during training. To address this, we analyze the training process of prior FAT work and observe that catastrophic overfitting is accompanied by the appearance of loss convergence outliers. Therefore, we argue a moderately smooth loss convergence process will be a stable FAT process that solves catastrophic overfitting. To obtain a smooth loss convergence process, we propose a novel oscillatory constraint (dubbed ConvergeSmooth) to limit the loss difference between adjacent epochs. The convergence stride of ConvergeSmooth is introduced to balance convergence and smoothing. Likewise, we design weight centralization without introducing additional hyperparameters other than the loss balance coefficient. Our proposed methods are attack-agnostic and thus can improve the training stability of various FAT techniques. Extensive experiments on popular datasets show that the proposed methods efficiently avoid catastrophic overfitting and outperform all previous FAT methods. Code is available at https://github.com/FAT-CS/ConvergeSmooth.

In recent years, there has been increasing interest in studying the adversarial attack, which poses potential risks to deep learning applications and has stimulated numerous researches, e.g. improving the robustness of deep neural networks. In this work, we propose a novel double-stream architecture – Guided Erasable Adversarial Attack (GEAA) – for protecting high-quality labeled data with high commercial values under data-sharing scenarios. GEAA contains three phases, the double-stream adversarial attack, denoising reconstruction, and watermark extraction. Specifically, the double-stream adversarial attack injects erasable perturbations into the training data to avoid database abuse. The denoising reconstruction rebuilds the traceable denoising data from adversarial examples. The watermark extraction recovers identity information from the denoised data for copyright protection. Additionally, we introduce the annealing optimization strategy to balance these phases and a boundary constraint to degrade the availability of adversarial examples. Through extensive experiments, we demonstrate the effectiveness of the proposed framework in data protection. The Pytorch® implementations of GEAA can be downloaded from an open-source Github project https://github.com/Dlut-lab-zmn/ GEAA-for-data-protection.

Recent studies highlight the vulnerability of convolutional neural networks (CNNs) to adversarial attacks, which also calls into question the reliability of forensic methods. Existing adversarial attacks generate one-to-one noise, which means these methods have not learned the fingerprint information. Therefore, we introduce two powerful attacks, fingerprint copy-move attack, and joint feature-based auto-learning attack. To validate the performance of attack methods, we move a step ahead and introduce the higher possible defense mechanism relation mismatch. which expands the characterization differences of classifiers in the same classification network. Extensive experiments show that relation mismatch is superior in recognizing adversarial examples and prove that the proposed fingerprint-based attacks are more powerful. Both proposed attacks show excellent attack transferability to unknown samples. The Pytorch® implementations of these methods can download from an open-source GitHub projecthttps://github.com/Dlut-lab-zmn/Source-attack.

Deep neural networks (DNNs) have seen extensive studies on image recognition and classification, image segmentation, and related topics. However, recent studies show that DNNs are vulnerable in defending adversarial examples. The classification network can be deceived by adding a small amount of perturbation to clean samples. There are challenges when researchers want to design a general approach to defend against a wide variety of adversarial examples. To solve this problem, we introduce a defensive method to prevent adversarial examples from generating. Instead of designing a stronger classifier, we built a more robust classification system that can be viewed as a structural black box. After adding a buffer to the classification system, attackers can be efficiently deceived. The real evaluation results of the generated adversarial examples are often contrary to what the attacker thinks. Additionally, we do not assume a specific attack method premise. This incognizance to underlying attacks demonstrates the generalizability of the buffer to potential adversarial attacks. Extensive experiments indicate that the defense method greatly improves the security performance of DNNs.