Vision-Language Pre-training (VLP) models exhibit pronounced vulnerability to multimodal adversarial examples, necessitating rigorous robustness research, particularly for transferable attacks in black-box scenarios. Current research predominantly enhances attack transferability across VLP models by diversifying image and text inputs. However, during adversarial example generation, these methods often prioritize amplifying inter-modal semantic discrepancies (i.e., modality-discrepancy features) while overlooking model-specific semantic features critical to transferable attacks. To address this limitation, we pro pose a transferable Gradient Pruning Interactive Attack (GPI Attack), which integrates gradient-pruned image perturbations with semantic-oriented text perturbations through modality interaction. For image attacks, extreme backpropagated gradients may cause adversarial examples to highlight certain model specific features, leading to poor transferability. To suppress this feature, the textual modality guides the pruning of extreme gradients within intermediate VLP blocks, and these pruned gradients are subsequently employed to direct the generation of adversarial images. For text attacks, we consolidate the perturbation process solely at the embedding level, which reduces semantic discrepancies across hierarchical structures and significantly enhances the generalizability of adversarial texts. Experimental results demonstrate the effectiveness of GPI-Attack in image text retrieval tasks on multimodal datasets such as Flickr30K and MSCOCO. Additionally, the proposed gradient pruning technique is plug-and-play, showing performance improvements even when applied to baseline methods, indicating its potential as a valuable enhancement for attack performance.

Out-of-distribution (OOD) detection is crucial for ensuring the reliability and safety of machine learning models in real-world applications. While zero-shot OOD detection, which requires no training on in-distribution (ID) data, has become feasible with the emergence of vision-language models like CLIP, existing methods primarily focus on semantic matching and fail to fully capture distributional discrepancies. To address these limitations, we propose OT-DETECTOR, a novel framework that employs Optimal Transport (OT) to quantify both semantic and distributional discrepancies between test samples and ID labels. Specifically, we introduce cross-modal transport mass and transport cost as semantic-wise and distribution-wise OOD scores, respectively, enabling more robust detection of OOD samples. Additionally, we present a semantic-aware content refinement (SaCR) module, which utilizes semantic cues from ID labels to amplify the distributional discrepancy between ID and hard OOD samples. Extensive experiments on several benchmarks demonstrate that OT-DETECTOR achieves state-of-the-art performance across various OOD detection tasks, particularly in challenging hard-OOD scenarios.

Recent advances have heightened the interest in the adversarial transferability of Vision-Language Pre-training (VLP) models. However, most existing strategies constrained by two persistent limitations: suboptimal utilization of cross-modal interactive information, and inherent discrepancies across hierarchical textual representation. To address these challenges, we propose the Modality-Specific Interactive Attack (MSI-Attack), a novel approach that integrates semantic-level image perturbations with embedding-level text perturbations, all while maintaining minimal inter-modal constraints. In our image attack methodology, we introduce Multi-modal Integrated Gradients (MIG) to guide perturbations toward the core semantics of images, enriched by their associated deeply text information. This technique enhances transferability by capturing consistent features across various models, thereby effectively misleading similar-model perception areas. Additionally, we employ a momentum iteration strategy in conjunction with MIG, which amalgamates current and historical gradients to expedite the perturbation updates. For text attacks, we streamline the perturbation process by operating exclusively at the embedding level. This reduces semantic gaps across hierarchical structures and significantly enhances the generalizability of adversarial text. Moreover, we delve deeper into how semantic perturbations with varying degrees of similarity affect the overall attack effectiveness. Our experimental results on image-text retrieval tasks using the multi-modal datasets Flickr30K and MSCOCO underscore the efficacy of MSI-Attack. Our method achieves superior performance, setting a new state-of-the-art benchmark, all without the need for additional mechanisms.