Improving the safety of vision-language models like CLIP via fine-tuning often comes at a steep price, causing significant drops in their generalization performance. We find this trade-off stems from rigid alignment strategies that force unsafe concepts toward single, predefined safe targets, disrupting the model's learned semantic structure. To address this, we propose a proximity-aware approach: redirecting unsafe concepts to their semantically closest safe alternatives to minimize representational change. We introduce SafeR-CLIP, a fine-tuning framework that applies this principle of minimal intervention. SafeR-CLIP successfully reconciles safety and performance, recovering up to 8.0% in zero-shot accuracy over prior methods while maintaining robust safety. To support more rigorous evaluation, we also contribute NSFWCaps, a new benchmark of 1,000 highly-aligned pairs for testing safety under distributional shift. Our work shows that respecting the geometry of pretrained representations is key to achieving safety without sacrificing performance.

Adversarial attacks perturb data to modify a model’s prediction. These perturbations can be crafted in a white-box or black-box setting, depending on whether the target model architecture/weights are known or unknown. Compromised architecture and weights of a model makes it vulnerable to the more powerful white-box attacks. In this work, we determine if a deep model is compromised by distinguishing white-box from black-box adversarial attacks. The proposed method utilizes the internal representations of the target model and a proxy model to increase the detector efficacy. Additionally, it employs a spatial smoothing module to control the strength of white-box attacks relative to black-box attacks, and a proxy module to aid in measuring the transferability of the attack. Both modules work in tandem to increase the contrast of the internal representations between white-box and black-box attacks for better discrimination. We perform a detailed ablation of our method to showcase the importance of the different modules, and show that the spatial smoothing and proxy defense techniques enable our framework to significantly outperform the simple classification baseline on common vision datasets.