自校准一致性可反击视觉-语言模型中的对抗鲁棒性
Self-Calibrated Consistency can Fight Back for Adversarial Robustness in Vision-Language Models
AI总结：
针对CLIP等视觉-语言模型易受对抗攻击且现有防御依赖标注数据的问题，提出自校准一致性（SCC）测试时防御方法，通过语义与空间一致性模块提升零样本鲁棒性，在22个基准上验证有效性。
摘要：
预训练的视觉-语言模型（如CLIP）在多个领域展现出强大的零样本能力，但仍高度易受对抗性扰动的影响，这些扰动破坏图像-文本对齐并损害可靠性。现有防御通常依赖带标签数据的对抗微调，限制了其在零样本设置中的适用性。在本工作中，我们识别了当前CLIP对抗攻击的两个关键弱点——缺乏语义引导和对视角变化的脆弱性——统称为语义脆弱性和视角脆弱性。为应对这些挑战，我们提出自校准一致性（SCC），一种有效的测试时防御方法。SCC包含两个互补模块：语义一致性，利用来自反击热身和多视角预测的软伪标签来正则化跨模态对齐，并将目标嵌入与易混淆的负样本分离；以及空间一致性，通过增强视图对齐受扰动的视觉预测，以在对抗扰动下稳定推理。这些模块共同构成一种即插即用的推理策略。在22个基准上的多种攻击设置下进行的大量实验表明，SCC持续提升CLIP的零样本鲁棒性，同时保持准确性，并可无缝集成到其他视觉-语言模型中以获得进一步提升。这些发现凸显了从CLIP建立对抗鲁棒范式的巨大潜力，其影响扩展到更广泛的视觉-语言领域，如BioMedCLIP。
英文摘要：
Pre-trained vision-language models (VLMs) such as CLIP have demonstrated strong zero-shot capabilities across diverse domains, yet remain highly vulnerable to adversarial perturbations that disrupt image-text alignment and compromise reliability. Existing defenses typically rely on adversarial fine-tuning with labeled data, limiting their applicability in zero-shot settings. In this work, we identify two key weaknesses of current CLIP adversarial attacks -- lack of semantic guidance and vulnerability to view variations -- collectively termed semantic and viewpoint fragility. To address these challenges, we propose Self-Calibrated Consistency (SCC), an effective test-time defense. SCC consists of two complementary modules: Semantic consistency, which leverages soft pseudo-labels from counterattack warm-up and multi-view predictions to regularize cross-modal alignment and separate the target embedding from confusable negatives; and Spatial consistency, aligning perturbed visual predictions via augmented views to stabilize inference under adversarial perturbations. Together, these modules form a plug-and-play inference strategy. Extensive experiments on 22 benchmarks under diverse attack settings show that SCC consistently improves the zero-shot robustness of CLIP while maintaining accuracy, and can be seamlessly integrated with other VLMs for further gains. These findings highlight the great potential of establishing an adversarially robust paradigm from CLIP, with implications extending to broader vision-language domains such as BioMedCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
