弥合任务差距：CLIP及其衍生模型中的多任务对抗可迁移性
Bridging the Task Gap: Multi-Task Adversarial Transferability in CLIP and Its Derivatives
AI总结：
本文研究CLIP跨任务对抗迁移风险，提出MT-AdvCLIP框架，通过任务感知特征聚合损失增强细粒度任务扰动的跨任务泛化，在不增加扰动预算下将平均攻击成功率提升逾39%。
摘要：
作为一种通用视觉-语言预训练模型，CLIP在图文对齐任务中展现出强大的泛化能力，并已被广泛应用于图像分类和图文检索等下游任务。然而，它在目标检测和语义分割等细粒度任务上表现不佳。尽管许多变体旨在提升CLIP在这些任务上的性能，但其对对抗扰动的鲁棒性仍未得到充分研究。理解对抗样本如何跨任务迁移，是评估CLIP泛化极限和安全风险的关键。在本研究中，我们对基于CLIP的模型在对抗扰动下于图文检索、目标检测和语义分割任务中的跨任务迁移行为进行了系统的实证分析。我们发现，从细粒度任务（例如目标检测和语义分割）生成的对抗样本通常比来自粗粒度任务的对抗样本表现出更强的迁移潜力，从而能够更有效地攻击原始CLIP模型。受此观察启发，我们提出了一个新颖框架Multi-Task Adversarial CLIP（MT-AdvCLIP），该框架引入任务感知特征聚合损失，并生成具有增强跨任务泛化能力的扰动。该设计增强了细粒度任务模型对共享CLIP骨干网络的攻击有效性。在多个公开数据集上的实验结果表明，MT-AdvCLIP在不增加扰动预算的情况下，显著提高了针对各种CLIP衍生模型的对抗迁移成功率（跨多个任务的平均攻击成功率提升超过39%）。本研究揭示了多任务CLIP模型中对抗样本的迁移机制，为多任务鲁棒性评估和对抗样本设计提供了新的见解。
英文摘要：
As a general-purpose vision-language pretraining model, CLIP demonstrates strong generalization ability in image-text alignment tasks and has been widely adopted in downstream applications such as image classification and image-text retrieval. However, it struggles with fine-grained tasks such as object detection and semantic segmentation. While many variants aim to improve CLIP on these tasks, its robustness to adversarial perturbations remains underexplored. Understanding how adversarial examples transfer across tasks is key to assessing CLIP's generalization limits and security risks. In this work, we conduct a systematic empirical analysis of the cross-task transfer behavior of CLIP-based models on image-text retrieval, object detection, and semantic segmentation under adversarial perturbations. We find that adversarial examples generated from fine-grained tasks (e.g., object detection and semantic segmentation) often exhibit stronger transfer potential than those from coarse-grained tasks, enabling more effective attacks against the original CLIP model. Motivated by this observation, we propose a novel framework, Multi-Task Adversarial CLIP (MT-AdvCLIP), which introduces a task-aware feature aggregation loss and generates perturbations with enhanced cross-task generalization capability. This design strengthens the attack effectiveness of fine-grained task models on the shared CLIP backbone. Experimental results on multiple public datasets show that MT-AdvCLIP significantly improves the adversarial transfer success rate (The average attack success rate across multiple tasks is improved by over 39%.) against various CLIP-derived models, without increasing the perturbation budget. This study reveals the transfer mechanism of adversarial examples in multi-task CLIP models, offering new insights into multi-task robustness evaluation and adversarial example design.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
