XSPA：构建不可察觉的X形稀疏对抗扰动以对视觉语言模型的可迁移攻击
XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs
AI总结：
本文提出XSPA攻击，通过限制扰动为两条相交对角线，测试VLMs在稀疏扰动下的鲁棒性，实验表明其能显著破坏跨任务语义。
摘要：
视觉语言模型（VLMs）依赖于共享的视觉-文本表示空间来执行零样本分类、图像描述生成和视觉问答（VQA）等任务。尽管这种共享空间实现了强大的跨任务泛化能力，但也可能引入一种共同的漏洞：微小的视觉扰动可以通过共享嵌入空间传播，并导致跨任务的语义故障。这种风险在交互性和决策支持场景中尤为重要，但目前尚不清楚VLMs是否对高度受限、稀疏且几何固定的扰动具有鲁棒性。为此，我们提出了X形稀疏像素攻击（XSPA），一种不可察觉的结构化攻击，将扰动限制在两条相交的对角线。与密集扰动或灵活的局部化补丁相比，XSPA在更严格的攻击预算下运行，因此提供了对VLM鲁棒性的更严格测试。在这一稀疏支持下，XSPA联合优化分类目标、跨任务语义引导以及扰动幅度和沿线平滑度的正则化，诱导出可迁移的分类错误以及在描述生成和VQA中的语义漂移，同时保持视觉的细微差别。在默认设置下，XSPA仅修改约1.76%的图像像素。在COCO数据集上的实验表明，XSPA在所有三个任务上都一致地降低了性能。零样本准确率在OpenAI CLIP ViT-L/14上下降了52.33点，在OpenCLIP ViT-B/16上下降了67.00点，而由GPT-4评估的描述一致性下降了高达58.60点，VQA正确率下降了高达44.38点。这些结果表明，即使是非常稀疏且视觉上微妙的扰动，具有固定几何先验，也能显著破坏VLMs中的跨任务语义，揭示了当前多模态系统中显著的鲁棒性差距。
英文摘要：
Vision-language models (VLMs) share visual-textual representations across zero-shot classification, image captioning, and visual question answering (VQA), creating a pathway through which subtle perturbations may cause failures across tasks. We introduce X-shaped Sparse Pixel Attack (XSPA), a structured attack that restricts perturbations to two intersecting diagonal lines. Within this fixed support, XSPA jointly optimizes a classification objective, cross-task semantic guidance, perturbation magnitude, and linewise smoothness. Under the default setting, it changes about 1.04\% of image pixels. On COCO, XSPA reduces zero-shot accuracy by 52.33 points on OpenAI CLIP ViT-L/14 and 67.00 points on OpenCLIP ViT-B/16; GPT-4-based caption consistency and VQA correctness decrease by up to 58.60 and 44.25 points, respectively. Matched-budget experiments show that an adaptive saliency Top-k support achieves higher attack success than the fixed X-shaped support, while XSPA provides better SSIM and LPIPS but worse PSNR and MAE. Same-image analysis finds no three-task targeted success, indicating that strong surrogate classification effects do not reliably translate into targeted output-space failures. These results position XSPA as a controlled stress test for studying how sparse fixed geometry affects VLM robustness, rather than as a universally superior attack.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
