AI 中文总结
研究视觉语言模型对抗攻击问题，提出GhostPrompt方法，通过联合优化提炼图像不变对抗特征，跨图像引导模型输出，相比现有基线攻击成功率显著提高且计算时间大幅减少。
AI 中文摘要
视觉语言模型（VLM）容易受到对抗攻击，即对图像或文本的细微扰动会导致错误输出。大多数基于文本的攻击是从以语言模型为中心的方法改编而来，优化过程中视觉输入固定，导致对抗性提示与特定图像相关，限制了攻击效果。为此，我们引入跨图像可转移性这一新视角，提出GhostPrompt，通过联合优化将图像不变的对抗特征提炼到提示中，可跨不同图像引导VLM输出。实验表明，与现有基线相比，我们的方法攻击成功率提高超30%，计算时间减少约70%。
英文摘要
Vision-Language Models (VLMs) are known to be vulnerable to adversarial attacks, where subtle perturbations to images or texts induce erroneous outputs. However, most text-based attacks are adapted from language-model-centric methods, in which the visual input is fixed during optimization, resulting in adversarial prompts that are tied to specific images and thus limiting their attack effectiveness. To this end, we first introduce a new research perspective: cross-image transferability for adversarial prompts. We then propose GhostPrompt, an adversarial prompt that is optimized once and reused to steer VLM outputs toward attacker-specified responses across diverse images. GhostPrompt employs a joint optimization that distills image-invariant adversarial features into the prompt by "worst-case" generation. Specifically, it alternates between constructing hard visual conditions for the current prompt and updating the prompt to remain effective under these conditions. Extensive experiments on prevalent VLMs verify that \ourmethod achieves an improvement of over 30% in attack success rates compared to state-of-the-art (SoTA) baselines, while reducing computation time by ~70%. Our code is avalable at https://github.com/Ye-ze-yu/GhostPrompt.
CommentsAccepted to ACM MM 2026. Code: this https://github.com/Ye-ze-yu/GhostPrompt
