基于图像的提示注入：通过视觉嵌入的对抗性指令劫持多模态大语言模型
Image-based Prompt Injection: Hijacking Multimodal LLMs through Visually Embedded Adversarial Instructions
AI总结：
研究提出基于图像的提示注入攻击方法，通过视觉嵌入对抗性指令，成功操控多模态大语言模型输出，揭示了该攻击的有效性和潜在威胁。
摘要：
多模态大语言模型（MLLMs）整合了视觉和文本以驱动应用，但这种整合引入了新的漏洞。我们研究了基于图像的提示注入（IPI），这是一种黑盒攻击，其中对抗性指令被嵌入到自然图像中以覆盖模型行为。我们的端到端IPI管道结合了基于分割的区域选择、自适应字体缩放和背景感知渲染，以隐藏提示以避免被人类感知，同时保持模型的可解释性。使用COCO数据集和GPT-4-turbo，我们评估了12种对抗性提示策略和多种嵌入配置。结果表明，IPI可以可靠地操控模型的输出，最有效的配置在 stealth 约束下可达到64%的攻击成功率。这些发现突显了IPI在黑盒设置中的实际威胁，并强调了对多模态提示注入的防御需求。
英文摘要：
Multimodal Large Language Models (MLLMs) integrate vision and text to power applications, but this integration introduces new vulnerabilities. We study Image-based Prompt Injection (IPI), a black-box attack in which adversarial instructions are embedded into natural images to override model behavior. Our end-to-end IPI pipeline incorporates segmentation-based region selection, adaptive font scaling, and background-aware rendering to conceal prompts from human perception while preserving model interpretability. Using the COCO dataset and GPT-4-turbo, we evaluate 12 adversarial prompt strategies and multiple embedding configurations. The results show that IPI can reliably manipulate the output of the model, with the most effective configuration achieving up to 64\% attack success under stealth constraints. These findings highlight IPI as a practical threat in black-box settings and underscore the need for defenses against multimodal prompt injection.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
