利用冗长图像诱导大型视觉语言模型的高能耗-延迟
Inducing High Energy-Latency of Large Vision-Language Models with Verbose Images
AI总结:
本文提出冗长图像攻击方法,通过设计三种损失目标延迟EOS标记并增加生成标记的不确定性与多样性,诱导大型视觉语言模型在推理时生成超长序列,从而大幅增加能耗与延迟。
摘要:
诸如 GPT-4 等大型视觉语言模型(VLMs)在各种多模态任务中取得了卓越的性能。然而,VLMs 的部署需要大量的能耗和计算资源。一旦攻击者在 VLMs 推理期间恶意诱导高能耗和延迟时间(能耗-延迟成本),将耗尽计算资源。本文探索了关于 VLMs 可用性的这一攻击面,旨在诱导 VLMs 推理期间的高能耗-延迟成本。我们发现,可以通过最大化生成序列的长度来操纵 VLMs 推理期间的高能耗-延迟成本。为此,我们提出了冗长图像,目标是制作一种难以察觉的扰动,诱导 VLMs 在推理期间生成长句。具体而言,我们设计了三个损失目标。首先,提出一种损失来延迟序列结束(EOS)标记的出现,其中 EOS 标记是 VLMs 停止生成更多标记的信号。此外,提出了不确定性损失和标记多样性损失,分别增加每个生成标记的不确定性和整个生成序列中所有标记之间的多样性,这可以打破标记级和序列级的输出依赖性。此外,提出了一种时间权重调整算法,可以有效平衡这些损失。大量实验表明,与 MS-COCO 和 ImageNet 数据集上的原始图像相比,我们的冗长图像可将生成序列的长度分别增加 7.87 倍和 8.56 倍,这为各种应用带来了潜在挑战。代码可在 https://github.com/KuofengGao/Verbose_Images 获取。
英文摘要:
Large vision-language models (VLMs) such as GPT-4 have achieved exceptional performance across various multi-modal tasks. However, the deployment of VLMs necessitates substantial energy consumption and computational resources. Once attackers maliciously induce high energy consumption and latency time (energy-latency cost) during inference of VLMs, it will exhaust computational resources. In this paper, we explore this attack surface about availability of VLMs and aim to induce high energy-latency cost during inference of VLMs. We find that high energy-latency cost during inference of VLMs can be manipulated by maximizing the length of generated sequences. To this end, we propose verbose images, with the goal of crafting an imperceptible perturbation to induce VLMs to generate long sentences during inference. Concretely, we design three loss objectives. First, a loss is proposed to delay the occurrence of end-of-sequence (EOS) token, where EOS token is a signal for VLMs to stop generating further tokens. Moreover, an uncertainty loss and a token diversity loss are proposed to increase the uncertainty over each generated token and the diversity among all tokens of the whole generated sequence, respectively, which can break output dependency at token-level and sequence-level. Furthermore, a temporal weight adjustment algorithm is proposed, which can effectively balance these losses. Extensive experiments demonstrate that our verbose images can increase the length of generated sequences by 7.87 times and 8.56 times compared to original images on MS-COCO and ImageNet datasets, which presents potential challenges for various applications. Our code is available at https://github.com/KuofengGao/Verbose_Images.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。