GLOV:引导大语言模型作为视觉语言模型的隐式优化器
GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models
AI总结:
本文提出GLOV框架,利用大语言模型作为视觉语言模型的隐式优化器,通过排序提示和偏移向量引导生成,在目标识别和VLM安全性任务上显著提升性能。
摘要:
在这项工作中,我们提出了GLOV,它使大语言模型(LLMs)能够作为视觉语言模型(VLMs)的隐式优化器,以增强下游视觉任务。GLOV通过向下游任务描述提示LLM,查询其合适的VLM提示(例如,用于CLIP的零样本分类)。这些提示根据其对下游视觉任务的适应度进行排序。在每个相应的优化步骤中,排序后的提示作为上下文示例(连同其准确率)输入,以使LLM了解下游VLM偏好的提示类型。此外,我们在每个优化步骤中通过向网络的中间层添加一个偏移向量——该向量由先前正负解之间的嵌入差异计算得出——来明确引导LLM的生成。该偏移向量使LLM的生成偏向于下游VLM偏好的语言类型,从而提升下游视觉任务的性能。我们在两个任务上全面评估了我们的GLOV:目标识别以及增强VLM安全性这一关键任务。我们的GLOV在双编码器(如CLIP)和编码器-解码器(如LlaVA)模型的目标识别上分别实现了最高15.0%和57.5%的性能提升,并将针对最先进VLMs的攻击成功率(ASR)降低了最高60.7%。
英文摘要:
In this work, we propose GLOV, which enables Large Language Models (LLMs) to act as implicit optimizers for Vision-Language Models (VLMs) to enhance downstream vision tasks. GLOV prompts an LLM with the downstream task description, querying it for suitable VLM prompts (e.g., for zero-shot classification with CLIP). These prompts are ranked according to their fitness for the downstream vision task. In each respective optimization step, the ranked prompts are fed as in-context examples (with their accuracies) to equip the LLM with the knowledge of the type of prompts preferred by the downstream VLM. Furthermore, we explicitly guide the LLM's generation at each optimization step by adding an offset vector -- calculated from the embedding differences between previous positive and negative solutions -- to the intermediate layer of the network for the next generation. This offset vector biases the LLM generation toward the type of language the downstream VLM prefers, resulting in enhanced performance on the downstream vision tasks. We comprehensively evaluate our GLOV on two tasks: object recognition and the critical task of enhancing VLM safety. Our GLOV shows performance improvement by up to 15.0% and 57.5% for dual-encoder (e.g., CLIP) and encoder-decoder (e.g., LlaVA) models for object recognition and reduces the attack success rate (ASR) on state-of-the-art VLMs by up to $60.7\%$.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。