作为视觉-语言模型黑盒优化器的语言模型
Language Models as Black-Box Optimizers for Vision-Language Models
AI总结：
针对闭源VLM无法白盒微调的问题，提出用聊天式LLM通过自动爬山流程、结合正负文本反馈搜索最优提示的黑盒优化方法，在1样本分类任务上优于CoOp等方法，且提示可解释性、迁移性强，还可应用于DALL-E 3的生成等任务。
摘要：
在网络规模数据集上预训练的视觉-语言模型（VLM）在仅用少量数据微调时，已在下游任务中展现出卓越性能。然而，许多VLM依赖专有数据且并非开源，这限制了白盒微调方法的使用。为此，我们旨在开发一种通过自然语言提示优化VLM的黑盒方法，从而无需访问模型参数、特征嵌入甚至输出logits。我们提出采用基于聊天的大语言模型（LLM）来搜索VLM的最佳文本提示。具体而言，我们采用自动爬山流程，通过评估当前提示的性能，并要求LLM基于文本反馈对其进行优化，最终收敛到有效的提示，整个过程在无需人工介入的对话流程中完成。在具有挑战性的1样本图像分类设置下，我们的简单方法在包括ImageNet在内的11个数据集上平均优于白盒连续提示方法（CoOp）1.5%。我们的方法也优于人工设计的提示和LLM生成的提示。我们强调了结合正负提示的对话反馈的优势，表明LLM可以利用文本反馈中的隐含梯度方向实现更高效的搜索。此外，我们发现通过我们的策略生成的文本提示不仅更具可解释性，还能以黑盒方式在不同VLM架构间良好迁移。最后，我们将我们的框架应用于优化最先进的黑盒VLM（DALL-E 3），用于文本到图像生成、提示反转和个性化任务。
英文摘要：
Vision-language models (VLMs) pre-trained on web-scale datasets have demonstrated remarkable capabilities on downstream tasks when fine-tuned with minimal data. However, many VLMs rely on proprietary data and are not open-source, which restricts the use of white-box approaches for fine-tuning. As such, we aim to develop a black-box approach to optimize VLMs through natural language prompts, thereby avoiding the need to access model parameters, feature embeddings, or even output logits. We propose employing chat-based LLMs to search for the best text prompt for VLMs. Specifically, we adopt an automatic hill-climbing procedure that converges to an effective prompt by evaluating the performance of current prompts and asking LLMs to refine them based on textual feedback, all within a conversational process without human-in-the-loop. In a challenging 1-shot image classification setup, our simple approach surpasses the white-box continuous prompting method (CoOp) by an average of 1.5% across 11 datasets including ImageNet. Our approach also outperforms both human-engineered and LLM-generated prompts. We highlight the advantage of conversational feedback that incorporates both positive and negative prompts, suggesting that LLMs can utilize the implicit gradient direction in textual feedback for a more efficient search. In addition, we find that the text prompts generated through our strategy are not only more interpretable but also transfer well across different VLM architectures in a black-box manner. Lastly, we apply our framework to optimize the state-of-the-art black-box VLM (DALL-E 3) for text-to-image generation, prompt inversion, and personalization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
