LLM-wrapper：面向指代表达理解的视觉语言模型黑盒语义感知自适应
LLM-wrapper: Black-Box Semantic-Aware Adaptation of Vision-Language Models for Referring Expression Comprehension
AI总结：
本文提出LLM-wrapper方法，利用轻量微调的大语言模型对黑盒视觉语言模型进行语义感知自适应，通过推理选择最佳边界框以解决指代表达理解任务，无需访问模型内部机制且具备强通用性。
摘要：
视觉语言模型（VLMs）在各种开放词汇任务中展现了卓越能力，但其零样本性能落后于特定任务微调模型，特别是在指代表达理解（REC）等复杂任务中。微调通常需要对模型架构和权重进行“白盒”访问，由于专有或隐私问题，这并不总是可行的。在本研究中，我们提出了LLM-wrapper，一种使用大语言模型（LLMs）对REC任务进行VLMs“黑盒”自适应的方法。LLM-wrapper利用经轻量微调提升的LLMs推理能力，从零样本黑盒VLM生成的候选中，选择与指代表达最匹配的边界框。我们的方法具有多项优势：无需访问内部机制即可实现闭源模型自适应；具有通用性，适用于任何VLM；可迁移至新的VLMs和数据集；并支持VLMs集合的自适应。我们在多个数据集上使用不同的VLMs和LLMs评估了LLM-wrapper，展示了显著的性能提升并突出了方法的通用性。尽管LLM-wrapper并非旨在直接与标准白盒微调竞争，但它为黑盒VLM自适应提供了一种实用且有效的替代方案。代码和检查点可在 https://github.com/valeoai/LLM_wrapper 获取。
英文摘要：
Vision Language Models (VLMs) have demonstrated remarkable capabilities in various open-vocabulary tasks, yet their zero-shot performance lags behind task-specific fine-tuned models, particularly in complex tasks like Referring Expression Comprehension (REC). Fine-tuning usually requires 'white-box' access to the model's architecture and weights, which is not always feasible due to proprietary or privacy concerns. In this work, we propose LLM-wrapper, a method for 'black-box' adaptation of VLMs for the REC task using Large Language Models (LLMs). LLM-wrapper capitalizes on the reasoning abilities of LLMs, improved with a light fine-tuning, to select the most relevant bounding box matching the referring expression, from candidates generated by a zero-shot black-box VLM. Our approach offers several advantages: it enables the adaptation of closed-source models without needing access to their internal workings, it is versatile as it works with any VLM, it transfers to new VLMs and datasets, and it allows for the adaptation of an ensemble of VLMs. We evaluate LLM-wrapper on multiple datasets using different VLMs and LLMs, demonstrating significant performance improvements and highlighting the versatility of our method. While LLM-wrapper is not meant to directly compete with standard white-box fine-tuning, it offers a practical and effective alternative for black-box VLM adaptation. Code and checkpoints are available at https://github.com/valeoai/LLM_wrapper .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
