改写、增强、推理：面向视觉-语言模型的问题视觉 grounding
Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models
AI总结：
本文提出无梯度框架 RepARe，通过改写问题并利用 LVLM 置信度选择最优候选，以视觉 grounding 减少输入欠明确性，在多个 VQA 基准上显著提升零样本准确率。
摘要：
通过将大型语言模型（LLMs）与视觉编码器结合，形成大型视觉-语言模型（LVLMs），越来越多的视觉-语言任务可以在极少甚至无需训练的情况下处理，即以零样本和少样本方式完成。尽管这具有巨大优势，例如不需要训练数据或定制架构，但输入如何呈现给 LVLM 会对零样本模型性能产生重大影响。特别是，以欠明确方式表述的输入可能由于缺少视觉信息、复杂的隐式推理或语言歧义等因素而导致错误答案。因此，将视觉 grounding 信息作为预先澄清添加到输入中，应能通过减少欠明确性来提升模型性能，例如通过定位对象和消解指代歧义。类似地，在 VQA 设置中，改变问题的表述方式可以使模型更容易回答。为此，我们提出了 Rephrase, Augment and Reason（RepARe），这是一个无梯度框架，利用底层 LVLM 作为图像描述器和推理器来提取图像中的显著细节，从而提出对原始问题的修改。然后，我们使用 LVLM 对生成答案的置信度作为无监督评分函数，以选择最有可能提升零样本性能的改写问题。聚焦于三个视觉问答任务，我们表明 RepARe 可以在 VQAv2 上带来 3.85%（绝对）的零样本准确率提升，在 A-OKVQA 和 VizWiz 上分别带来 6.41% 和 7.94% 个百分点的提升。此外，我们发现使用金标准答案进行预言机问题候选选择可以在 VQA 准确率上实现高达 14.41% 的显著提升。通过广泛分析，我们证明 RepARe 的输出提高了句法复杂度，并有效利用了视觉-语言交互和冻结的 LLM。
英文摘要：
An increasing number of vision-language tasks can be handled with little to no training, i.e., in a zero and few-shot manner, by marrying large language models (LLMs) to vision encoders, resulting in large vision-language models (LVLMs). While this has huge upsides, such as not requiring training data or custom architectures, how an input is presented to an LVLM can have a major impact on zero-shot model performance. In particular, inputs phrased in an underspecified way can result in incorrect answers due to factors like missing visual information, complex implicit reasoning, or linguistic ambiguity. Therefore, adding visually-grounded information to the input as a preemptive clarification should improve model performance by reducing underspecification, e.g., by localizing objects and disambiguating references. Similarly, in the VQA setting, changing the way questions are framed can make them easier for models to answer. To this end, we present Rephrase, Augment and Reason (RepARe), a gradient-free framework that extracts salient details about the image using the underlying LVLM as a captioner and reasoner, in order to propose modifications to the original question. We then use the LVLM's confidence over a generated answer as an unsupervised scoring function to select the rephrased question most likely to improve zero-shot performance. Focusing on three visual question answering tasks, we show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively. Additionally, we find that using gold answers for oracle question candidate selection achieves a substantial gain in VQA accuracy by up to 14.41%. Through extensive analysis, we demonstrate that outputs from RepARe increase syntactic complexity, and effectively utilize vision-language interaction and the frozen LLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
