ViCor：通过大语言模型衔接视觉理解与常识推理
ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models
AI总结：
本研究针对视觉常识推理中VLM与LLM各有优劣的问题，提出ViCor协作框架，由LLM分类问题类型并引导VLM获取视觉信息，在无需领域微调的情况下性能优于同类方法。
摘要：
在本研究中，我们探索了预训练视觉语言模型（VLM）与大语言模型（LLM）在视觉常识推理（VCR）问题上的协同能力。我们发现，基于VLM和LLM的决策流程擅长处理不同类型的VCR问题。预训练VLM在涉及理解视觉字面内容的问题（我们将其定义为视觉常识理解，VCU）上表现优异；而对于目标是推断超出图像内容的结论的问题（我们定义为视觉常识推断，VCI），VLM会面临困难，LLM在获得充足视觉证据的情况下，却能很好地利用常识推断答案。我们通过让LLM将VCR问题划分为这两个类别进行了实证验证，结果表明，在两个子问题上，VLM与带图像描述的LLM决策流程之间存在显著差异。此外，我们还发现VLM的被动感知存在一个挑战：它可能会遗漏关键上下文信息，进而导致LLM推理错误。基于这些发现，我们提出了一种名为ViCor的协作方法：预训练LLM充当问题分类器以分析问题类别，随后要么直接使用VLM回答问题，要么主动引导VLM聚焦并收集相关视觉元素，为潜在的常识推断提供支持。我们在两个VCR基准数据集上评估了该框架，其表现优于所有无需领域内微调的其他方法。
英文摘要：
In our work, we explore the synergistic capabilities of pre-trained vision-and-language models (VLMs) and large language models (LLMs) on visual commonsense reasoning (VCR) problems. We find that VLMs and LLMs-based decision pipelines are good at different kinds of VCR problems. Pre-trained VLMs exhibit strong performance for problems involving understanding the literal visual content, which we noted as visual commonsense understanding (VCU). For problems where the goal is to infer conclusions beyond image content, which we noted as visual commonsense inference (VCI), VLMs face difficulties, while LLMs, given sufficient visual evidence, can use commonsense to infer the answer well. We empirically validate this by letting LLMs classify VCR problems into these two categories and show the significant difference between VLM and LLM with image caption decision pipelines on two subproblems. Moreover, we identify a challenge with VLMs' passive perception, which may miss crucial context information, leading to incorrect reasoning by LLMs. Based on these, we suggest a collaborative approach, named ViCor, where pre-trained LLMs serve as problem classifiers to analyze the problem category, then either use VLMs to answer the question directly or actively instruct VLMs to concentrate on and gather relevant visual elements to support potential commonsense inferences. We evaluate our framework on two VCR benchmark datasets and outperform all other methods that do not require in-domain fine-tuning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
