大型语言模型是视觉推理协调器
Large Language Models are Visual Reasoning Coordinators
AI总结：
提出Cola范式，利用大型语言模型协调多个视觉语言模型进行视觉推理，在多项视觉任务上达到最先进或具竞争力的性能。
摘要：
视觉推理需要对世界的多模态感知和常识认知。近期，已有多种视觉语言模型（VLM）被提出，在各个领域展现出卓越的常识推理能力。然而，如何利用这些互补型 VLM 的集体力量却鲜有探索。诸如集成等现有方法在聚合这些模型以实现所需的高阶通信方面仍面临困难。在本研究中，我们提出了 Cola，这是一种协调多个 VLM 进行视觉推理的新范式。我们的核心洞察是，大型语言模型（LLM）能够通过促进自然语言通信，利用 VLM 独特且互补的能力来高效协调它们。大量实验表明，我们的指令微调变体 Cola-FT 在视觉问答（VQA）、外部知识 VQA、视觉蕴含以及视觉空间推理任务上取得了最先进的性能。此外，我们展示了我们的上下文学习变体 Cola-Zero 在零样本和少样本设置下表现出色，无需微调。通过系统的消融研究和可视化，我们验证了协调器 LLM 确实理解了指令提示以及 VLM 的各自功能；随后它协调这些模型，实现了令人印象深刻的视觉推理能力。
英文摘要：
Visual reasoning requires multimodal perception and commonsense cognition of the world. Recently, multiple vision-language models (VLMs) have been proposed with excellent commonsense reasoning ability in various domains. However, how to harness the collective power of these complementary VLMs is rarely explored. Existing methods like ensemble still struggle to aggregate these models with the desired higher-order communications. In this work, we propose Cola, a novel paradigm that coordinates multiple VLMs for visual reasoning. Our key insight is that a large language model (LLM) can efficiently coordinate multiple VLMs by facilitating natural language communication that leverages their distinct and complementary capabilities. Extensive experiments demonstrate that our instruction tuning variant, Cola-FT, achieves state-of-the-art performance on visual question answering (VQA), outside knowledge VQA, visual entailment, and visual spatial reasoning tasks. Moreover, we show that our in-context learning variant, Cola-Zero, exhibits competitive performance in zero and few-shot settings, without finetuning. Through systematic ablation studies and visualizations, we validate that a coordinator LLM indeed comprehends the instruction prompts as well as the separate functionalities of VLMs; it then coordinates them to enable impressive visual reasoning capabilities.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
