多模态大语言模型中推理的即插即用 Grounding
Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models
AI总结：
针对 MLLM 难辨高分辨率图像细节的问题，提出即插即用框架 P2G，借助外部专家智能体将推理锚定到关键视觉与文字元素，并构建 P2GB 基准，验证 7B 模型可达到接近 GPT-4V 的性能。
摘要：
多模态大语言模型（MLLM）凭借其先进的指令遵循与推理能力而兴起，显著推动了视觉推理领域的发展。然而，由于其图像分词过程存在局限，大多数 MLLM 难以捕捉图像中文字和物体的精细细节，尤其是在高分辨率样本中。为克服这一局限，我们提出 P2G，这是一种用于在 MLLM 中实现即插即用 grounding 的新型框架。P2G 利用 MLLM 的工具使用潜力，调用专家智能体将推理即时 grounding 到图像中的关键视觉和文字元素，从而通过多模态提示实现审慎推理。此外，我们构建了 P2GB，这是一个旨在评估 MLLM 在具有挑战性的高分辨率图像中理解物体间关系和文字内容能力的基准。在视觉推理任务上的大量实验证明了 P2G 的优越性：使用 7B 主干模型时，其在 P2GB 上取得了与 GPT-4V 相当的性能。我们的工作强调了在 MLLM 中利用外部智能体 grounding 推理的潜力，为单纯扩大模型规模提供了一种有前景的替代方案。
英文摘要：
The rise of Multimodal Large Language Models (MLLMs), renowned for their advanced instruction-following and reasoning capabilities, has significantly propelled the field of visual reasoning. However, due to limitations in their image tokenization processes, most MLLMs struggle to capture fine details of text and objects in images, especially in high-resolution samples. To overcome this limitation, we introduce P2G, a novel framework for plug-and-play grounding in MLLMs. P2G utilizes the tool-usage potential of MLLMs to employ expert agents for on-the-fly grounding of reasoning into critical visual and textual elements in images, thereby enabling deliberate reasoning through multimodal prompting. Additionally, we develop P2GB, a benchmark designed to evaluate MLLMs' proficiency in understanding inter-object relationships and textual content in challenging high-resolution images. Extensive experiments on visual reasoning tasks demonstrate the superiority of P2G, achieving performance comparable to GPT-4V on P2GB with a 7B backbone. Our work underscores the potential of grounding reasoning with external agents in MLLMs, presenting a promising alternative to mere model scaling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
