通过学习的代理令牌实现忠实的接地视觉推理
Faithful Grounded Visual Reasoning via Learned Proxy-Tokens
AI总结：
提出Composer模型，利用基于学习代理令牌的视觉接地机制，通过离散符号指针显式索引图像潜在空间，在保持答案准确性的同时将视觉接地准确率提升9.0个百分点。
摘要：
多模态大语言模型（MLLMs）在视觉问答（VQA）中取得了显著成功，但其“黑箱”性质阻碍了在关键领域的部署。接地视觉推理（GVR）方法试图通过将文本理由与视觉接地信息（通常是文本坐标）显式耦合来提高可解释性。这种机制缺乏与视觉特征的可学习语义连接，常常导致语义-空间差距，模型产生与图像证据不对应的坐标幻觉。在这项工作中，我们引入了Composer，一种利用基于学习代理令牌的新型视觉接地机制来促进忠实可解释性的MLLM。这些离散符号指针显式索引图像潜在空间，使模型能够将视觉区域作为可寻址、可语义操作集进行操作。为了严格验证我们的新型接地机制，我们构建了ComposerGCoT数据集，该数据集经过合成以支持对推理一致性和接地准确性的整体评估。实验结果表明，Composer在最终答案准确性上与其基于坐标的对应模型达到性能持平，同时将视觉接地准确性提高了9.0个百分点。通过证明离散代理令牌比典型文本坐标更有效地捕获空间语义，我们确立了具有可学习语义连接的视觉接地机制是通往可信赖和可靠MLLM的有前途的路径。
英文摘要：
Multimodal Large Language Models (MLLMs) have achieved remarkable success in Visual Question Answering (VQA), yet their "black-box" nature hinders deployment in critical domains. Grounded Visual Reasoning (GVR) approaches attempt to improve interpretability by explicitly couple textual rationales with visual grounding information, which are typically textual coordinates. This mechanism lacks a learnable semantic link to the visual features, often resulting in a semantic-spatial gap where the model hallucinates coordinates that do not correspond to image evidences. In this work, we introduce Composer, a MLLM that leverages a novel visual grounding mechanism based on learned proxy-tokens to promote faithful interpretability. These discrete symbolic pointers explicitly index the image latent space, allowing the model to manipulate visual regions as addressable, semantically manipulable sets. To rigorously validate our novel grounding mechanism, we constructed ComposerGCoT, a dataset synthesized to enable holistic assessment of reasoning consistency and grounding accuracy. Experimental results indicate that Composer achieves performance parity with its coordinate-based counterpart in final answer accuracy, while improving visual grounding accuracy by +9.0 points. By demonstrating that discrete proxy-tokens capture spatial semantics more effectively than typical textual coordinates, we establish that visual grounding mechanisms with learnable semantic links represent a promising path toward trustworthy and reliable MLLMs.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
