VoCoT：在大规模多模态模型中释放视觉 grounding 的多步推理能力
VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models
AI总结：
本文提出 VoCoT 框架，通过以对象为中心的视觉 grounding 多步推理路径，并构建指令微调数据集训练出 7B 模型 VolCano，在复杂推理基准上超越 GPT-4V 等 SOTA 模型。
摘要：
尽管大规模多模态模型（LMMs）在各类任务中展现出令人印象深刻的能力，但它们在处理复杂任务时的有效性一直受到主流单步推理范式的限制。为此，本文提出了 VoCoT，一个面向 LMM 推理的多步视觉 grounding 的以对象为中心的思维链推理框架。VoCoT 具有两个关键特征：（1）围绕跨模态共享的对象级信息展开的以对象为中心的推理路径；（2）以多模态交错和对齐方式对对象概念进行视觉 grounding 表示，从而在长期生成过程中有效弥合 LMM 内部的模态差距。为了使 LMM 适应 VoCoT 推理，我们进一步构建了一个指令微调数据集。通过将 VoCoT 与主流的开源 LMM 架构相结合，我们开发了基于 VoCoT 的模型 VolCano。仅凭 7B 参数和有限的输入图像分辨率，VolCano 在各种场景中展现出优异的性能。在 CLEVR 和 EmbSpatial 等对复杂推理能力要求极高的基准测试中，VolCano 超越了包括强大的 GPT-4V 在内的 SOTA 模型。相关代码、数据和模型已在 https://github.com/RupertLuo/VoCoT 发布。
英文摘要：
While large multi-modal models (LMMs) have exhibited impressive capabilities across diverse tasks, their effectiveness in handling complex tasks has been limited by the prevailing single-step reasoning paradigm. To this end, this paper proposes VoCoT, a multi-step Visually grounded object-centric Chain-of-Thought reasoning framework tailored for inference with LMMs. VoCoT is characterized by two key features: (1) object-centric reasoning paths that revolve around cross-modal shared object-level information, and (2) visually grounded representation of object concepts in a multi-modal interleaved and aligned manner, which effectively bridges the modality gap within LMMs during long-term generation. To adapt LMMs in reasoning with VoCoT, we further construct an instruction-tuning dataset. By combining VoCoT with the prevalent open-source LMM architectures, we develop a VoCoT-based model, VolCano. With only 7B parameters and limited input image resolution, VolCano demonstrates excellent performance across various scenarios. In benchmarks like CLEVR and EmbSpatial, which highly require complex reasoning capabilities, VolCano outperforms SOTA models, including powerful GPT-4V. Related code, data and models are released in https://github.com/RupertLuo/VoCoT.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
