ParGo：以局部与全局视图桥接视觉与语言
ParGo: Bridging Vision-Language with Partial and Global Views
AI总结：
本文提出融合全局与局部视图的 ParGo 投影器，并构建百万级 ParGoCap-1M-PT 数据集，以增强 MLLM 的视觉语言对齐和细节感知。
摘要：
本工作提出 ParGo，这是一种新颖的局部—全局投影器，旨在为多模态大语言模型（MLLM）连接视觉和语言模态。不同于以往依赖基于全局注意力的投影器的工作，我们的 ParGo 通过融合全局视图与局部视图，弥合分别预训练的视觉编码器与 LLM 之间的表示鸿沟，从而缓解对显著区域的过度强调。为促进 ParGo 的有效训练，我们收集了一个名为 ParGoCap-1M-PT 的大规模带细节描述图像—文本数据集，其中包含 100 万张配有高质量描述的图像。在多个 MLLM 基准上的大量实验证明了 ParGo 的有效性，并凸显其在对齐视觉与语言模态方面的优越性。与传统 Q-Former 投影器相比，我们的 ParGo 在 MME 基准上取得了 259.96 的提升。此外，实验表明，ParGo 显著优于其他投影器，尤其是在强调细节感知能力的任务中。
英文摘要：
This work presents ParGo, a novel Partial-Global projector designed to connect the vision and language modalities for Multimodal Large Language Models (MLLMs). Unlike previous works that rely on global attention-based projectors, our ParGo bridges the representation gap between the separately pre-trained vision encoders and the LLMs by integrating global and partial views, which alleviates the overemphasis on prominent regions. To facilitate the effective training of ParGo, we collect a large-scale detail-captioned image-text dataset named ParGoCap-1M-PT, consisting of 1 million images paired with high-quality captions. Extensive experiments on several MLLM benchmarks demonstrate the effectiveness of our ParGo, highlighting its superiority in aligning vision and language modalities. Compared to conventional Q-Former projector, our ParGo achieves an improvement of 259.96 in MME benchmark. Furthermore, our experiments reveal that ParGo significantly outperforms other projectors, particularly in tasks that emphasize detail perception ability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
