GROUNDHOG：将大型语言模型接地到整体分割
GROUNDHOG: Grounding Large Language Models to Holistic Segmentation
AI总结：
本文提出 GROUNDHOG，通过掩码特征提取、视觉实体标记及实体检索合并，将 MLLM 的语言接地从边界框扩展到像素级统一分割，并构建 M3G2 数据集，显著提升多任务接地表现、降低幻觉。
摘要：
大多数多模态大型语言模型（MLLM）通过因果语言建模学习语言到对象的接地，其中被接地对象由边界框表示为位置标记序列。该范式缺乏对细粒度视觉理解和诊断十分重要的像素级表示。在本工作中，我们提出 GROUNDHOG，这是一种通过将大型语言模型接地到整体分割而开发的 MLLM。GROUNDHOG 集成了掩码特征提取器，并将提取的特征转换为供 MLLM 主干使用的视觉实体标记；随后，主干通过检索并合并实体掩码，将可接地短语连接到统一的接地掩码。为训练 GROUNDHOG，我们利用一批具有丰富标注的分割接地数据集，精心构建了 M3G2，这是一个具备多模态、多粒度接地能力的接地视觉指令调优数据集。实验结果表明，GROUNDHOG 在无需任务特定微调的情况下，在多种语言接地任务上取得优越性能，并显著减少对象幻觉。GROUNDHOG 还对复杂形式的视觉输入表现出更好的接地能力，并在失败案例中提供易于理解的诊断。
英文摘要：
Most multimodal large language models (MLLMs) learn language-to-object grounding through causal language modeling where grounded objects are captured by bounding boxes as sequences of location tokens. This paradigm lacks pixel-level representations that are important for fine-grained visual understanding and diagnosis. In this work, we introduce GROUNDHOG, an MLLM developed by grounding Large Language Models to holistic segmentation. GROUNDHOG incorporates a masked feature extractor and converts extracted features into visual entity tokens for the MLLM backbone, which then connects groundable phrases to unified grounding masks by retrieving and merging the entity masks. To train GROUNDHOG, we carefully curated M3G2, a grounded visual instruction tuning dataset with Multi-Modal Multi-Grained Grounding, by harvesting a collection of segmentation-grounded datasets with rich annotations. Our experimental results show that GROUNDHOG achieves superior performance on various language grounding tasks without task-specific fine-tuning, and significantly reduces object hallucination. GROUNDHOG also demonstrates better grounding towards complex forms of visual input and provides easy-to-understand diagnosis in failure cases.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
