arXiv 2410.161632024-10-21cs.CV

Griffon-G：通过大型多模态模型桥接视觉-语言与视觉中心任务

Griffon-G: Bridging Vision-Language and Vision-Centric Tasks via Large Multimodal Models

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peng Cheng Laboratory（鹏城实验室）；Wuhan AI Research（武汉人工智能研究院）

Yufei Zhan, Hongyin Zhao, Yousong Zhu, Fan Yang, Ming Tang, Jinqiao Wang

备注：: This work has been submitted to the IEEE for possible publication. Codes and data will be later released at https://github.com/jefferyZhan/Griffon

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出通用大型多模态模型 Griffon-G 及多模态数据集 CCMD-8M，通过多层级数据筛选与多任务整合，在单一端到端范式中统一视觉中心与视觉-语言任务，解决联合优化中的训练崩溃问题，并在多项基准上取得专家级性能。

摘要：

大型多模态模型（LMMs）基于自回归建模，已在多种视觉-语言和视觉中心任务中取得重大突破。然而，这些模型通常只专注于视觉中心任务（如视觉定位和区域描述）或视觉-语言任务（如图像描述和多场景视觉问答），尚无任何一个 LMM 能在单一模型中全面统一这两类任务，正如自然语言处理领域中的大型语言模型所做的那样。此外，即使拥有丰富的多任务指令跟随数据，直接堆叠这些数据以实现通用能力扩展仍然具有挑战性。为解决这些问题，我们引入了一个新颖的多维度筛选与整合的多模态数据集，名为 CCMD-8M，通过多层次数据筛选和多任务整合，克服了统一视觉中心与视觉-语言任务的数据障碍。更重要的是，我们提出了 Griffon-G，这是一个通用的大型多模态模型，在单一端到端范式中同时处理视觉中心和视觉-语言任务。Griffon-G 解决了这些任务联合优化过程中遇到的训练崩溃问题，实现了更好的训练效率。在跨多模态基准、通用视觉问答（VQA）任务、场景文本中心 VQA 任务、文档相关 VQA 任务、指代表达理解以及目标检测上的评估表明，Griffon-G 超越了先进的大型多模态模型，并在复杂的视觉中心任务中达到了专家级性能。

英文摘要：

Large Multimodal Models (LMMs) have achieved significant breakthroughs in various vision-language and vision-centric tasks based on auto-regressive modeling. However, these models typically focus on either vision-centric tasks, such as visual grounding and region description, or vision-language tasks, like image caption and multi-scenario VQAs. None of the LMMs have yet comprehensively unified both types of tasks within a single model, as seen in Large Language Models in the natural language processing field. Furthermore, even with abundant multi-task instruction-following data, directly stacking these data for universal capabilities extension remains challenging. To address these issues, we introduce a novel multi-dimension curated and consolidated multimodal dataset, named CCMD-8M, which overcomes the data barriers of unifying vision-centric and vision-language tasks through multi-level data curation and multi-task consolidation. More importantly, we present Griffon-G, a general large multimodal model that addresses both vision-centric and vision-language tasks within a single end-to-end paradigm. Griffon-G resolves the training collapse issue encountered during the joint optimization of these tasks, achieving better training efficiency. Evaluations across multimodal benchmarks, general Visual Question Answering (VQA) tasks, scene text-centric VQA tasks, document-related VQA tasks, Referring Expression Comprehension, and object detection demonstrate that Griffon-G surpasses the advanced LMMs and achieves expert-level performance in complicated vision-centric tasks.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。