arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.02330cs.CV
收藏

RegionGPT:迈向区域理解视觉语言模型

RegionGPT: Towards Region Understanding Vision Language Model

The University of Hong Kong(香港大学);NVIDIA(英伟达)

Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun Cheung, Yizhou Yu, Ping Luo, Sifei Liu

AI总结:

本文提出 RGPT 框架,通过改进视觉编码器、任务引导提示和自动区域描述数据生成流程,增强 VLM 的区域级视觉理解与多任务性能。

摘要:

视觉语言模型(VLM)通过将大型语言模型(LLM)与图文对相结合取得了快速进展,但由于视觉编码器的空间感知能力有限,以及使用缺乏详细、区域专属描述的粗粒度训练数据,它们在细致的区域视觉理解方面仍存在困难。为解决这一问题,我们提出 RegionGPT(简称 RGPT),这是一个专为复杂区域级描述与理解设计的新颖框架。RGPT 通过对现有 VLM 中的视觉编码器进行简单而有效的修改,增强区域表征的空间感知能力。我们还通过在训练和推理阶段整合任务引导的指令提示,进一步提升模型在需要特定输出范围的任务上的性能,同时保持模型在通用任务上的多功能性。此外,我们开发了自动化的区域描述数据生成流程,用详细的区域级描述丰富训练集。我们证明,一个通用 RGPT 模型能够被有效应用于一系列区域级任务,并显著提升性能,这些任务包括但不限于复杂区域描述、推理、对象分类以及指代表达理解。

英文摘要:

Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the vision encoder, and the use of coarse-grained training data that lacks detailed, region-specific captions. To address this, we introduce RegionGPT (short as RGPT), a novel framework designed for complex region-level captioning and understanding. RGPT enhances the spatial awareness of regional representation with simple yet effective modifications to existing visual encoders in VLMs. We further improve performance on tasks requiring a specific output scope by integrating task-guided instruction prompts during both training and inference phases, while maintaining the model's versatility for general-purpose tasks. Additionally, we develop an automated region caption data generation pipeline, enriching the training set with detailed region-level captions. We demonstrate that a universal RGPT model can be effectively applied and significantly enhancing performance across a range of region-level tasks, including but not limited to complex region descriptions, reasoning, object classification, and referring expressions comprehension.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑