arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2311.14552cs.CVcs.AI
收藏

Griffon:利用大语言模型明确输出任意粒度下所有物体的位置

Griffon: Spelling out All Object Locations at Any Granularity with Large Language Models

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所基础模型研究中心);School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院);Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)

Yufei Zhan, Yousong Zhu, Zhiyang Chen, Fan Yang, Ming Tang, Jinqiao Wang

AI总结:

本研究针对大视觉语言模型定位能力受限的问题,挖掘其基础物体感知潜力,提出语言提示定位数据集与纯LVLM基线模型Griffon,在多个定位、检测基准上取得优异性能。

摘要:

对大视觉语言模型(LVLM)而言,复刻人类凭借自由形式文本、在任意粒度下检测所有物体的先天能力,仍是一项艰巨挑战。当前LVLM大多受限于只能定位单个已存在的物体,这一局限导致模型设计存在折衷,需要引入视觉专家模型或定制化头部结构。\n突破这些限制的过程中,我们的研究发现LVLM具备基础物体感知能力,能够准确识别并定位感兴趣的物体。基于这一洞见,我们提出了一种新型语言提示定位数据集(Language-prompted Localization Dataset),以充分释放LVLM在细粒度物体感知和精准位置感知方面的能力。\n更重要的是,我们提出了Griffon——一个纯LVLM基线模型,它未引入任何特殊token、专家模型或额外检测模块,仅通过统一各类定位相关场景的数据格式,保持与主流LVLM一致的结构,并通过精心设计的流水线进行端到端训练。\n综合实验表明,Griffon不仅在细粒度RefCOCO系列和Flickr30K Entities上取得了当前最优性能,在检测基准MSCOCO上的表现也接近专家模型Faster R-CNN的能力。数据集、代码与模型已发布于https://github.com/jefferyZhan/Griffon。

英文摘要:

Replicating the innate human ability to detect all objects based on free-form texts at any granularity remains a formidable challenge for Large Vision Language Models (LVLMs). Current LVLMs are predominantly constrained to locate a single, pre-existing object. This limitation leads to a compromise in model design, necessitating the introduction of visual expert models or customized head structures. Beyond these constraints, our research uncovers LVLMs' capability for basic object perception, allowing them to accurately identify and locate objects of interest. Building on this insight, we introduce a novel Language-prompted Localization Dataset to fully unleash the capabilities of LVLMs in fine-grained object perception and precise location awareness. More importantly, we present Griffon, a purely LVLM-based baseline, which does not introduce any special tokens, expert models, or additional detection modules. It simply maintains a consistent structure with popular LVLMs by unifying data formats across various localization-related scenarios and is trained end-to-end through a well-designed pipeline. Comprehensive experiments demonstrate that Griffon not only achieves state-of-the-art performance on the fine-grained RefCOCO series and Flickr30K Entities but also approaches the capabilities of the expert model Faster RCNN on the detection benchmark MSCOCO. Data, codes, and models are released at https://github.com/jefferyZhan/Griffon.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑