arXiv 2403.093332025-08-11cs.CVcs.AI

Griffon v2：通过高分辨率缩放与视觉-语言共指代推进多模态感知

Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring

Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所基础模型研究中心）；School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Peng Cheng Laboratory, Shenzhen, China（鹏城实验室）；Wuhan AI Research, Wuhan, China（武汉人工智能研究所）

Yufei Zhan, Shurong Zheng, Yousong Zhu, Hongyin Zhao, Fan Yang, Ming Tang, Jinqiao Wang

备注：: Accepted by ICCV 2025. Codes and datasets are released at https://github.com/jefferyZhan/Griffon

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

提出统一高分辨率通用模型 Griffon v2，通过轻量下采样投影器和即插即用视觉分词器实现视觉-语言共指代，在多项视觉感知任务上达到 SOTA 并超越专家模型。

摘要：

大型视觉语言模型已实现细粒度目标感知，但图像分辨率的限制仍是超越复杂密集场景中任务特定专家性能的重大障碍。这种限制进一步制约了模型在 GUI Agents、计数等领域实现细微视觉与语言指代的潜力。为解决此问题，我们引入统一的高分辨率通用模型 Griffon v2，支持通过视觉和文本提示进行灵活目标指代。为高效提升图像分辨率，我们设计了简单轻量的下采样投影器，以克服大型语言模型中的输入 token 约束。该设计固有地保留完整上下文与精细细节，显著提升多模态感知能力，尤其是对小目标。在此基础上，我们通过即插即用的视觉分词器进一步赋予模型视觉-语言共指代能力。它支持用户友好的交互，可处理灵活的目标图像、自由格式文本甚至坐标。实验表明，Griffon v2 能通过视觉与文本指代定位感兴趣目标，在 REC 和短语定位上达到 SOTA 性能，并在目标检测、目标计数和 REG 上超越专家模型。数据和代码发布于 https://github.com/jefferyZhan/Griffon。

英文摘要：

Large Vision Language Models have achieved fine-grained object perception, but the limitation of image resolution remains a significant obstacle to surpassing the performance of task-specific experts in complex and dense scenarios. Such limitation further restricts the model's potential to achieve nuanced visual and language referring in domains such as GUI Agents, counting, \textit{etc}. To address this issue, we introduce a unified high-resolution generalist model, Griffon v2, enabling flexible object referring with visual and textual prompts. To efficiently scale up image resolution, we design a simple and lightweight down-sampling projector to overcome the input tokens constraint in Large Language Models. This design inherently preserves the complete contexts and fine details and significantly improves multimodal perception ability, especially for small objects. Building upon this, we further equip the model with visual-language co-referring capabilities through a plug-and-play visual tokenizer. It enables user-friendly interaction with flexible target images, free-form texts, and even coordinates. Experiments demonstrate that Griffon v2 can localize objects of interest with visual and textual referring, achieve state-of-the-art performance on REC and phrase grounding, and outperform expert models in object detection, object counting, and REG. Data and codes are released at https://github.com/jefferyZhan/Griffon.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。