Groma：用于多模态大语言模型定位的局部视觉标记化
Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models
AI总结：
Groma通过局部视觉标记化机制，将图像分解为区域标记并整合到指令与响应中，实现区域描述和视觉定位，在基准测试中优于依赖外部模块的MLLM。
摘要：
我们提出了Groma，一种具有定位和细粒度视觉感知能力的多模态大语言模型（MLLM）。除了整体图像理解之外，Groma还擅长区域级任务，如区域描述和视觉定位。这些能力建立在局部视觉标记化机制之上，其中图像输入被分解为感兴趣的区域，随后编码为区域标记。通过将区域标记整合到用户指令和模型响应中，我们无缝地使Groma能够理解用户指定的区域输入，并将其文本输出定位到图像上。此外，为了增强Groma的定位聊天能力，我们利用强大的GPT-4V和视觉提示技术策划了一个视觉定位指令数据集。与依赖语言模型或外部模块进行定位的MLLM相比，Groma在标准引用和定位基准测试中始终表现出更优越的性能，突显了将定位嵌入图像标记化的优势。项目页面：https://groma-mllm.github.io/。
英文摘要：
We introduce Groma, a Multimodal Large Language Model (MLLM) with grounded and fine-grained visual perception ability. Beyond holistic image understanding, Groma is adept at region-level tasks such as region captioning and visual grounding. Such capabilities are built upon a localized visual tokenization mechanism, where an image input is decomposed into regions of interest and subsequently encoded into region tokens. By integrating region tokens into user instructions and model responses, we seamlessly enable Groma to understand user-specified region inputs and ground its textual output to images. Besides, to enhance the grounded chat ability of Groma, we curate a visually grounded instruction dataset by leveraging the powerful GPT-4V and visual prompting techniques. Compared with MLLMs that rely on the language model or external module for localization, Groma consistently demonstrates superior performances in standard referring and grounding benchmarks, highlighting the advantages of embedding localization into image tokenization. Project page: https://groma-mllm.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
