EAGLE：面向多模态大语言模型的高效任意指代视觉提示理解
EAGLE: Towards Efficient Arbitrary Referring Visual Prompts Comprehension for Multimodal Large Language Models
AI总结：
针对多模态大语言模型在理解任意指代视觉提示时存在的架构冗余和泛化难题，提出EAGLE模型，通过将提示作为彩色色块渲染并引入几何无关学习范式，以更少训练量实现了高效理解。
摘要：
近期，多模态大语言模型（MLLMs）因其卓越的内容推理和指令遵循能力引发了极大的研究兴趣。为了有效地向MLLM下达指令，除了传统的语言表达外，在图像上用画笔涂抹以指代物体的做法已成为一种流行工具（被称为“指代视觉提示”），因为它能有效地将用户意图与特定图像区域对齐。为了适应最常见的指代视觉提示，即点、框和掩码，现有方法首先利用专门的特性编码模块来捕获这些提示所指示的高亮区域的语义。随后，这些编码后的区域特征通过在精心构建的多模态指令数据集上进行微调，以适配MLLMs。然而，此类设计存在架构冗余问题。此外，在现实场景中遇到各种任意指代视觉提示时，它们在有效泛化方面面临挑战。为了解决上述问题，我们提出了EAGLE，一种新型MLLM，它以比现有方法更少的训练量实现了对任意指代视觉提示的理解。具体而言，我们的EAGLE保持了指代视觉提示的固有格式，将其作为在给定图像上渲染的彩色色块来进行指令微调。我们的方法将指代视觉提示嵌入为传达特定空间区域的空间概念，使MLLM能够理解这些区域，而这些区域的语义理解则源自MLLM本身。此外，我们还提出了一种几何无关学习范式（GAL），以进一步解耦MLLM的区域级理解与指代视觉提示的具体格式。我们进行了大量实验以证明所提方法的有效性。
英文摘要：
Recently, Multimodal Large Language Models (MLLMs) have sparked great research interests owing to their exceptional content-reasoning and instruction-following capabilities. To effectively instruct an MLLM, in addition to conventional language expressions, the practice of referring to objects by painting with brushes on images has emerged as a prevalent tool (referred to as "referring visual prompts") due to its efficacy in aligning the user's intention with specific image regions. To accommodate the most common referring visual prompts, namely points, boxes, and masks, existing approaches initially utilize specialized feature encoding modules to capture the semantics of the highlighted areas indicated by these prompts. Subsequently, these encoded region features are adapted to MLLMs through fine-tuning on a meticulously curated multimodal instruction dataset. However, such designs suffer from redundancy in architecture. Moreover, they face challenges in effectively generalizing when encountering a diverse range of arbitrary referring visual prompts in real-life scenarios. To address the above issues, we propose EAGLE, a novel MLLM that empowers comprehension of arbitrary referring visual prompts with less training efforts than existing approaches. Specifically, our EAGLE maintains the innate format of the referring visual prompts as colored patches rendered on the given image for conducting the instruction tuning. Our approach embeds referring visual prompts as spatial concepts conveying specific spatial areas comprehensible to the MLLM, with the semantic comprehension of these regions originating from the MLLM itself. Besides, we also propose a Geometry-Agnostic Learning paradigm (GAL) to further disentangle the MLLM's region-level comprehension with the specific formats of referring visual prompts. Extensive experiments are conducted to prove the effectiveness of our proposed method.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
