具有细粒度视觉感知的多模态指令微调大语言模型
Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception
AI总结：
本文提出通用多模态大模型AnyRef，通过重聚焦机制利用LLM推理注意力分数实现细粒度像素级对象感知，在多模态指代分割等基准上取得SOTA。
摘要：
多模态大语言模型（MLLMs）利用大语言模型作为处理各种视觉-语言任务的认知框架。近期的研究致力于赋予MLLMs视觉感知和定位能力。然而，在提供细粒度的像素级感知以及将交互扩展至特定文本输入之外方面，仍存在差距。在这项工作中，我们提出了AnyRef，这是一个通用的MLLM模型，能够根据文本、边界框、图像或音频等多模态引用生成逐像素的对象感知和自然语言描述。这一创新使用户能够更灵活地与模型交互，超越了文本和区域提示的限制，且无需特定模态的设计。通过我们提出的重聚焦机制，生成的定位输出被引导以更好地聚焦于被引用的对象，隐式地引入了额外的像素级监督。这一简单的修改利用了LLM推理过程中生成的注意力分数，消除了额外计算的需求，同时在定位掩码和指代表达生成方面均展现出性能提升。仅使用公开可用的训练数据，我们的模型在多个基准测试中取得了最先进的结果，包括多样化的模态指代分割和区域级指代表达生成。
英文摘要：
Multimodal Large Language Model (MLLMs) leverages Large Language Models as a cognitive framework for diverse visual-language tasks. Recent efforts have been made to equip MLLMs with visual perceiving and grounding capabilities. However, there still remains a gap in providing fine-grained pixel-level perceptions and extending interactions beyond text-specific inputs. In this work, we propose {\bf{AnyRef}}, a general MLLM model that can generate pixel-wise object perceptions and natural language descriptions from multi-modality references, such as texts, boxes, images, or audio. This innovation empowers users with greater flexibility to engage with the model beyond textual and regional prompts, without modality-specific designs. Through our proposed refocusing mechanism, the generated grounding output is guided to better focus on the referenced object, implicitly incorporating additional pixel-level supervision. This simple modification utilizes attention scores generated during the inference of LLM, eliminating the need for extra computations while exhibiting performance enhancements in both grounding masks and referring expressions. With only publicly available training data, our model achieves state-of-the-art results across multiple benchmarks, including diverse modality referring segmentation and region-level referring expression generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
