大型多模态模型中的涌现视觉定位能力，无需定位监督
Emergent Visual Grounding in Large Multimodal Models Without Grounding Supervision
AI总结：
本文发现无需定位监督，大型多模态模型可涌现视觉定位能力，提出注意-分割方法和DIFFLMM模型，在多个基准上取得有竞争力性能，并在接地对话生成上超越强监督模型GLaMM。
摘要：
当前的大型多模态模型（LMMs）在定位任务上面临挑战，该任务要求模型将语言成分与视觉实体相关联。与常见做法（即使用额外的定位监督对LMMs进行微调）相反，我们发现定位能力实际上可以在没有显式定位监督训练的情况下涌现于LMMs中。为了揭示这种涌现的定位能力，我们引入了一种“注意-分割”方法，该方法利用标准LMMs的注意力图来执行像素级分割。此外，为了增强定位能力，我们提出了DIFFLMM，一种利用基于扩散的视觉编码器（而非标准的CLIP视觉编码器）的LMM，并使用相同的弱监督进行训练。由于不受定位特定监督数据的偏见和有限规模的约束，我们的方法更具泛化性和可扩展性。我们在定位特定基准和通用视觉问答基准上分别与定位LMMs和通用LMMs相比，取得了有竞争力的性能。值得注意的是，我们在没有任何定位监督的情况下，在接地对话生成任务上实现了44.2的定位掩码召回率，超过了经过大量监督的模型GLaMM。项目页面：https://GroundLMM-ICCV.github.io。
英文摘要：
Current large multimodal models (LMMs) face challenges in grounding, which requires the model to relate language components to visual entities. Contrary to the common practice that fine-tunes LMMs with additional grounding supervision, we find that the grounding ability can in fact emerge in LMMs trained without explicit grounding supervision. To reveal this emerging grounding, we introduce an "attend-and-segment" method which leverages attention maps from standard LMMs to perform pixel-level segmentation. Furthermore, to enhance the grounding ability, we propose DIFFLMM, an LMM utilizing a diffusion-based visual encoder, as opposed to the standard CLIP visual encoder, and trained with the same weak supervision. Without being constrained by the biases and limited scale of grounding-specific supervision data, our approach is more generalizable and scalable. We achieve competitive performance on both grounding-specific and general visual question answering benchmarks, compared with grounding LMMs and generalist LMMs, respectively. Notably, we achieve a 44.2 grounding mask recall on grounded conversation generation without any grounding supervision, outperforming the extensively supervised model GLaMM. Project page: https://GroundLMM-ICCV.github.io.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
