GLaMM：像素级接地大型多模态模型
GLaMM: Pixel Grounding Large Multimodal Model
AI总结：
GLaMM是首个能生成自然语言响应并与分割掩码无缝交织的大型多模态模型，支持多粒度交互，并引入GCG任务和GranD数据集，在多项下游任务中表现优异。
摘要：
大型多模态模型（LMMs）将大型语言模型扩展到视觉领域。最初的LMMs使用整体图像和文本提示来生成未接地的文本响应。最近，区域级LMMs已被用于生成视觉接地的响应。然而，它们仅限于一次只引用单一对象类别，需要用户指定区域，或者无法提供密集的像素级对象接地。在这项工作中，我们提出了接地LMM（GLaMM），这是第一个能够生成自然语言响应并与相应对象分割掩码无缝交织的模型。GLaMM不仅接地对话中出现的对象，而且足够灵活以接受文本和可选的视觉提示（感兴趣区域）作为输入。这使用户能够在文本和视觉域中以各种粒度级别与模型交互。由于缺乏针对视觉接地对话生成（GCG）这一新颖设置的标准基准，我们引入了一个综合评估协议，并附有我们精心策划的接地对话。我们提出的GCG任务要求在大规模自然场景中密集接地概念。为此，我们提出了一个密集标注的接地任何数据集（GranD），使用我们提出的自动化标注管道，该数据集包含7.5M个独特概念，这些概念接地于总共810M个区域，并附有分割掩码。除了GCG，GLaMM还在多个下游任务上表现有效，例如，引用表达式分割、图像和区域级描述以及视觉-语言对话。
英文摘要：
Large Multimodal Models (LMMs) extend Large Language Models to the vision domain. Initial LMMs used holistic images and text prompts to generate ungrounded textual responses. Recently, region-level LMMs have been used to generate visually grounded responses. However, they are limited to only referring to a single object category at a time, require users to specify the regions, or cannot offer dense pixel-wise object grounding. In this work, we present Grounding LMM (GLaMM), the first model that can generate natural language responses seamlessly intertwined with corresponding object segmentation masks. GLaMM not only grounds objects appearing in the conversations but is flexible enough to accept both textual and optional visual prompts (region of interest) as input. This empowers users to interact with the model at various levels of granularity, both in textual and visual domains. Due to the lack of standard benchmarks for the novel setting of visually Grounded Conversation Generation (GCG), we introduce a comprehensive evaluation protocol with our curated grounded conversations. Our proposed GCG task requires densely grounded concepts in natural scenes at a large-scale. To this end, we propose a densely annotated Grounding-anything Dataset (GranD) using our proposed automated annotation pipeline that encompasses 7.5M unique concepts grounded in a total of 810M regions available with segmentation masks. Besides GCG, GLaMM also performs effectively on several downstream tasks, e.g., referring expression segmentation, image and region-level captioning and vision-language conversations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
