LLaVA-Grounding：基于大型多模态模型的接地视觉聊天
LLaVA-Grounding: Grounded Visual Chat with Large Multimodal Models
AI总结：
本文提出LLaVA-Grounding，通过创建接地视觉聊天数据集和Grounding-Bench基准，并连接分割模型与语言模型，使大型多模态模型同时具备接地与聊天能力，在多个基准上取得领先或竞争性表现。
摘要：
随着大型多模态模型（LMMs）近期的显著进展，其在视觉聊天中的接地能力的重要性日益受到认可。尽管近期已有工作尝试使LMMs支持接地，但其接地能力与聊天能力通常是分离的，且当被要求进行接地时，聊天性能会急剧下降。问题在于缺乏用于接地视觉聊天（GVC）的数据集。现有的接地数据集仅包含简短描述。为解决这一问题，我们创建了GVC数据，使得接地能力与聊天能力能够结合。为了更好地评估GVC能力，我们引入了一个名为Grounding-Bench的基准。此外，我们提出了一种模型设计，通过将分割模型与语言模型连接，能够支持GVC以及多种类型的视觉提示。实验结果表明，我们的模型在Grounding-Bench上优于其他LMMs。此外，我们的模型在经典接地基准如RefCOCO/+/g和Flickr30K Entities上取得了具有竞争力的性能。我们的代码将在https://github.com/UX-Decoder/LLaVA-Grounding发布。
英文摘要：
With the recent significant advancements in large multi-modal models (LMMs), the importance of their grounding capability in visual chat is increasingly recognized. Despite recent efforts to enable LMMs to support grounding, their capabilities for grounding and chat are usually separate, and their chat performance drops dramatically when asked to ground. The problem is the lack of a dataset for grounded visual chat (GVC). Existing grounding datasets only contain short captions. To address this issue, we have created GVC data that allows for the combination of grounding and chat capabilities. To better evaluate the GVC capabilities, we have introduced a benchmark called Grounding-Bench. Additionally, we have proposed a model design that can support GVC and various types of visual prompts by connecting segmentation models with language models. Experimental results demonstrate that our model outperforms other LMMs on Grounding-Bench. Furthermore, our model achieves competitive performance on classic grounding benchmarks like RefCOCO/+/g and Flickr30K Entities. Our code will be released at https://github.com/UX-Decoder/LLaVA-Grounding .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
