大型视觉-语言模型中语义接地的评估与增强
Evaluation and Enhancement of Semantic Grounding in Large Vision-Language Models
AI总结：
本文针对大型视觉-语言模型语义接地能力不足的问题，构建大规模细粒度评估数据集并系统评估七种主流模型，进而提出基于多模态指令微调的数据中心增强方法，有效减少错误接地现象。
摘要：
大型视觉-语言模型（LVLMs）在多种视觉-语言任务中展现出显著优势。然而，阻碍其在现实场景中应用——尤其是在安全性、鲁棒性和可靠性方面——的一个挑战是其受限的语义接地能力，该能力涉及将语言与图像中所指涉的物理世界实体或概念相连接。因此，亟需开展一项全面研究，以评估广泛使用的LVLMs的语义接地能力。尽管该问题具有重要意义，目前这一方向上的充分研究仍然缺乏。我们的工作通过设计一个用于生成大规模评估数据集的流程来填补这一空白，该流程覆盖细粒度语义信息，如颜色、数量、材料等，并对七种主流LVLMs的语义接地能力进行了全面评估。结果凸显了在不同方面和程度上普遍存在的错误接地现象。为解决这一问题，我们提出了一种以数据为中心的增强方法，旨在通过对细粒度对话进行多模态指令微调来提升LVLMs的语义接地能力。在增强后的LVLMs上进行的实验表明，在解决错误接地问题方面取得了显著改进。
英文摘要：
Large Vision-Language Models (LVLMs) offer remarkable benefits for a variety of vision-language tasks. However, a challenge hindering their application in real-world scenarios, particularly regarding safety, robustness, and reliability, is their constrained semantic grounding ability, which pertains to connecting language to the physical-world entities or concepts referenced in images. Therefore, a crucial need arises for a comprehensive study to assess the semantic grounding ability of widely used LVLMs. Despite the significance, sufficient investigation in this direction is currently lacking. Our work bridges this gap by designing a pipeline for generating large-scale evaluation datasets covering fine-grained semantic information, such as color, number, material, etc., along with a thorough assessment of seven popular LVLMs' semantic grounding ability. Results highlight prevalent misgrounding across various aspects and degrees. To address this issue, we propose a data-centric enhancement method that aims to improve LVLMs' semantic grounding ability through multimodal instruction tuning on fine-grained conversations. Experiments on enhanced LVLMs demonstrate notable improvements in addressing misgrounding issues.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
