逐项列出：多模态大语言模型的新数据源与学习范式
List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs
AI总结：
本文提出“逐项列出”学习范式，通过让模型按字母数字顺序枚举图像上的视觉标签，结合新数据集微调，显著提升多模态大语言模型的视觉推理能力并减少幻觉，即使推理时移除标签仍有效。
摘要：
Set-of-Mark（SoM）提示通过使模型能够将视觉对象与插入图像上的标签相关联，释放了GPT-4V的视觉定位能力。这些用字母数字标记的标签可以通过文本令牌进行索引，便于引用。尽管GPT-4V表现出非凡的性能，我们观察到其他多模态大语言模型（MLLMs）难以理解这些视觉标签。为了促进开源模型对SoM提示的学习，我们提出了一种新的学习范式：“逐项列出”，该范式要求模型按照标签的字母数字顺序枚举并描述图像上的所有视觉标签。通过将我们精心策划的数据集与其他视觉指令调整数据集相结合，我们能够使现有的MLLMs具备SoM提示能力。此外，我们在五个MLLM基准上评估了我们微调的SoM模型。我们发现，这个新数据集，即使规模相对较小（10k-30k张带标签的图像），也能显著增强MLLMs的视觉推理能力并减少幻觉。也许令人惊讶的是，即使在推理过程中从输入图像中省略视觉标签，这些改进仍然存在。这表明“逐项列出”作为训练MLLMs的新范式的潜力，通过在训练阶段使用视觉标签来加强对象-文本对齐。最后，我们通过探测训练模型来分析SoM的工作机制。我们的代码和数据可在\url{https://github.com/zzxslp/SoM-LLaVA}获取。
英文摘要：
Set-of-Mark (SoM) Prompting unleashes the visual grounding capability of GPT-4V, by enabling the model to associate visual objects with tags inserted on the image. These tags, marked with alphanumerics, can be indexed via text tokens for easy reference. Despite the extraordinary performance from GPT-4V, we observe that other Multimodal Large Language Models (MLLMs) struggle to understand these visual tags. To promote the learning of SoM prompting for open-source models, we propose a new learning paradigm: "list items one by one," which asks the model to enumerate and describe all visual tags placed on the image following the alphanumeric orders of tags. By integrating our curated dataset with other visual instruction tuning datasets, we are able to equip existing MLLMs with the SoM prompting ability. Furthermore, we evaluate our finetuned SoM models on five MLLM benchmarks. We find that this new dataset, even in a relatively small size (10k-30k images with tags), significantly enhances visual reasoning capabilities and reduces hallucinations for MLLMs. Perhaps surprisingly, these improvements persist even when the visual tags are omitted from input images during inference. This suggests the potential of "list items one by one" as a new paradigm for training MLLMs, which strengthens the object-text alignment through the use of visual tags in the training stage. Finally, we conduct analyses by probing trained models to understand the working mechanism of SoM. Our code and data are available at \url{https://github.com/zzxslp/SoM-LLaVA}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
