图标记：通过基于图的视觉提示提升多模态语言模型的空间推理能力
Graph-of-Mark: Promote Spatial Reasoning in Multimodal Language Models with Graph-Based Visual Prompting
AI总结：
本文提出Graph-of-Mark，一种基于图的视觉提示方法，通过在输入图像上叠加场景图来增强多模态语言模型的空间推理能力，实验表明其在视觉问答和定位任务中提升了11个百分点的准确率。
摘要：
近期，训练无关的视觉提示方法，如Set-of-Mark，已成为增强多模态语言模型（MLMs）接地能力的有前景方向。这些技术通过将输入图像划分为物体区域并用标记（主要为带有数字标识的框）标注后，将增强的图像输入MLM。然而，这些方法将标记对象视为孤立实体，未能捕捉它们之间的关系。在此基础上，我们提出了Graph-of-Mark（GoM），这是首个像素级视觉提示技术，通过在输入图像上叠加场景图以实现空间推理任务。我们评估了GoM在3个开源MLMs和4个不同数据集上的表现，对绘制组件进行了广泛的消融实验，并研究了文本提示中辅助图描述的影响。我们的结果表明，GoM在零样本情况下持续提高了MLMs对物体位置和相对方向的解释能力，在视觉问答和定位任务中基线准确率提高了高达11个百分点。
英文摘要：
Recent advances in training-free visual prompting, such as Set-of-Mark, have emerged as a promising direction for enhancing the grounding capabilities of multimodal language models (MLMs). These techniques operate by partitioning the input image into object regions and annotating them with marks, predominantly boxes with numeric identifiers, before feeding the augmented image to the MLM. However, these approaches treat marked objects as isolated entities, failing to capture the relationships between them. On these premises, we propose Graph-of-Mark (GoM), the first pixel-level visual prompting technique that overlays scene graphs onto the input image for spatial reasoning tasks. We evaluate GoM across 3 open-source MLMs and 4 different datasets, conducting extensive ablations on drawn components and investigating the impact of auxiliary graph descriptions in the text prompt. Our results demonstrate that GoM consistently improves the zero-shot capability of MLMs in interpreting object positions and relative directions, improving base accuracy in visual question answering and localization up to 11 percentage points.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
