搭建坐标以促进大型多模态模型中的视觉-语言协调
Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models
AI总结：
提出Scaffold prompting方法，通过在图像中叠加点矩阵和多维坐标，促进大型多模态模型中的视觉-语言协调，实验证明其优于GPT-4V的文本CoT提示。
摘要：
最先进的大型多模态模型（LMMs）在视觉语言任务中展现了卓越的能力。尽管具备高级功能，但在需要多层级视觉信息复杂推理的挑战性场景中，LMMs的性能仍受限。现有的LMM提示技术要么侧重于改进文本推理，要么利用工具进行图像预处理，缺乏一种简单通用的视觉提示方案来促进LMMs中的视觉-语言协调。在本研究中，我们提出了Scaffold prompting，通过搭建坐标来促进视觉-语言协调。具体而言，Scaffold在图像内叠加点矩阵作为视觉信息锚点，并利用多维坐标作为文本位置参考。在广泛的视觉语言任务上的大量实验表明，Scaffold优于使用文本CoT提示的GPT-4V。代码发布于https://github.com/leixy20/Scaffold。
英文摘要：
State-of-the-art Large Multi-Modal Models (LMMs) have demonstrated exceptional capabilities in vision-language tasks. Despite their advanced functionalities, the performances of LMMs are still limited in challenging scenarios that require complex reasoning with multiple levels of visual information. Existing prompting techniques for LMMs focus on either improving textual reasoning or leveraging tools for image preprocessing, lacking a simple and general visual prompting scheme to promote vision-language coordination in LMMs. In this work, we propose Scaffold prompting that scaffolds coordinates to promote vision-language coordination. Specifically, Scaffold overlays a dot matrix within the image as visual information anchors and leverages multi-dimensional coordinates as textual positional references. Extensive experiments on a wide range of challenging vision-language tasks demonstrate the superiority of Scaffold over GPT-4V with the textual CoT prompting. Our code is released in https://github.com/leixy20/Scaffold.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
