基于令牌的多模态大语言模型的 grounding
Grounding Everything in Tokens for Multimodal Large Language Models
AI总结：
本文提出GETok方法，通过整合可学习的令牌词汇提升多模态大语言模型在2D空间中的物体定位能力，实验显示其在多种指引用例任务中表现更优。
摘要：
多模态大语言模型（MLLMs）在视觉理解和推理方面取得了显著进展。然而，MLLMs所使用的自回归Transformer架构需要对输入图像进行分词，这限制了其在2D图像空间中准确地将对象接地的能力。为此，本文提出了一种空间表示方法GETok，通过整合可学习的令牌词汇到MLLMs中，利用网格令牌将图像平面划分为结构化的空间锚点，并通过偏移令牌实现局部化预测的精确迭代优化。通过直接将空间关系嵌入令牌中，GETok显著提升了MLLMs在原生2D空间推理中的能力，而无需修改自回归架构。大量实验表明，GETok在监督微调和强化学习设置下的各种指引用例任务中均优于现有最先进方法。
英文摘要：
Multimodal large language models (MLLMs) have made significant advancements in vision understanding and reasoning. However, the autoregressive Transformer architecture used by MLLMs requries tokenization on input images, which limits their ability to accurately ground objects within the 2D image space. This raises an important question: how can sequential language tokens be improved to better ground objects in 2D spatial space for MLLMs? To address this, we present a spatial representation method for grounding objects, namely GETok, that integrates a specialized vocabulary of learnable tokens into MLLMs. GETok first uses grid tokens to partition the image plane into structured spatial anchors, and then exploits offset tokens to enable precise and iterative refinement of localization predictions. By embedding spatial relationships directly into tokens, GETok significantly advances MLLMs in native 2D space reasoning without modifying the autoregressive architecture. Extensive experiments demonstrate that GETok achieves superior performance over the state-of-the-art methods across various referring tasks in both supervised fine-tuning and reinforcement learning settings.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
