通过引导视觉令牌注意力实现直接视觉 grounding
Direct Visual Grounding by Directing Attention of Visual Tokens
AI总结：
该研究针对VLMs中视觉令牌注意力不足的问题，提出KL注意力损失，结合下一个令牌预测损失，在多个视觉任务上取得显著提升，还引入了用于评估VLMs线条追踪能力的新数据集。
摘要：
视觉语言模型（Vision Language Models, VLMs）会混合视觉令牌与文本令牌。一个令人困惑的问题是，在VLMs的LLM模块的最终层中，与查询最相关的视觉令牌几乎没有或根本没有得到答案令牌的关注，在该层中所有令牌（尤其是LLM注意力层中的视觉和语言令牌）都被同等对待。我们的实验结果证实，这一事实可能会导致视觉问题的答案出错。标准的下一个令牌预测（next-token prediction, NTP）损失似乎无法为引导对视觉令牌的注意力提供足够的信号。我们假设，对VLMs的LLM模块中视觉令牌对相应语言令牌的注意力进行更直接的监督，将提升视觉任务的性能。为了证明这确实可行，我们提出了一种新的损失函数，直接监督视觉令牌的注意力。该函数通过引导答案语言令牌关注相关视觉令牌，直接将其与图像中的视觉令牌进行grounding。这一过程通过KL散度将视觉令牌的注意力分布与真实注意力图对齐来实现。真实注意力图从合成案例的任务几何中获取，或从真实图像的标准grounding标注（如边界框或点标注）中获取，用于LLM内部的注意力监督，无需新的标签。将所得的KL注意力损失（KL attention loss, KLAL）与NTP结合，可鼓励VLMs在生成答案令牌时关注相关视觉令牌。实验表明，这在几何任务、指向任务以及合成和真实数据上的指代表达理解任务中均取得了显著提升。我们还引入了一个新的数据集，用于评估VLMs的线条追踪能力，令人惊讶的是，即使是商用VLMs在该任务上的表现也不佳。
英文摘要：
Vision Language Models (VLMs) mix visual tokens and text tokens. A puzzling issue is the fact that visual tokens most related to the query receive little to no attention in the final layers of the LLM module of VLMs from the answer tokens, where all tokens are treated equally, in particular, visual and language tokens in the LLM attention layers. This fact may result in wrong answers to visual questions, as our experimental results confirm. It appears that the standard next-token prediction (NTP) loss provides an insufficient signal for directing attention to visual tokens. We hypothesize that a more direct supervision of the attention of visual tokens to corresponding language tokens in the LLM module of VLMs will lead to improved performance on visual tasks. To demonstrate that this is indeed the case, we propose a novel loss function that directly supervises the attention of visual tokens. It directly grounds the answer language tokens in images by directing their attention to the relevant visual tokens. This is achieved by aligning the attention distribution of visual tokens to ground truth attention maps with KL divergence. The ground truth attention maps are obtained from task geometry in synthetic cases or from standard grounding annotations (e.g., bounding boxes or point annotations) in real images, and are used inside the LLM for attention supervision without requiring new labels. The obtained KL attention loss (KLAL) when combined with NTP encourages VLMs to attend to relevant visual tokens while generating answer tokens. This results in notable improvements across geometric tasks, pointing, and referring expression comprehension on both synthetic and real-world data, as demonstrated by our experiments. We also introduce a new dataset to evaluate the line tracing abilities of VLMs. Surprisingly, even commercial VLMs do not perform well on this task.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
