基于MLLM的视觉定位的视觉位置提示
Visual Position Prompt for MLLM based Visual Grounding
AI总结：
针对MLLMs在视觉定位中坐标对齐能力弱的问题，提出结合全局与局部视觉位置提示的VPP-LLaVA模型，并构建0.6M样本数据集VPP-SFT进行高效训练，在标准基准上取得SOTA且具备强零样本泛化能力。
摘要：
尽管多模态大语言模型（MLLMs）在各种图像相关任务中表现出色，但它们在精确对齐坐标与图像内空间信息方面遇到挑战，特别是在视觉定位等位置感知任务中。这一局限源于两个关键因素。首先，MLLMs缺乏显式的空间参考，难以将文本描述与精确的图像位置关联。其次，其特征提取过程优先考虑全局上下文而非细粒度的空间细节，导致定位能力较弱。为解决这些问题，我们引入了VPP-LLaVA，一种通过视觉位置提示（VPP）增强以提升其定位能力的MLLM。VPP-LLaVA集成了两种互补机制：全局VPP将一个可学习的、类似坐标轴的张量叠加到输入图像上以提供结构化空间线索，而局部VPP引入位置感知查询以支持细粒度定位。为利用空间指导有效训练模型，我们进一步引入了VPP-SFT，一个包含0.6M高质量视觉定位样本的精选数据集。它采用紧凑格式设计，能够实现高效训练，且显著小于其他MLLM使用的数据集（例如MiniGPT-v2中的约21M样本），但依然提供了强大的性能提升。最终模型VPP-LLaVA不仅在标准视觉定位基准上取得了最先进的结果，还对具有挑战性的未见数据集展现出强大的零样本泛化能力。代码和数据集可在https://github.com/WayneTomas/VPP-LLaVA获取。
英文摘要：
Although Multimodal Large Language Models (MLLMs) excel at various image-related tasks, they encounter challenges in precisely aligning coordinates with spatial information within images, particularly in position-aware tasks such as visual grounding. This limitation arises from two key factors. First, MLLMs lack explicit spatial references, making it difficult to associate textual descriptions with precise image locations. Second, their feature extraction processes prioritize global context over fine-grained spatial details, leading to weak localization capability. To address these issues, we introduce VPP-LLaVA, an MLLM enhanced with Visual Position Prompt (VPP) to improve its grounding capability. VPP-LLaVA integrates two complementary mechanisms: the global VPP overlays a learnable, axis-like tensor onto the input image to provide structured spatial cues, while the local VPP incorporates position-aware queries to support fine-grained localization.To effectively train our model with spatial guidance, we further introduce VPP-SFT, a curated dataset of 0.6M high-quality visual grounding samples. Designed in a compact format, it enables efficient training and is significantly smaller than datasets used by other MLLMs (e.g., ~21M samples in MiniGPT-v2), yet still provides a strong performance boost. The resulting model, VPP-LLaVA, not only achieves state-of-the-art results on standard visual grounding benchmarks but also demonstrates strong zero-shot generalization to challenging unseen datasets. The code and dataset are available at https://github.com/WayneTomas/VPP-LLaVA.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
