Ground-V：教授视觉语言模型（VLM）在像素层面锚定复杂指令
Ground-V: Teaching VLMs to Ground Complex Instructions in Pixels
AI总结：
本研究提出自动扩展指令数据的工作流，构建像素级锚定数据集Ground-V，有效提升VLM复杂指令下的锚定能力，在多个基准上取得显著性能增益。
摘要：
本研究提出了一种简单有效的工作流，用于自动扩展指令遵循数据，以激发视觉语言模型（VLM）在复杂指令下的像素级锚定能力。我们重点解决了基于文本指令的锚定任务中五个关键的现实挑战：幻觉指代、多目标场景、推理、多粒度以及部件级指代。通过利用预训练教师模型的知识蒸馏，我们的方法生成了与现有像素级标注关联的高质量指令-响应对，最大程度减少了对昂贵人工标注的需求。由此构建的数据集Ground-V蕴含丰富的目标定位知识和精细的像素级指代表达。实验结果表明，在Ground-V上训练的模型在各类锚定任务中均取得了显著提升。具体而言，训练中加入Ground-V后，LISA模型在六个基准的gIoU指标上平均准确率提升4.4%，PSALM模型提升7.9%；该方法还在RefCOCO/+/g等标准基准上创下了新的最优结果。值得注意的是，在gRefCOCO数据集上，我们的N-Acc达到83.3%，超过此前最优水平20%以上。
英文摘要：
This work presents a simple yet effective workflow for automatically scaling instruction-following data to elicit pixel-level grounding capabilities of VLMs under complex instructions. In particular, we address five critical real-world challenges in text-instruction-based grounding: hallucinated references, multi-object scenarios, reasoning, multi-granularity, and part-level references. By leveraging knowledge distillation from a pre-trained teacher model, our approach generates high-quality instruction-response pairs linked to existing pixel-level annotations, minimizing the need for costly human annotation. The resulting dataset, Ground-V, captures rich object localization knowledge and nuanced pixel-level referring expressions. Experiment results show that models trained on Ground-V exhibit substantial improvements across diverse grounding tasks. Specifically, incorporating Ground-V during training directly achieves an average accuracy boost of 4.4% for LISA and a 7.9% for PSALM across six benchmarks on the gIoU metric. It also sets new state-of-the-art results on standard benchmarks such as RefCOCO/+/g. Notably, on gRefCOCO, we achieve an N-Acc of 83.3%, exceeding the previous state-of-the-art by more than 20%.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
