从生成式视觉与语言模型学习视觉定位
Learning Visual Grounding from Generative Vision and Language Model
AI总结：
本研究利用生成式视觉语言模型通过提示生成目标级描述，结合属性与空间关系建模，构建大规模视觉定位数据集，并在RefCOCO基准上零样本迁移取得显著优于现有方法的性能。
摘要：
视觉定位任务旨在根据自然语言指称定位图像区域。在这项工作中，我们探索是否可以利用主要基于图像-文本数据训练的生成式视觉语言模型（VLM）来扩展视觉定位数据的文本标注。我们发现定位知识已存在于生成式VLM中，并可通过适当的提示来激发。因此，我们通过向VLM输入来自现有目标检测数据集的目标区域，提示其生成目标级描述。我们进一步提出属性建模以显式捕获重要的目标属性，以及空间关系建模以捕获目标间关系，这两者都是指称表达中常见的语言模式。我们构建的数据集（50万张图像、100万个目标、1600万条指称表达）是迄今为止最大的定位数据集之一，也是首个具有纯模型生成查询和人工标注目标的定位数据集。为验证该数据的质量，我们在流行的RefCOCO基准上进行了零样本迁移实验，涵盖指称表达理解（REC）和分割（RES）任务。在这两项任务上，我们的模型均显著优于最先进的方法，且未使用人工标注的视觉定位数据。我们的结果证明了生成式VLM在现实世界中扩展视觉定位的潜力。代码和模型将发布。
英文摘要：
Visual grounding tasks aim to localize image regions based on natural language references. In this work, we explore whether generative VLMs predominantly trained on image-text data could be leveraged to scale up the text annotation of visual grounding data. We find that grounding knowledge already exists in generative VLM and can be elicited by proper prompting. We thus prompt a VLM to generate object-level descriptions by feeding it object regions from existing object detection datasets. We further propose attribute modeling to explicitly capture the important object attributes, and spatial relation modeling to capture inter-object relationship, both of which are common linguistic pattern in referring expression. Our constructed dataset (500K images, 1M objects, 16M referring expressions) is one of the largest grounding datasets to date, and the first grounding dataset with purely model-generated queries and human-annotated objects. To verify the quality of this data, we conduct zero-shot transfer experiments to the popular RefCOCO benchmarks for both referring expression comprehension (REC) and segmentation (RES) tasks. On both tasks, our model significantly outperform the state-of-the-art approaches without using human annotated visual grounding data. Our results demonstrate the promise of generative VLM to scale up visual grounding in the real world. Code and models will be released.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
