多模态参考视觉定位
Multimodal Reference Visual Grounding
AI总结：
针对视觉定位中相似物体难以区分的问题，提出多模态参考视觉定位任务及MRVG-Net方法，利用参考图像和大型语言模型实现优于Qwen2.5-VL-72B的定位性能。
摘要：
视觉定位聚焦于根据语言表达从图像中检测物体。近期的大型视觉语言模型（LVLMs）通过使用大规模数据集训练大型模型，显著提升了视觉定位的性能。然而，当输入图像中出现相似物体时，该问题仍然具有挑战性。例如，一个LVLM可能无法区分图像中的健怡可乐和普通可乐。在这种情况下，如果能够提供健怡可乐和普通可乐的额外参考图像，将有助于相似物体的视觉定位。\n在本工作中，我们引入了一个名为多模态参考视觉定位（MRVG）的新任务。在该任务中，模型可以访问数据库中物体的参考图像集。基于这些参考图像和语言表达，模型需要从查询图像中检测出目标物体。我们首先引入了一个新数据集来研究MRVG问题。然后，我们提出了一种名为MRVG-Net的新方法来解决这一视觉定位问题。我们表明，通过利用少样本目标检测高效地使用参考图像，并使用大型语言模型（LLMs）进行物体匹配，我们的方法在视觉定位性能上优于最先进的LVLMs（如Qwen2.5-VL-72B）。我们的方法弥合了少样本检测与视觉定位之间的差距，解锁了视觉理解的新能力，在机器人领域具有广泛的应用。项目页面（包含我们的视频、代码和数据集）：https://irvlutd.github.io/MultiGrounding
英文摘要：
Visual grounding focuses on detecting objects from images based on language expressions. Recent Large Vision-Language Models (LVLMs) have significantly advanced visual grounding performance by training large models with large-scale datasets. However, the problem remains challenging, especially when similar objects appear in the input image. For example, an LVLM may not be able to differentiate Diet Coke and regular Coke in an image. In this case, if additional reference images of Diet Coke and regular Coke are available, it can help the visual grounding of similar objects. In this work, we introduce a new task named Multimodal Reference Visual Grounding (MRVG). In this task, a model has access to a set of reference images of objects in a database. Based on these reference images and a language expression, the model is required to detect a target object from a query image. We first introduce a new dataset to study the MRVG problem. Then we introduce a novel method, named MRVG-Net, to solve this visual grounding problem. We show that by efficiently using reference images with few-shot object detection and using Large Language Models (LLMs) for object matching, our method achieves superior visual grounding performance compared to the state-of-the-art LVLMs such as Qwen2.5-VL-72B. Our approach bridges the gap between few-shot detection and visual grounding, unlocking new capabilities for visual understanding, which has wide applications in robotics. Project page with our video, code, and dataset: https://irvlutd.github.io/MultiGrounding
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
