ResVG：增强多实例视觉指代中的关系与语义理解
ResVG: Enhancing Relation and Semantic Understanding in Multiple Instances for Visual Grounding
AI总结：
提出ResVG模型，通过注入文本语义先验和关系敏感的数据增强方法，显著提升了模型在多实例干扰场景下的视觉指代定位性能。
摘要：
视觉指代旨在根据自然语言查询定位图像中所指代的目标。尽管近期取得了一些进展，但在多实例干扰（与目标同类的多个物体）存在的情况下，准确定位目标物体仍然是一个重大挑战。现有方法在图像中存在多个干扰物时表现出显著的性能下降，这表明对细粒度语义以及物体间的空间关系理解不足。在本文中，我们提出了一种新方法，即关系与语义敏感的视觉指代（ResVG）模型，来解决这一问题。首先，我们通过将源自文本查询的语义先验信息注入模型，增强了模型对细粒度语义的理解。这是通过利用文本到图像生成模型，生成代表查询中所描述目标物体语义属性的图像来实现的。其次，我们通过引入一种关系敏感的数据增强方法，解决了缺乏包含多个干扰物的训练样本的问题。该方法通过合成包含同类多个物体的图像以及基于它们空间关系的伪查询，生成了额外的训练数据。提出的 ResVG 模型显著提升了模型理解物体语义和空间关系的能力，从而在视觉指代任务中取得了更好的性能，特别是在多实例干扰场景下。我们在五个数据集上进行了广泛的实验，以验证我们方法的有效性。代码可在 https://github.com/minghangz/ResVG 获取。
英文摘要：
Visual grounding aims to localize the object referred to in an image based on a natural language query. Although progress has been made recently, accurately localizing target objects within multiple-instance distractions (multiple objects of the same category as the target) remains a significant challenge. Existing methods demonstrate a significant performance drop when there are multiple distractions in an image, indicating an insufficient understanding of the fine-grained semantics and spatial relationships between objects. In this paper, we propose a novel approach, the Relation and Semantic-sensitive Visual Grounding (ResVG) model, to address this issue. Firstly, we enhance the model's understanding of fine-grained semantics by injecting semantic prior information derived from text queries into the model. This is achieved by leveraging text-to-image generation models to produce images representing the semantic attributes of target objects described in queries. Secondly, we tackle the lack of training samples with multiple distractions by introducing a relation-sensitive data augmentation method. This method generates additional training data by synthesizing images containing multiple objects of the same category and pseudo queries based on their spatial relationships. The proposed ReSVG model significantly improves the model's ability to comprehend both object semantics and spatial relations, leading to enhanced performance in visual grounding tasks, particularly in scenarios with multiple-instance distractions. We conduct extensive experiments to validate the effectiveness of our methods on five datasets. Code is available at https://github.com/minghangz/ResVG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
