面向鲁棒视觉定位的上下文解耦与原型继承
Context Disentangling and Prototype Inheriting for Robust Visual Grounding
AI总结：
针对现有视觉定位方法低估上下文判别信息、开放词汇场景泛化性差的问题，提出含上下文解耦与原型继承的框架，在两类场景下性能均超越现有最优水平。
摘要：
视觉定位（VG）旨在根据给定的语言查询在图像中定位特定目标。来自上下文的判别性信息对于区分目标与其他物体至关重要，尤其是当目标与其他物体属于同一类别时。然而，以往大多数方法都低估了这类信息的作用。此外，这些方法通常是为标准场景（不存在任何新物体）设计的，这限制了它们在开放词汇场景下的泛化能力。本文提出了一种包含上下文解耦与原型继承的新型鲁棒视觉定位框架，可同时处理上述两种场景。具体而言，上下文解耦模块将指代对象与上下文特征解耦，从而实现更好的区分度。原型继承模块通过原型库继承从解耦视觉特征中发现的原型，以充分利用已见数据，这在开放词汇场景下尤为有效。为避免大幅调整两类特征的重要性，我们对解耦后的语言特征与原型视觉特征进行哈达玛积运算得到融合特征，随后为其附加一个特殊token并输入视觉Transformer编码器进行边界框回归。我们在标准场景和开放词汇场景下均开展了大量实验。性能对比结果表明，我们的方法在两种场景下的表现均优于现有最优方法。代码可在https://github.com/WayneTomas/TransCP获取。
英文摘要：
Visual grounding (VG) aims to locate a specific target in an image based on a given language query. The discriminative information from context is important for distinguishing the target from other objects, particularly for the targets that have the same category as others. However, most previous methods underestimate such information. Moreover, they are usually designed for the standard scene (without any novel object), which limits their generalization to the open-vocabulary scene. In this paper, we propose a novel framework with context disentangling and prototype inheriting for robust visual grounding to handle both scenes. Specifically, the context disentangling disentangles the referent and context features, which achieves better discrimination between them. The prototype inheriting inherits the prototypes discovered from the disentangled visual features by a prototype bank to fully utilize the seen data, especially for the open-vocabulary scene. The fused features, obtained by leveraging Hadamard product on disentangled linguistic and visual features of prototypes to avoid sharp adjusting the importance between the two types of features, are then attached with a special token and feed to a vision Transformer encoder for bounding box regression. Extensive experiments are conducted on both standard and open-vocabulary scenes. The performance comparisons indicate that our method outperforms the state-of-the-art methods in both scenarios. {The code is available at https://github.com/WayneTomas/TransCP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
