基于实例感知联合学习改进广义视觉定位
Improving Generalized Visual Grounding with Instance-aware Joint Learning
AI总结：
提出具备实例感知能力的多任务广义视觉定位框架InstanceVG，通过实例查询统一边界框与掩码的联合及一致性预测，在多数据集上实现最先进性能。
摘要：
广义视觉定位任务，包括广义指代表达式理解（GREC）和分割（GRES），通过适应多目标和非目标场景，扩展了经典的视觉定位范式。具体而言，GREC专注于在粗略边界框级别准确识别所有指代对象，而GRES旨在实现细粒度的像素级感知。然而，现有方法通常独立处理这些任务，忽略了联合训练GREC和GRES以确保一致的多粒度预测并简化整体流程的好处。此外，当前方法通常将GRES视为语义分割任务，忽略了实例感知能力的关键作用以及确保实例级边界框和掩码之间预测一致的必要性。为了解决这些局限性，我们提出了InstanceVG，这是一个具备实例感知能力的多任务广义视觉定位框架，它利用实例查询来统一实例级边界框和掩码的联合与一致性预测。据我们所知，InstanceVG是第一个同时处理GREC和GRES，并将实例感知能力纳入广义视觉定位的框架。为了实例化该框架，我们为每个实例查询分配一个先验参考点，该参考点也作为目标匹配的额外基础。这种设计促进了同一实例的点、边界框和掩码的一致预测。在四个任务的十个数据集上进行的大量实验表明，InstanceVG实现了最先进的性能，在各项评估指标上显著超越了现有方法。代码和模型将在 https://github.com/Dmmm1997/InstanceVG 公开提供。
英文摘要：
Generalized visual grounding tasks, including Generalized Referring Expression Comprehension (GREC) and Segmentation (GRES), extend the classical visual grounding paradigm by accommodating multi-target and non-target scenarios. Specifically, GREC focuses on accurately identifying all referential objects at the coarse bounding box level, while GRES aims for achieve fine-grained pixel-level perception. However, existing approaches typically treat these tasks independently, overlooking the benefits of jointly training GREC and GRES to ensure consistent multi-granularity predictions and streamline the overall process. Moreover, current methods often treat GRES as a semantic segmentation task, neglecting the crucial role of instance-aware capabilities and the necessity of ensuring consistent predictions between instance-level boxes and masks. To address these limitations, we propose InstanceVG, a multi-task generalized visual grounding framework equipped with instance-aware capabilities, which leverages instance queries to unify the joint and consistency predictions of instance-level boxes and masks. To the best of our knowledge, InstanceVG is the first framework to simultaneously tackle both GREC and GRES while incorporating instance-aware capabilities into generalized visual grounding. To instantiate the framework, we assign each instance query a prior reference point, which also serves as an additional basis for target matching. This design facilitates consistent predictions of points, boxes, and masks for the same instance. Extensive experiments obtained on ten datasets across four tasks demonstrate that InstanceVG achieves state-of-the-art performance, significantly surpassing the existing methods in various evaluation metrics. The code and model will be publicly available at https://github.com/Dmmm1997/InstanceVG.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
