arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.03339cs.CV
收藏

通过场景图发现实现开放词汇目标检测

Open-Vocabulary Object Detection via Scene Graph Discovery

Monash University(莫纳什大学)

Hengcan Shi, Munawar Hayat, Jianfei Cai

AI总结:

本文提出基于场景图的发现网络SGDN,通过场景图引导注意力、偏移回归和跨模态学习,利用场景图线索提升开放词汇目标检测性能,并首次实现OV场景图检测。

摘要:

近年来,开放词汇(OV)目标检测吸引了越来越多的研究关注。与传统检测只能识别固定类别的目标不同,OV检测旨在检测开放类别集合中的目标。以往的工作通常利用视觉-语言(VL)训练数据(例如指称定位数据)来识别OV目标。然而,它们仅使用VL数据中的名词与单个目标之间的配对,而这些数据通常包含更丰富的信息,例如场景图,这些信息对OV检测同样至关重要。本文提出了一种新颖的基于场景图的发现网络(SGDN),利用场景图线索进行OV检测。首先,提出了一个基于场景图的解码器(SGDecoder),其中包含稀疏场景图引导注意力(SSGA)。该解码器捕获场景图并利用其发现OV目标。其次,我们提出了基于场景图的预测(SGPred),在其中构建了基于场景图的偏移回归(SGOR)机制,以实现场景图提取与目标定位之间的相互增强。第三,我们在SGPred中设计了一种跨模态学习机制。该机制以场景图作为桥梁,提升跨模态嵌入之间的一致性,从而用于OV目标分类。在COCO和LVIS上的实验证明了我们方法的有效性。此外,我们展示了模型在OV场景图检测上的能力,而以往的OV场景图生成方法无法完成该任务。

英文摘要:

In recent years, open-vocabulary (OV) object detection has attracted increasing research attention. Unlike traditional detection, which only recognizes fixed-category objects, OV detection aims to detect objects in an open category set. Previous works often leverage vision-language (VL) training data (e.g., referring grounding data) to recognize OV objects. However, they only use pairs of nouns and individual objects in VL data, while these data usually contain much more information, such as scene graphs, which are also crucial for OV detection. In this paper, we propose a novel Scene-Graph-Based Discovery Network (SGDN) that exploits scene graph cues for OV detection. Firstly, a scene-graph-based decoder (SGDecoder) including sparse scene-graph-guided attention (SSGA) is presented. It captures scene graphs and leverages them to discover OV objects. Secondly, we propose scene-graph-based prediction (SGPred), where we build a scene-graph-based offset regression (SGOR) mechanism to enable mutual enhancement between scene graph extraction and object localization. Thirdly, we design a cross-modal learning mechanism in SGPred. It takes scene graphs as bridges to improve the consistency between cross-modal embeddings for OV object classification. Experiments on COCO and LVIS demonstrate the effectiveness of our approach. Moreover, we show the ability of our model for OV scene graph detection, while previous OV scene graph generation methods cannot tackle this task.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑