arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.14940cs.CV
收藏

基于视觉语言模型的开放词汇目标检测提示学习

Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model

Tsinghua University(清华大学);Microsoft Research Asia(微软亚洲研究院);King’s College London(伦敦国王学院)

Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, Guoqi Li

摘要:

最近,视觉语言预训练在开放词汇目标检测中显示出巨大的潜力,其中在基类上训练的检测器被设计用于检测新的类。类文本嵌入首先通过向预训练的视觉语言模型的文本编码器馈送提示来生成。然后将其用作区域分类器来监督检测器的训练。这一模式成功的关键在于恰当的提示语,这需要精心的词语调整和巧妙的设计。为了避免费力的提示工程,存在针对图像分类任务提出的一些提示表示学习方法,然而,当应用于检测任务时,这些方法只能是次优解决方案。在本文中,我们介绍了一种新的方法,检测提示(DetPro),学习连续提示表示开放词汇的目标检测的基础上预先训练的视觉语言模型。与以往面向分类的方法不同,DetPro有两个亮点:1)背景解释方案,将图像背景中的建议纳入提示训练; 2)上下文分级方案,将图像前景中的建议分离,以进行量身定制的提示训练。我们将DetPro与ViLD(一种最新的开放世界对象检测器)组装在一起,并在LVIS上进行实验,并在Pascal VOC,COCO,Objects 365数据集上进行迁移学习。实验结果表明,我们的DetPro优于基线ViLD在所有设置,例如,对LVIS的新类进行了+3.4 APbox和+3.0 APmask改进。代码和型号可在https://github.com/dyabel/detpro上获得。

英文摘要:

Recently, vision-language pre-training shows great potential in open-vocabulary object detection, where detectors trained on base classes are devised for detecting new classes. The class text embedding is firstly generated by feeding prompts to the text encoder of a pre-trained vision-language model. It is then used as the region classifier to supervise the training of a detector. The key element that leads to the success of this model is the proper prompt, which requires careful words tuning and ingenious design. To avoid laborious prompt engineering, there are some prompt representation learning methods being proposed for the image classification task, which however can only be sub-optimal solutions when applied to the detection task. In this paper, we introduce a novel method, detection prompt (DetPro), to learn continuous prompt representations for open-vocabulary object detection based on the pre-trained vision-language model. Different from the previous classification-oriented methods, DetPro has two highlights: 1) a background interpretation scheme to include the proposals in image background into the prompt training; 2) a context grading scheme to separate proposals in image foreground for tailored prompt training. We assemble DetPro with ViLD, a recent state-of-the-art open-world object detector, and conduct experiments on the LVIS as well as transfer learning on the Pascal VOC, COCO, Objects365 datasets. Experimental results show that our DetPro outperforms the baseline ViLD in all settings, e.g., +3.4 APbox and +3.0 APmask improvements on the novel classes of LVIS. Code and models are available at https://github.com/dyabel/detpro.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑