CapDet：统一密集字幕和开放世界检测预训练
CapDet: Unifying Dense Captioning and Open-World Detection Pretraining
摘要：
得益于大规模的视觉语言预训练，开放世界检测方法在zero-shot或Few-Shot检测环境下表现出了优越的泛化能力。然而，一个预定义的类别空间仍然需要在现有的方法的推理阶段，只有属于该空间的对象将被预测。为了介绍一个“真正的”开放世界检测器，在本文中，我们提出了一种新的方法，名为CapDet，要么预测下一个给定的类别列表或直接生成预测的边界框的类别。具体来说，我们统一的开放世界检测和密集字幕任务到一个单一而有效的框架，通过引入一个额外的密集字幕头生成区域接地字幕。此外，添加字幕任务将反过来有利于检测性能的泛化，因为字幕数据集涵盖更多的概念。实验结果表明，通过统一密集字幕任务，我们的CapDet获得了显着的性能改善（例如，+2.1%mAP（LVIS罕见类别），超过LVIS基线方法（1203个类别）。此外，我们的CapDet还在密集字幕任务上实现了最先进的性能，例如，VG V1.2上的mAP为15.44%，VG-COCO数据集上的mAP为13.98%。
英文摘要：
Benefiting from large-scale vision-language pre-training on image-text pairs, open-world detection methods have shown superior generalization ability under the zero-shot or few-shot detection settings. However, a pre-defined category space is still required during the inference stage of existing methods and only the objects belonging to that space will be predicted. To introduce a "real" open-world detector, in this paper, we propose a novel method named CapDet to either predict under a given category list or directly generate the category of predicted bounding boxes. Specifically, we unify the open-world detection and dense caption tasks into a single yet effective framework by introducing an additional dense captioning head to generate the region-grounded captions. Besides, adding the captioning task will in turn benefit the generalization of detection performance since the captioning dataset covers more concepts. Experiment results show that by unifying the dense caption task, our CapDet has obtained significant performance improvements (e.g., +2.1% mAP on LVIS rare classes) over the baseline method on LVIS (1203 classes). Besides, our CapDet also achieves state-of-the-art performance on dense captioning tasks, e.g., 15.44% mAP on VG V1.2 and 13.98% on the VG-COCO dataset.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
