arXiv 2303.054992024-07-19cs.CV

Grounding DINO：将DINO与接地预训练相结合用于开放集目标检测

Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection

International Digital Economy Academy (IDEA)（国际数字经济学院（IDEA））；Institute for AI, Tsinghua-Bosch Joint Center for ML, Tsinghua University（清华大学人工智能研究院、清华-博世联合机器学习中心）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong (Shenzhen)（香港中文大学（深圳））；Microsoft Research（微软研究院）；South China University of Technology（华南理工大学）

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, Lei Zhang

备注：: Code will be available at https://github.com/IDEA-Research/GroundingDINO

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出Grounding DINO，一种结合DINO与接地预训练的开放集目标检测器，通过紧密融合语言和视觉模态，在COCO、LVIS、ODinW和RefCOCO等基准上实现零样本检测，并刷新了ODinW纪录。

摘要：

在本文中，我们提出了一种名为Grounding DINO的开放集目标检测器，通过将基于Transformer的检测器DINO与接地预训练相结合，该检测器能够检测任意对象，输入可以是类别名称或指代表达式等人类输入。开放集目标检测的关键解决方案是将语言引入到封闭集检测器中，以实现开放集概念的泛化。为了有效融合语言和视觉模态，我们概念性地将封闭集检测器划分为三个阶段，并提出了一种紧密融合解决方案，该方案包括特征增强器、语言引导的查询选择以及用于跨模态融合的跨模态解码器。虽然以往的工作主要在新颖类别上评估开放集目标检测，但我们提出还要对指定属性的对象的指代表达理解进行评估。Grounding DINO在三种设置下均表现出色，包括COCO、LVIS、ODinW和RefCOCO/+/g基准测试。Grounding DINO在COCO检测零样本迁移基准上达到了52.5的AP，即未使用任何COCO训练数据。它在ODinW零样本基准上以平均26.1的AP创造了新纪录。代码将在https://github.com/IDEA-Research/GroundingDINO提供。

英文摘要：

In this paper, we present an open-set object detector, called Grounding DINO, by marrying Transformer-based detector DINO with grounded pre-training, which can detect arbitrary objects with human inputs such as category names or referring expressions. The key solution of open-set object detection is introducing language to a closed-set detector for open-set concept generalization. To effectively fuse language and vision modalities, we conceptually divide a closed-set detector into three phases and propose a tight fusion solution, which includes a feature enhancer, a language-guided query selection, and a cross-modality decoder for cross-modality fusion. While previous works mainly evaluate open-set object detection on novel categories, we propose to also perform evaluations on referring expression comprehension for objects specified with attributes. Grounding DINO performs remarkably well on all three settings, including benchmarks on COCO, LVIS, ODinW, and RefCOCO/+/g. Grounding DINO achieves a $52.5$ AP on the COCO detection zero-shot transfer benchmark, i.e., without any training data from COCO. It sets a new record on the ODinW zero-shot benchmark with a mean $26.1$ AP. Code will be available at \url{https://github.com/IDEA-Research/GroundingDINO}.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。