基于视觉-语言模型重新审视少样本目标检测
Revisiting Few-Shot Object Detection with Vision-Language Models
AI总结：
本文基于视觉-语言模型重新审视少样本目标检测，提出 Foundational FSOD 基准协议，利用多模态（文本和视觉）K-shot 样本对齐基础模型，并分享 CVPR 2024 竞赛见解。
摘要：
在网络规模数据集上训练的视觉-语言模型（VLMs）时代，对“开放世界”感知的传统定义提出了挑战。在本研究中，我们重新审视了近期基础 VLMs 背景下的少样本目标检测（FSOD）任务。首先，我们指出，诸如 GroundingDINO 等 VLMs 的零样本预测在 COCO 数据集上显著优于最先进的少样本检测器（48 对 33 AP）。尽管它们具有强大的零样本性能，此类基础模型仍可能不是最优的。例如，网络上的卡车定义可能与自动驾驶感知等目标应用中的卡车定义不同。我们认为，少样本识别任务可以重新定义为：使用少量样本将基础模型对齐到目标概念。有趣的是，这些样本可以是多模态的，同时使用文本和视觉线索，模仿在定义感兴趣的目标概念时通常给人类标注员的指令。具体而言，我们提出了 Foundational FSOD，这是一种新的基准协议，用于评估在任何外部数据上预训练并在每个目标类的多模态（文本和视觉）K-shot 样本上微调的检测器。我们将 nuImages 重新用于 Foundational FSOD，对几种流行的开源 VLMs 进行基准测试，并对最先进的方法进行实证分析。最后，我们讨论了最近的 CVPR 2024 Foundational FSOD 竞赛，并分享了社区的见解。值得注意的是，获胜团队以 23.3 mAP 的优势显著优于我们的基线！我们的代码和数据集划分可在 https://github.com/anishmadan23/foundational_fsod 获取。
英文摘要：
The era of vision-language models (VLMs) trained on web-scale datasets challenges conventional formulations of "open-world" perception. In this work, we revisit the task of few-shot object detection (FSOD) in the context of recent foundational VLMs. First, we point out that zero-shot predictions from VLMs such as GroundingDINO significantly outperform state-of-the-art few-shot detectors (48 vs. 33 AP) on COCO. Despite their strong zero-shot performance, such foundation models may still be sub-optimal. For example, trucks on the web may be defined differently from trucks for a target application such as autonomous vehicle perception. We argue that the task of few-shot recognition can be reformulated as aligning foundation models to target concepts using a few examples. Interestingly, such examples can be multi-modal, using both text and visual cues, mimicking instructions that are often given to human annotators when defining a target concept of interest. Concretely, we propose Foundational FSOD, a new benchmark protocol that evaluates detectors pre-trained on any external data and fine-tuned on multi-modal (text and visual) K-shot examples per target class. We repurpose nuImages for Foundational FSOD, benchmark several popular open-source VLMs, and provide an empirical analysis of state-of-the-art methods. Lastly, we discuss our recent CVPR 2024 Foundational FSOD competition and share insights from the community. Notably, the winning team significantly outperforms our baseline by 23.3 mAP! Our code and dataset splits are available at https://github.com/anishmadan23/foundational_fsod
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
