LLMs Meet VLMs: 利用细粒度描述符提升开放词汇目标检测
LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptors
AI总结：
本文提出DVDet，一种利用条件上下文提示和分层文本描述符的开放词汇检测器，通过大语言模型挖掘细粒度描述，实现精确区域-文本对齐，并在多个基准上大幅超越现有方法。
摘要：
受视觉语言模型（VLMs）在图像分类任务中出色零样本能力的启发，开放词汇目标检测通过将广泛的VLM知识蒸馏到检测器训练中而日益受到关注。然而，大多数现有的开放词汇检测器仅通过将区域嵌入与类别标签（如自行车）对齐来学习，忽视了VLM在将视觉嵌入与对象部件的细粒度文本描述（如踏板和铃铛）对齐方面的能力。本文提出了DVDet，一种描述符增强的开放词汇检测器，它引入了条件上下文提示和分层文本描述符，实现了精确的区域-文本对齐以及通用的开放词汇检测训练。具体而言，条件上下文提示将区域嵌入转换为类似图像的表示，可直接集成到通用开放词汇检测训练中。此外，我们引入大语言模型作为交互式和隐式知识库，能够迭代挖掘和细化面向视觉的文本描述符，以实现精确的区域-文本对齐。在多个大规模基准上的广泛实验表明，DVDet持续大幅优于最先进的方法。
英文摘要：
Inspired by the outstanding zero-shot capability of vision language models (VLMs) in image classification tasks, open-vocabulary object detection has attracted increasing interest by distilling the broad VLM knowledge into detector training. However, most existing open-vocabulary detectors learn by aligning region embeddings with categorical labels (e.g., bicycle) only, disregarding the capability of VLMs on aligning visual embeddings with fine-grained text description of object parts (e.g., pedals and bells). This paper presents DVDet, a Descriptor-Enhanced Open Vocabulary Detector that introduces conditional context prompts and hierarchical textual descriptors that enable precise region-text alignment as well as open-vocabulary detection training in general. Specifically, the conditional context prompt transforms regional embeddings into image-like representations that can be directly integrated into general open vocabulary detection training. In addition, we introduce large language models as an interactive and implicit knowledge repository which enables iterative mining and refining visually oriented textual descriptors for precise region-text alignment. Extensive experiments over multiple large-scale benchmarks show that DVDet outperforms the state-of-the-art consistently by large margins.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
