arXiv 2410.056502024-10-08cs.CVcs.MM

SIA-OVD：用于弥合开放词汇检测中图像-区域差距的形状不变适配器

SIA-OVD: Shape-Invariant Adapter for Bridging the Image-Region Gap in Open-Vocabulary Detection

Wangxuan Institute of Computer Technology, Peking University（北京大学王选计算机研究所）；School of Intelligence Science and Technology, University of Science and Technology Beijing（北京科技大学智能科学与技术学院）

Zishuo Wang, Wenhao Zhou, Jinglin Xu, Yuxin Peng

备注：: 9 pages, 7 figures
DOI：: 10.1145/3664647.3680642

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对开放词汇检测中CLIP因图像-区域表征差距导致区域分类不准的问题，提出形状不变适配器SIA-OVD，通过为不同形状区域分配适配特征缩小差距，在COCO-OVD基准上取得显著性能提升。

摘要：

开放词汇检测（OVD）旨在无需实例级标注即可检测新类别物体，以更低成本实现开放世界目标检测。现有OVD方法主要依赖CLIP等视觉语言预训练模型（VLM）强大的开放词汇图文对齐能力。但CLIP基于图文对训练，缺乏对图像内局部区域的感知能力，导致图像表征与区域表征存在差距，直接将CLIP用于OVD会出现区域分类不准确的问题。我们发现图像-区域差距主要由感兴趣区域（RoI）提取过程中区域特征图的形变导致。为缓解OVD中区域分类不准确的问题，我们提出一种名为SIA-OVD的新型形状不变适配器，以弥合OVD任务中的图像-区域差距。SIA-OVD为不同形状的区域学习一组特征适配器，并设计了一种新的适配器分配机制，为每个区域选择最优适配器。经适配的区域表征能与CLIP学习到的文本表征更好地对齐。大量实验表明，SIA-OVD通过解决形状形变导致的图像与区域之间的差距，有效提升了区域分类准确率。在COCO-OVD基准上，SIA-OVD相比代表性方法取得了大幅提升。代码可在https://github.com/PKU-ICST-MIPL/SIA-OVD_ACMMM2024获取。

英文摘要：

Open-vocabulary detection (OVD) aims to detect novel objects without instance-level annotations to achieve open-world object detection at a lower cost. Existing OVD methods mainly rely on the powerful open-vocabulary image-text alignment capability of Vision-Language Pretrained Models (VLM) such as CLIP. However, CLIP is trained on image-text pairs and lacks the perceptual ability for local regions within an image, resulting in the gap between image and region representations. Directly using CLIP for OVD causes inaccurate region classification. We find the image-region gap is primarily caused by the deformation of region feature maps during region of interest (RoI) extraction. To mitigate the inaccurate region classification in OVD, we propose a new Shape-Invariant Adapter named SIA-OVD to bridge the image-region gap in the OVD task. SIA-OVD learns a set of feature adapters for regions with different shapes and designs a new adapter allocation mechanism to select the optimal adapter for each region. The adapted region representations can align better with text representations learned by CLIP. Extensive experiments demonstrate that SIA-OVD effectively improves the classification accuracy for regions by addressing the gap between images and regions caused by shape deformation. SIA-OVD achieves substantial improvements over representative methods on the COCO-OVD benchmark. The code is available at https://github.com/PKU-ICST-MIPL/SIA-OVD_ACMMM2024.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。