arXiv 2312.113762023-12-19cs.CV

CLIM：用于区域表征的对比语言-图像拼接

CLIM: Contrastive Language-Image Mosaic for Region Representation

Nanyang Technological University（南洋理工大学）；The Chinese University of Hong Kong（香港中文大学）；The University of Hong Kong（香港大学）；SenseTime Research（商汤科技研究院）；Shanghai AI Laboratory（上海人工智能实验室）

Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Wentao Liu, Chen Change Loy

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对开放词汇目标检测中区域-文本对齐标注成本高的问题，提出CLIM方法，通过拼接图像构造伪区域，利用对比损失和图像-文本对学习对齐，有效提升开放词汇检测器性能及视觉语言模型区域表征能力。

摘要：

从大词汇表或开放词汇表中准确检测目标，需要在区域表征上实现视觉-语言对齐。然而，通过获取带有文本标签或描述的高质量边界框标注来学习这种区域-文本对齐，成本高昂且难以实现。相比之下，收集图像-文本对更简单，但缺乏将区域与文本关联的精确目标位置信息。本文提出一种名为Contrastive Language-Image Mosaic（CLIM，对比语言-图像拼接）的新方法，可有效利用大规模图像-文本对实现区域与文本表征的对齐。CLIM将多张图像组合成一张拼接图像，并将每张图像视为一个“伪区域”。该方法提取每个伪区域的特征，通过对比损失训练使其与对应文本嵌入相似、与其他文本嵌入相异，让模型无需昂贵的边界框标注即可学习区域-文本对齐。作为一种通用方法，CLIM持续提升了多种使用字幕监督的开放词汇目标检测方法的性能。此外，CLIM能有效增强视觉-语言模型的区域表征，从而为开放词汇目标检测器提供更强的骨干网络。实验结果表明，在OV-COCO和OV-LVIS基准测试上，CLIM均大幅提升了多种基线开放词汇目标检测器的性能。代码可在https://github.com/wusize/CLIM获取。

英文摘要：

Detecting objects accurately from a large or open vocabulary necessitates the vision-language alignment on region representations. However, learning such a region-text alignment by obtaining high-quality box annotations with text labels or descriptions is expensive and infeasible. In contrast, collecting image-text pairs is simpler but lacks precise object location information to associate regions with texts. In this paper, we propose a novel approach called Contrastive Language-Image Mosaic (CLIM), which leverages large-scale image-text pairs effectively for aligning region and text representations. CLIM combines multiple images into a mosaicked image and treats each image as a `pseudo region'. The feature of each pseudo region is extracted and trained to be similar to the corresponding text embedding while dissimilar from others by a contrastive loss, enabling the model to learn the region-text alignment without costly box annotations. As a generally applicable approach, CLIM consistently improves different open-vocabulary object detection methods that use caption supervision. Furthermore, CLIM can effectively enhance the region representation of vision-language models, thus providing stronger backbones for open-vocabulary object detectors. Our experimental results demonstrate that CLIM improves different baseline open-vocabulary object detectors by a large margin on both OV-COCO and OV-LVIS benchmarks. The code is available at https://github.com/wusize/CLIM.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。