对比式局部化语言-图像预训练
Contrastive Localized Language-Image Pre-Training
AI总结：
本文提出 CLOC 预训练方法，通过区域-文本对比损失、可提示嵌入及大规模区域伪标签框架增强 CLIP 的细粒度定位能力，并可即插即用地提升 MLLM。
摘要：
对比式语言-图像预训练（CLIP）一直是一种备受赞誉的方法，用于训练视觉编码器以生成促进各类应用的图像/文本表示。近来，CLIP 已被广泛用作多模态大语言模型（MLLM）的视觉骨干，以连接图像输入并进行语言交互。CLIP 作为视觉-语言基础模型的成功依赖于在图像层级对齐网络爬取的带噪声文本标注。然而，对于需要细粒度视觉表示的下游任务，这种准则可能会变得不足，尤其是在 MLLM 需要区域级理解时。本文中，我们通过若干改进提升 CLIP 的定位能力。我们提出一种名为 Contrastive Localized Language-Image Pre-training（CLOC）的预训练方法，通过区域-文本对比损失和模块对 CLIP 进行补充。我们提出了一个新概念，即可提示嵌入（promptable embeddings），编码器由此产生的图像嵌入在给定空间提示时易于转换为区域表示。为支持大规模预训练，我们设计了一个视觉丰富且空间局部化的描述框架，以有效大规模生成区域-文本伪标签。通过扩展到数十亿张带标注图像，CLOC 为图像区域识别和检索任务提供高质量区域嵌入，并可作为 CLIP 的即插即用替代方案来增强 MLLM，尤其是在指代表达和定位任务上。
英文摘要：
Contrastive Language-Image Pre-training (CLIP) has been a celebrated method for training vision encoders to generate image/text representations facilitating various applications. Recently, CLIP has been widely adopted as the vision backbone of multimodal large language models (MLLMs) to connect image inputs for language interactions. The success of CLIP as a vision-language foundation model relies on aligning web-crawled noisy text annotations at image levels. Nevertheless, such criteria may become insufficient for downstream tasks in need of fine-grained vision representations, especially when region-level understanding is demanding for MLLMs. In this paper, we improve the localization capability of CLIP with several advances. We propose a pre-training method called Contrastive Localized Language-Image Pre-training (CLOC) by complementing CLIP with region-text contrastive loss and modules. We formulate a new concept, promptable embeddings, of which the encoder produces image embeddings easy to transform into region representations given spatial hints. To support large-scale pre-training, we design a visually-enriched and spatially-localized captioning framework to effectively generate region-text pseudo-labels at scale. By scaling up to billions of annotated images, CLOC enables high-quality regional embeddings for image region recognition and retrieval tasks, and can be a drop-in replacement of CLIP to enhance MLLMs, especially on referring and grounding tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
