UMG-CLIP：面向开放世界理解的统一多粒度视觉通用模型
UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understanding
AI总结：
针对现有CLIP类模型忽略局部区域与文本对齐的问题，提出多粒度对齐的UMG-CLIP框架，构建多粒度伪标注数据集，经参数高效微调后在多项图像理解基准上取得SOTA性能。
摘要：
以Contrastive Language-Image Pre-training（CLIP，对比语言-图像预训练）为代表的视觉-语言基础模型因可联合理解视觉与文本任务而受到越来越多的关注。然而现有方法主要聚焦于训练模型将全局图像表示与文本描述匹配，忽略了局部区域与对应文本标记之间的关键对齐。本文通过多粒度对齐对CLIP进行扩展。值得注意的是，我们特意构建了一个包含多种粒度伪标注的新数据集，涵盖图像级、区域级以及像素级的描述与标签。相应地，我们开发了统一多粒度学习框架，命名为UMG-CLIP，该框架可同时赋予模型跨不同细节层级的多样感知能力。通过参数高效微调，UMG-CLIP超越了当前广泛使用的CLIP变体，在多种图像理解基准（包括开放世界识别、检索、语义分割和全景分割任务）上取得了最优性能。我们认为UMG-CLIP是视觉-语言基础模型领域的一项有价值的进展。代码可在https://github.com/lygsbw/UMG-CLIP获取。
英文摘要：
Vision-language foundation models, represented by Contrastive Language-Image Pre-training (CLIP), have gained increasing attention for jointly understanding both vision and textual tasks. However, existing approaches primarily focus on training models to match global image representations with textual descriptions, thereby overlooking the critical alignment between local regions and corresponding text tokens. This paper extends CLIP with multi-granularity alignment. Notably, we deliberately construct a new dataset comprising pseudo annotations at various levels of granularities, encompassing image-level, region-level as well as pixel-level captions and tags. Accordingly, we develop a Unified Multi-Granularity learning framework, termed UMG-CLIP, which simultaneously empowers the model with versatile perception abilities across different levels of detail. With parameter efficient tuning, UMG-CLIP surpasses current widely used CLIP variants and achieves state-of-the-art performance on diverse image understanding benchmarks, including open-world recognition, retrieval, semantic segmentation, and panoptic segmentation tasks. We believe that UMG-CLIP represents a valuable advancement in vision-language foundation models. The code is available at https://github.com/lygsbw/UMG-CLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
