HarmoCLIP：在对比视觉语言模型中协调全局和区域表示
HarmoCLIP: Harmonizing Global and Regional Representations in Contrastive Vision-Language Models
AI总结：
HarmoCLIP通过引入细粒度语义监督和区域-语言对齐策略，协调CLIP中全局与区域表示，提升检索和边界框分类性能，实现平衡高效的全局-局部权衡解决方案。
摘要：
对比语言-图像预训练（CLIP）已展示了显著的泛化能力和在广泛视觉语言任务中的强大性能。然而，由于缺乏区域级监督，CLIP在细粒度语义理解上表现有限。尽管有几种方法试图缓解这一问题，但它们无意中破坏了全局对齐，导致一种持久的权衡，即改进局部感知的同时会降级全局一致性。在本文中，我们提出HarmoCLIP，一个旨在协调CLIP中全局和区域表示的新框架。我们首先发现，局部文本和视觉语义之间缺乏直接对齐是权衡的根本原因。为此，HarmoCLIP引入了一个显式的细粒度语义监督项，直接对齐文本段与其对应的视觉区域，有效地将图像区域空间与文本空间连接起来。为了进一步加强局部层面的表示能力，我们的方法引入了一种新的区域-语言对齐监督策略，促进细粒度语义学习而不影响全局语义一致性。广泛的实验表明，HarmoCLIP在全局检索任务中实现了最先进的性能（提升高达69.78%），并在边界框分类任务中实现了显著的3.2%准确率提升，持续优于先前的方法，同时提供了一个平衡、高效且易于使用的解决方案，以解决CLIP中的全局-局部权衡问题。代码可在https://github.com/Erosist/HarmoCLIP上获得。
英文摘要：
Contrastive Language-Image Pre-training (CLIP) has demonstrated remarkable generalization ability and strong performance across a wide range of vision-language tasks. However, due to the lack of region-level supervision, CLIP exhibits limited fine-grained semantic understanding. Although several methods attempt to mitigate this issue, they unintentionally disrupt the global alignment, resulting in a persistent trade-off where improving local perception simultaneously degrades global coherence. In this paper, we propose HarmoCLIP, a novel framework designed to harmonize global and region representations within CLIP. We first identify that the absence of direct alignment between local textual and visual semantics is the fundamental cause of the trade-off. To address this, HarmoCLIP introduces an explicit fine-grained semantic supervision term that directly aligns textual segments with their corresponding visual regions, effectively bridging the image region space and the textual space. To further strengthen the representation capability at the local level, our method introduces a novel Region-Language Alignment supervision strategy that promotes fine-grained semantic learning without compromising global semantic consistency. Extensive experiments demonstrate that HarmoCLIP achieves state-of-the-art (improvement up to 69.78%) performance on the global task of retrieval and yields a substantial 3.2% improvement in Top-1 accuracy on the region task of bounding-box classification, consistently outperforming prior approaches while providing a balanced, efficient, and plug-and-play solution to the global-local trade-off in CLIP. Code is available at https://github.com/Erosist/HarmoCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
