$β$-CLIP：多粒度视觉-语言对齐的文本条件对比学习
$β$-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignment
AI总结：
$β$-CLIP通过多粒度文本条件对比学习提升视觉-语言对齐，实现从完整描述到短语的层次化对齐，改进细粒度和长文本检索性能。
摘要：
CLIP通过对齐全局视觉和文本表示实现了强大的零样本图像-文本检索，但在细粒度任务上即使在长详细描述上微调仍表现不佳。本文提出$β$-CLIP，一种多粒度文本条件对比学习框架，旨在实现从完整描述到句子和短语的多粒度文本与相应视觉区域的分层对齐。对于每个粒度级别，$β$-CLIP利用交叉注意力动态池化图像块，生成上下文化的视觉嵌入。为了解决这种层次结构中的语义重叠问题，我们引入了$β$-Contextualized Contrastive Alignment Loss（$β$-CAL）。该目标参数化了严格查询特定匹配与放松的图像内上下文化之间的权衡，支持软交叉熵和硬二元交叉熵的两种形式。我们发现每种损失与层次监督的交互方式不同：CE的softmax增强细粒度辨别，而BCE的sigmoid有利于长文本检索，两者均受益于层次结构。通过广泛的实验，我们证明$β$-CLIP显著提高了密集对齐：在Urban1K上达到91.8% T2I和92.3% I2T在R@1，FG-OVD（Hard）上达到30.9%，在无硬负样本训练的方法中设定了最先进的水平。$β$-CLIP建立了稳健、适应性强的密集视觉-语言对应基线。代码和模型已发布在https://github.com/fzohra/B-CLIP。
英文摘要：
CLIP achieves strong zero-shot image-text retrieval by aligning global vision and text representations, yet it falls behind on fine-grained tasks even when fine-tuned on long, detailed captions. In this work, we propose $β$-CLIP, a multi-granular text-conditioned contrastive learning framework designed to achieve hierarchical alignment between multiple textual granularities-from full captions to sentences and phrases-and their corresponding visual regions. For each level of granularity, $β$-CLIP utilizes cross-attention to dynamically pool image patches, producing contextualized visual embeddings. To address the semantic overlap inherent in this hierarchy, we introduce the $β$-Contextualized Contrastive Alignment Loss ($β$-CAL). This objective parameterizes the trade-off between strict query-specific matching and relaxed intra-image contextualization, supporting both soft Cross-Entropy and hard Binary Cross-Entropy formulations. We find that each loss interacts differently with hierarchical supervision: CE's softmax sharpens fine-grained discrimination, while BCE's sigmoid favors long-text retrieval while both benefit from hierarchy. Through extensive experiments, we demonstrate that $β$-CLIP significantly improves dense alignment: achieving 91.8% T2I 92.3% I2T at R@1 on Urban1K and 30.9% on FG-OVD (Hard), setting state-of-the-art among methods trained without hard negatives. $β$-CLIP establishes a robust, adaptive baseline for dense vision-language correspondence. The code and models are released at https://github.com/fzohra/B-CLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
