SmartCLIP: 基于识别保证的模块化视觉-语言对齐
SmartCLIP: Modular Vision-language Alignment with Identification Guarantees
AI总结：
本文提出SmartCLIP，通过理论条件实现文本与视觉表征的灵活对齐，确保语义信息完整保留并解耦视觉表征，提升多任务性能。
摘要：
对比学习语言-图像预训练（CLIP）已成为计算机视觉和多模态学习的关键模型，通过对比学习实现视觉与文本表征对齐。然而，CLIP在许多图像-文本数据集中存在潜在信息错位问题，并且表征纠缠。本文建立了理论条件，使模型能够在不同粒度层面灵活对齐文本与视觉表征。具体而言，我们的框架确保模型不仅能完整保留跨模态语义信息，还能解耦视觉表征以捕捉细粒度文本概念。基于此，我们引入了\ours方法，通过模块化方式识别并对齐最相关的视觉和文本表征。在各种任务上的优越性能证明了其处理信息错位和支撑识别理论的能力。代码可在https://github.com/Mid-Push/SmartCLIP获取。
英文摘要：
Contrastive Language-Image Pre-training (CLIP)~\citep{radford2021learning} has emerged as a pivotal model in computer vision and multimodal learning, achieving state-of-the-art performance at aligning visual and textual representations through contrastive learning. However, CLIP struggles with potential information misalignment in many image-text datasets and suffers from entangled representation. On the one hand, short captions for a single image in datasets like MSCOCO may describe disjoint regions in the image, leaving the model uncertain about which visual features to retain or disregard. On the other hand, directly aligning long captions with images can lead to the retention of entangled details, preventing the model from learning disentangled, atomic concepts -- ultimately limiting its generalization on certain downstream tasks involving short prompts. In this paper, we establish theoretical conditions that enable flexible alignment between textual and visual representations across varying levels of granularity. Specifically, our framework ensures that a model can not only \emph{preserve} cross-modal semantic information in its entirety but also \emph{disentangle} visual representations to capture fine-grained textual concepts. Building on this foundation, we introduce \ours, a novel approach that identifies and aligns the most relevant visual and textual representations in a modular manner. Superior performance across various tasks demonstrates its capability to handle information misalignment and supports our identification theory. The code is available at https://github.com/Mid-Push/SmartCLIP.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
