CLIP-PING：通过近邻固有邻居引导提升轻量级视觉语言模型性能
CLIP-PING: Boosting Lightweight Vision-Language Models with Proximus Intrinsic Neighbors Guidance
AI总结：
针对轻量级视觉语言模型仅靠图文对比学习性能不佳的问题，提出CLIP-PING训练范式，利用预训练编码器的近邻样本特征提供额外对比监督，以低开销提升模型跨模态对齐效果与零样本、检索及下游迁移性能。
摘要：
在对比语言-图像预训练（Contrastive Language-Image Pre-training, CLIP）取得成功之后，近期的研究趋势转向探索轻量级视觉语言模型在资源受限场景下的适用性。这类模型仅依赖单一的图文对比学习目标时性能往往不佳，凸显了需要更有效的训练机制来保障鲁棒的跨模态特征对齐。本研究提出CLIP-PING：基于近邻固有邻居引导的对比语言-图像预训练，这是一种新颖且简单高效的训练范式，旨在以最小的计算开销和更低的数据需求提升轻量级视觉语言模型的性能。CLIP-PING利用从任意预训练编码器提取的单模态特征进行自举，获取近邻样本的固有引导，即最近邻（nearest-neighbor, NN）和交叉最近邻（cross nearest-neighbor, XNN）。我们发现这些邻居提供的额外对比监督能显著增强跨模态对齐，使轻量级模型能够学习到更具通用性、语义多样性更丰富的特征。大量实验表明，CLIP-PING在零样本泛化和跨模态检索任务上显著优于同类方法。具体而言，当使用在300万（图像，文本）对上训练的ViT-XS图像编码器时，与原始CLIP相比，其在零样本ImageNet1K分类任务上提升了5.5%，在Flickr30K检索任务上，图像到文本（I2T）提升10.7%，文本到图像（T2I）提升5.7%。此外，CLIP-PING在多个下游任务的线性评估协议下展现出强大的迁移能力。
英文摘要：
Beyond the success of Contrastive Language-Image Pre-training (CLIP), recent trends mark a shift toward exploring the applicability of lightweight vision-language models for resource-constrained scenarios. These models often deliver suboptimal performance when relying solely on a single image-text contrastive learning objective, spotlighting the need for more effective training mechanisms that guarantee robust cross-modal feature alignment. In this work, we propose CLIP-PING: Contrastive Language-Image Pre-training with Proximus Intrinsic Neighbors Guidance, a novel yet simple and efficient training paradigm designed to boost the performance of lightweight vision-language models with minimal computational overhead and lower data demands. CLIP-PING bootstraps unimodal features extracted from arbitrary pre-trained encoders to obtain intrinsic guidance of proximus neighbor samples, i.e., nearest-neighbor (NN) and cross nearest-neighbor (XNN). We find that extra contrastive supervision from these neighbors substantially boosts cross-modal alignment, enabling lightweight models to learn more generic features with rich semantic diversity. Extensive experiments reveal that CLIP-PING notably surpasses its peers in zero-shot generalization and cross-modal retrieval tasks. Specifically, a 5.5% gain on zero-shot ImageNet1K classification with 10.7% (I2T) and 5.7% (T2I) on Flickr30K retrieval, compared to the original CLIP when using ViT-XS image encoder trained on 3 million (image, text) pairs. Moreover, CLIP-PING showcases a strong transferability under the linear evaluation protocol across several downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
