RWKV-CLIP：一个鲁棒的视觉-语言表示学习器
RWKV-CLIP: A Robust Vision-Language Representation Learner
AI总结：
本文提出RWKV-CLIP，首个结合transformers并行训练与RNN高效推理的视觉-语言表示学习模型，并引入利用LLMs生成高质量描述的数据框架，在多个下游任务上取得最先进性能。
摘要：
对比语言-图像预训练（CLIP）通过利用从网站获取的图像-文本对扩充数据集，显著提升了各种视觉-语言任务的性能。本文从数据和模型架构的角度进一步探索CLIP。为了解决噪声数据的普遍存在问题并提高从互联网抓取的大规模图像-文本数据的质量，我们引入了一个多样化的描述生成框架，该框架可以利用大型语言模型（LLMs）来合成和提炼基于网络的文本、合成字幕和检测标签的内容。此外，我们提出了RWKV-CLIP，这是第一个由RWKV驱动的视觉-语言表示学习模型，它结合了transformers的有效并行训练和RNN的高效推理。在各种模型规模和预训练数据集上进行的综合实验表明，RWKV-CLIP是一个鲁棒且高效的视觉-语言表示学习器，它在多个下游任务中取得了最先进的性能，包括线性探测、零样本分类和零样本图像-文本检索。为了促进未来的研究，代码和预训练模型已发布于https://github.com/deepglint/RWKV-CLIP。
英文摘要：
Contrastive Language-Image Pre-training (CLIP) has significantly improved performance in various vision-language tasks by expanding the dataset with image-text pairs obtained from websites. This paper further explores CLIP from the perspectives of data and model architecture. To address the prevalence of noisy data and enhance the quality of large-scale image-text data crawled from the internet, we introduce a diverse description generation framework that can leverage Large Language Models (LLMs) to synthesize and refine content from web-based texts, synthetic captions, and detection tags. Furthermore, we propose RWKV-CLIP, the first RWKV-driven vision-language representation learning model that combines the effective parallel training of transformers with the efficient inference of RNNs. Comprehensive experiments across various model scales and pre-training datasets demonstrate that RWKV-CLIP is a robust and efficient vision-language representation learner, it achieves state-of-the-art performance in several downstream tasks, including linear probe, zero-shot classification, and zero-shot image-text retrieval. To facilitate future research, the code and pre-trained models are released at https://github.com/deepglint/RWKV-CLIP
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
