HQ-CLIP：利用大型视觉语言模型创建高质量图像-文本数据集与CLIP模型
HQ-CLIP: Leveraging Large Vision-Language Models to Create High-Quality Image-Text Datasets and CLIP Models
AI总结：
本研究提出由大型视觉语言模型驱动的数据精炼流水线，生成多粒度文本标注构建VLM-150M数据集，结合含额外监督信号的对比学习范式训练HQ-CLIP，在多项任务中实现性能突破。
摘要：
大规模但存在噪声的图像-文本对数据为对比语言-图像预训练（Contrastive Language-Image Pretraining，CLIP）的成功奠定了基础。作为基础视觉编码器，CLIP反过来又成为大多数大型视觉语言模型（large vision-language models，LVLMs）的基石。这种相互依存关系自然引出了一个有趣的问题：我们能否反过来利用LVLMs提升图像-文本对数据的质量，从而开启一个持续改进的自我强化循环的可能性？在本研究中，我们朝着这一愿景迈出了重要一步，提出了一种由LVLM驱动的数据精炼流水线。我们的框架利用LVLMs处理图像及其原始替代文本，生成四种互补的文本形式：长正例描述、长负例描述、短正例标签和短负例标签。将该流水线应用于经过整理的DFN-Large数据集，得到了VLM-150M——一个富含多粒度标注的精炼数据集。基于该数据集，我们进一步提出了一种训练范式，通过纳入负例描述和短标签作为额外监督信号，对传统对比学习进行了扩展。由此得到的模型名为HQ-CLIP，在各类基准测试中展现出显著提升。在训练数据规模相当的情况下，我们的方法在零样本分类、跨模态检索和细粒度视觉理解任务上取得了当前最优性能。在检索基准测试中，HQ-CLIP甚至超越了在DFN-2B数据集上训练的标准CLIP模型，而DFN-2B的训练数据量是我们的10倍。所有代码、数据和模型均可在https://zxwei.site/hqclip获取。
英文摘要：
Large-scale but noisy image-text pair data have paved the way for the success of Contrastive Language-Image Pretraining (CLIP). As the foundation vision encoder, CLIP in turn serves as the cornerstone for most large vision-language models (LVLMs). This interdependence naturally raises an interesting question: Can we reciprocally leverage LVLMs to enhance the quality of image-text pair data, thereby opening the possibility of a self-reinforcing cycle for continuous improvement? In this work, we take a significant step toward this vision by introducing an LVLM-driven data refinement pipeline. Our framework leverages LVLMs to process images and their raw alt-text, generating four complementary textual formulas: long positive descriptions, long negative descriptions, short positive tags, and short negative tags. Applying this pipeline to the curated DFN-Large dataset yields VLM-150M, a refined dataset enriched with multi-grained annotations. Based on this dataset, we further propose a training paradigm that extends conventional contrastive learning by incorporating negative descriptions and short tags as additional supervised signals. The resulting model, namely HQ-CLIP, demonstrates remarkable improvements across diverse benchmarks. Within a comparable training data scale, our approach achieves state-of-the-art performance in zero-shot classification, cross-modal retrieval, and fine-grained visual understanding tasks. In retrieval benchmarks, HQ-CLIP even surpasses standard CLIP models trained on the DFN-2B dataset, which contains 10$\times$ more training data than ours. All code, data, and models are available at https://zxwei.site/hqclip.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
