通过知识蒸馏增强CLIP的概念嵌入
Enhancing CLIP Conceptual Embedding through Knowledge Distillation
AI总结：
针对CLIP图文编码器提取细粒度知识能力不足的问题，提出基于Llama 2知识蒸馏的Knowledge-CLIP方法，通过文本嵌入蒸馏、概念学习、对比学习三个模块，同时提升CLIP图文编码器的性能。
摘要：
近年来，CLIP已成为多模态场景下实现图文对齐的重要模型。然而研究人员发现，CLIP的文本编码器与图像编码器从图文描述对中提取细粒度知识的能力存在局限。针对这一问题，本文提出Knowledge-CLIP这一创新方法，该方法通过整合基于Llama 2的新型知识蒸馏（KD）技术来提升CLIP的性能。该方法围绕三个核心目标展开：文本嵌入蒸馏、概念学习和对比学习。首先，文本嵌入蒸馏阶段训练Knowledge-CLIP的文本编码器，使其拟合教师模型Llama 2的输出；其次，概念学习阶段通过对Llama 2生成的文本数据执行离线K-means聚类，为每个图文描述对分配软概念标签，让Knowledge-CLIP从这些软概念标签中学习；最后，对比学习阶段对齐文本嵌入与图像嵌入。实验结果表明，所提模型同时提升了文本编码器与图像编码器的性能。
英文摘要：
Recently, CLIP has become an important model for aligning images and text in multi-modal contexts. However, researchers have identified limitations in the ability of CLIP's text and image encoders to extract detailed knowledge from pairs of captions and images. In response, this paper presents Knowledge-CLIP, an innovative approach designed to improve CLIP's performance by integrating a new knowledge distillation (KD) method based on Llama 2. Our approach focuses on three key objectives: Text Embedding Distillation, Concept Learning, and Contrastive Learning. First, Text Embedding Distillation involves training the Knowledge-CLIP text encoder to mirror the teacher model, Llama 2. Next, Concept Learning assigns a soft concept label to each caption-image pair by employing offline K-means clustering on text data from Llama 2, enabling Knowledge-CLIP to learn from these soft concept labels. Lastly, Contrastive Learning aligns the text and image embeddings. Our experimental findings show that the proposed model improves the performance of both text and image encoders.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
