通过自蒸馏编码器加速对比语言-图像预训练
Expediting Contrastive Language-Image Pretraining via Self-distilled Encoders
AI总结：
提出ECLIPSE框架，通过共享文本编码器的自蒸馏架构，在统一文本嵌入空间中加速在线图像编码器，解决弱相关图像-文本对的数据效率问题，实现更优性能与更快推理。
摘要：
近年来，视觉语言预训练（VLP）的进展很大程度上归功于从网络收集的大规模数据。然而，未经整理的数据集包含弱相关的图像-文本对，导致数据效率低下。为解决这一问题，研究者们探索了知识蒸馏方法，但代价是需要额外的图像和文本动量编码器来为错配的图像-文本对生成教学信号。在本文中，我们的目标是通过一个高效的蒸馏框架来解决错配问题。为此，我们提出了ECLIPSE：通过自蒸馏编码器加速对比语言-图像预训练。ECLIPSE采用了一种独特的蒸馏架构，其中在线图像编码器和动量图像编码器之间共享一个文本编码器。这一策略性设计选择使得蒸馏能够在统一的文本嵌入投影空间内进行，从而获得更好的性能。基于统一的文本嵌入空间，ECLIPSE通过加速在线图像编码器来补偿动量图像编码器带来的额外计算成本。通过大量实验，我们验证了在加速与蒸馏之间存在一个最佳平衡点，在该平衡点上，来自加速后的在线图像编码器的部分视图与动量教师互补交互。最终，ECLIPSE在显著提升推理速度的同时，性能优于同类方法。
英文摘要：
Recent advances in vision language pretraining (VLP) have been largely attributed to the large-scale data collected from the web. However, uncurated dataset contains weakly correlated image-text pairs, causing data inefficiency. To address the issue, knowledge distillation have been explored at the expense of extra image and text momentum encoders to generate teaching signals for misaligned image-text pairs. In this paper, our goal is to resolve the misalignment problem with an efficient distillation framework. To this end, we propose ECLIPSE: Expediting Contrastive Language-Image Pretraining with Self-distilled Encoders. ECLIPSE features a distinctive distillation architecture wherein a shared text encoder is utilized between an online image encoder and a momentum image encoder. This strategic design choice enables the distillation to operate within a unified projected space of text embedding, resulting in better performance. Based on the unified text embedding space, ECLIPSE compensates for the additional computational cost of the momentum image encoder by expediting the online image encoder. Through our extensive experiments, we validate that there is a sweet spot between expedition and distillation where the partial view from the expedited online image encoder interacts complementarily with the momentum teacher. As a result, ECLIPSE outperforms its counterparts while achieving substantial acceleration in inference speed.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
