递归CLIP:使用循环Transformer块的视觉-语言蒸馏
Recursive CLIP: Vision-Language Distillation with Looped Transformer Blocks
AI总结:
该研究提出递归CLIP系列模型,通过复用Transformer块提升参数效率,在对比训练和知识蒸馏下验证其在零样本等任务的性能,确立递归块复用为视觉-语言模型的实用设计方向。
摘要:
CLIP等对比式视觉-语言模型通常在各层使用参数独立的Transformer块。我们研究这些块能否循环复用同时保留CLIP的迁移能力。我们研究递归Transformer骨干,其中前向传播时对少量N个Transformer块应用M次。通过将N·M固定为标准ViT-B的深度,我们的模型在保留计算深度的同时逐步增加参数共享。我们考虑两类递归CLIP模型:rCLIP(仅视觉编码器递归)和r²CLIP(文本编码器也引入递归)。我们在直接对比训练和从在DFN-2B上预训练的强ViT-B/16教师模型进行知识蒸馏两种场景下研究这些模型,使用规模小得多的CC3M和CC12M数据集。我们在零样本分类和跨模态检索上评估所得模型,还通过少样本学习、视觉 grounding 和概念遗忘研究其适应性。结果表明CLIP可容忍大量递归块复用,尽管其效果取决于编码器和下游任务;递归还天生具备参数效率,因为只需存储和适配更少的独立Transformer块。我们的结果确立递归块复用是视觉-语言模型的实用设计空间,并强调其对知识迁移、下游适配和参数效率的意义。
英文摘要:
Contrastive vision-language models such as CLIP typically rely on independently parameterized transformer blocks across depth. We investigate the extent to which these blocks can instead be reused recurrently while preserving the transferable capabilities of CLIP. We study recursive transformer backbones in which a small set of $N$ transformer blocks is applied $M$ times per forward pass. By holding $N \cdot M$ fixed to the depth of a standard ViT-B, our models preserve computational depth while progressively increasing parameter sharing. We consider two recursive CLIP model families: rCLIP, where only the vision encoder is recursive, and r$^2$CLIP, where recurrence is also introduced in the text encoder. We study these models under both direct contrastive training and through knowledge distillation from a strong ViT-B/16 teacher pretrained on DFN-2B, using the substantially smaller CC3M and CC12M datasets. We evaluate the resulting models on zero-shot classification and cross-modal retrieval, and further study their adaptability through few-shot learning, visual grounding, and concept unlearning. Our results show that CLIP can tolerate substantial recurrent block reuse, although its effect depends on the encoder and downstream task. Recurrence also provides parameter efficiency by construction, since fewer distinct transformer blocks need to be stored and adapted. Our results establish recurrent block reuse as a practical design space for vision-language models and highlight its implications for knowledge transfer, downstream adaptation, and parameter efficiency.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。