Patch Ranking：通过学习局部补丁排序实现高效CLIP
Patch Ranking: Efficient CLIP by Learning to Rank Local Patches
AI总结：
针对CLIP因ViT骨干计算量高的问题，提出通过贪心搜索建立“黄金排序”并训练轻量级预测器逼近该排序，融入可学习视觉token补偿剪枝损失，在减少40%补丁token的同时仅损失0.3平均精度，提升CLIP效率。
摘要：
CLIP等对比图像-文本预训练模型对下游任务表现出卓越的适应性，但因Vision Transformer（ViT）骨干的高计算需求面临挑战。当前提升ViT效率的策略聚焦于剪枝补丁token，但未能充分解决CLIP的多模态特性，也难以识别最优token子集以实现最大性能。为此，我们提出贪心搜索方法建立“黄金排序”，并引入轻量级预测器专门训练以逼近该排序。为补偿token剪枝导致的性能下降，我们融入可学习视觉token，助力恢复并可能提升模型性能。我们对CLIP模型ViT骨干内的token剪枝展开全面系统研究。通过我们的框架，成功在CLIP的ViT中减少40%的补丁token，同时在七个数据集上仅遭受0.3的最小平均精度损失。我们的研究为构建更具计算效率的多模态模型奠定基础，且不牺牲性能，解决了先进视觉-语言模型应用中的关键挑战。
英文摘要：
Contrastive image-text pre-trained models such as CLIP have shown remarkable adaptability to downstream tasks. However, they face challenges due to the high computational requirements of the Vision Transformer (ViT) backbone. Current strategies to boost ViT efficiency focus on pruning patch tokens but fall short in addressing the multimodal nature of CLIP and identifying the optimal subset of tokens for maximum performance. To address this, we propose greedy search methods to establish a "Golden Ranking" and introduce a lightweight predictor specifically trained to approximate this Ranking. To compensate for any performance degradation resulting from token pruning, we incorporate learnable visual tokens that aid in restoring and potentially enhancing the model's performance. Our work presents a comprehensive and systematic investigation of pruning tokens within the ViT backbone of CLIP models. Through our framework, we successfully reduced 40% of patch tokens in CLIP's ViT while only suffering a minimal average accuracy loss of 0.3 across seven datasets. Our study lays the groundwork for building more computationally efficient multimodal models without sacrificing their performance, addressing a key challenge in the application of advanced vision-language models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
