CLIP Tricks You: 面向大型视觉-语言模型中高效像素定位的无训练令牌剪枝
CLIP Tricks You: Training-free Token Pruning for Efficient Pixel Grounding in Large VIsion-Language Models
AI总结：
提出LiteLVLM，一种无需训练、文本引导的令牌剪枝策略，通过反转CLIP视觉-文本相似度排序，保留指代区域令牌并恢复上下文令牌，实现高效像素定位推理，在多种令牌预算下性能提升超5%，保持90%原始性能同时加速22%并减少2.3倍内存。
摘要：
在大型视觉-语言模型中，视觉令牌通常构成输入令牌的大部分，导致大量计算开销。为了解决这个问题，最近的研究探索了为图像理解任务剪枝冗余或信息量较少的视觉令牌。然而，这些方法在像素定位任务中表现不佳，因为令牌重要性高度依赖于输入文本。通过对CLIP的深入分析，我们观察到指代区域内的视觉令牌与其文本表示的相似度通常较低。受此启发，我们引入了LiteLVLM，一种无需训练、文本引导的令牌剪枝策略，用于高效的像素定位推理。通过反转CLIP视觉-文本相似度的排序，LiteLVLM有效地保留了覆盖指代区域的视觉令牌，同时恢复上下文令牌以实现清晰的前景-背景分离。大量实验表明，LiteLVLM在不同令牌预算下均显著优于现有方法，性能提升超过5%。无需任何训练或微调，LiteLVLM在保持90%原始性能的同时，实现了22%的加速和2.3倍的内存减少。我们的代码可在https://github.com/sejong-rcv/LiteLVLM获取。
英文摘要：
In large vision-language models, visual tokens typically constitute the majority of input tokens, leading to substantial computational overhead. To address this, recent studies have explored pruning redundant or less informative visual tokens for image understanding tasks. However, these methods struggle with pixel grounding tasks, where token importance is highly contingent on the input text. Through an in-depth analysis of CLIP, we observe that visual tokens within referent regions often exhibit low similarity to their textual representation. Motivated by this insight, we introduce LiteLVLM, a training-free, text-guided token pruning strategy for efficient pixel grounding inference. By reversing the ranking of CLIP's visual-text similarity, LiteLVLM effectively retains visual tokens covering the referent regions, while recovering context tokens to enable clear foreground-background separation. Extensive experiments demonstrate that LiteLVLM significantly outperforms existing methods by over 5% across diverse token budgets. Without any training or fine-tuning, LiteLVLM maintains 90% of the original performance with a 22% speedup and a 2.3X memory reduction. Our code is available at https://github.com/sejong-rcv/LiteLVLM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
